Relayer's pending transactions queue seems stuck

Support Defender
1

Hi Team,

I am running a relayer on Arbitrum Goerli and transactions are not coming through - I can see the latest one from Relayer's address on arbiscan was submitted around 10 hours ago and I already reached a limit of 120 pending txns.

Two questions:

  1. Is there any way I can see why it is failing? A relayer either does not process the queue or constantly tries to resubmit transactions that fail for some reason.
  2. Is there a way to purge queued transactions?

Thank you!
:computer: Environment
Relayer + Autotask Webhooks on Arbitrum Goerli

:memo:Details

:1234: Code to reproduce

2

Do you have sample code we can look on?

4

This is autotask source - it is basic stuff mostly from OZ workshop repository. A testing script is also written based on OZ samples, I just translated it to Typescript. Please let me know if you want me to post samples from it too. Thanks

import { AutotaskEvent, AutotaskRequestData } from "defender-autotask-utils";
import { AutotaskRelayerParams } from "defender-relay-client";
import { DefenderRelayProvider, DefenderRelaySigner } from "defender-relay-client/lib/ethers";
import { ethers } from "ethers";

const ABI_FORWARDER = [{"inputs":[],"stateMutability":"nonpayable","type":"constructor"},{"inputs":[{"components":[{"internalType":"address","name":"from","type":"address"},{"internalType":"address","name":"to","type":"address"},{"internalType":"uint256","name":"value","type":"uint256"},{"internalType":"uint256","name":"gas","type":"uint256"},{"internalType":"uint256","name":"nonce","type":"uint256"},{"internalType":"bytes","name":"data","type":"bytes"}],"internalType":"struct MinimalForwarder.ForwardRequest","name":"req","type":"tuple"},{"internalType":"bytes","name":"signature","type":"bytes"}],"name":"execute","outputs":[{"internalType":"bool","name":"","type":"bool"},{"internalType":"bytes","name":"","type":"bytes"}],"stateMutability":"payable","type":"function"},{"inputs":[{"internalType":"address","name":"from","type":"address"}],"name":"getNonce","outputs":[{"internalType":"uint256","name":"","type":"uint256"}],"stateMutability":"view","type":"function"},{"inputs":[{"components":[{"internalType":"address","name":"from","type":"address"},{"internalType":"address","name":"to","type":"address"},{"internalType":"uint256","name":"value","type":"uint256"},{"internalType":"uint256","name":"gas","type":"uint256"},{"internalType":"uint256","name":"nonce","type":"uint256"},{"internalType":"bytes","name":"data","type":"bytes"}],"internalType":"struct MinimalForwarder.ForwardRequest","name":"req","type":"tuple"},{"internalType":"bytes","name":"signature","type":"bytes"}],"name":"verify","outputs":[{"internalType":"bool","name":"","type":"bool"}],"stateMutability":"view","type":"function"}];
const ADDRESS_FORWADER: string = "0xA4FA1C803E9FeF752641C4C0aD7808a9681CEb13"; // arb goerli

async function relay(forwarder: any, request: any, signature: any): Promise<any> {

    // Validate request on the forwarder contract
    const valid = await forwarder.verify(request, signature);
    if (!valid) throw new Error(`Invalid request`);

    // Send meta-tx through relayer to the forwarder contract
    const gasLimit = (parseInt(request.gas) + 50000).toString();
    return await forwarder.execute(request, signature, { gasLimit });
}

export async function handler(event: AutotaskEvent) {
    const body = event.request?.body as Record<string, unknown>;

    const { request, signature } = body;
    console.log(`Relaying ${JSON.stringify(request)}; signature ${signature}`);

    // Initialize Relayer provider and signer, and forwarder contract
    const credentials = { ... event } as AutotaskRelayerParams;
    const provider = new DefenderRelayProvider(credentials);
    const signer = new DefenderRelaySigner(credentials, provider, { speed: 'fast' });
    const forwarder = new ethers.Contract(ADDRESS_FORWADER, ABI_FORWARDER, signer);
  
    // Relay transaction!
    const tx = await relay(forwarder, request, signature);
    console.log(`Sent meta-tx: ${tx.hash}`);
    return { txHash: tx.hash };
}
5

You need to do some troubleshooting as to why the transactions are not going through, you may want to check the logs of the relayer to see if there are any error messages that can help identify the cause of the issue. It is possible that the transactions are failing due to a problem with the relayer, the Ethereum Goerli test network, or some other factor.

You can post the specific you're getting with sample codes.

6

I'd love to do troubleshooting on my own. Can't find any Relayer logs I could check, could you point me, please? I only see autotask logs which show a relayer returns transaction hash and completes successfully. I can't find those transactions on the chain though.

7

Hello All, nothing changed since yesterday, 120 txns limit is reached, and all pending transactions were executed only once at the time of submission (CREATED == LAST SENT), like the one on the screenshot below. It is definitely an issue with the Relay service, isn't it?

How this can be resolved?

image
image1499×198 14.9 KB

8

To check why the transactions are failing, you can check the logs of the transaction by calling the getTransactionReceipt method on your provider object, which should return a receipt object containing the transaction logs. You can then access the logs property of the receipt object and iterate through the logs to see if any of them contain an error message.

To purge the queued transactions, you can use the sendTransaction method on the signer object with a nonce value equal to the current nonce of the relayer's address. This will reset the nonce value to the current block's nonce, which should allow the relayer to send new transactions that are not stuck in the queue.
example of how you can do this:

// Get the current nonce of the relayer's address
const currentNonce = await provider.getTransactionCount(signer.address);

// Send a transaction with the current nonce to reset the nonce value
await signer.sendTransaction({ nonce: currentNonce });

Note that this will only work if the relayer's address has a balance sufficient to pay for the transaction fees for the reset transaction.

1 Like
9

I reset a transaction with the current nonce on the chain which was 119 and that transaction has gone from the pending list. However, I have a long list of other pending transactions with nonces higher than 119. Does that mean I need a script to loop through all pending transactions in the Relay queue and issue a corresponding replacement transaction for each? Or should I issue only one with the higher nonce I see in that pending list?

10

Well, this is what I get when trying to send transactions with nonce 120, 121, and so on:

  response: {
    status: 400,
    statusText: 'Bad Request',
    data: { message: 'Cannot replace transaction in state failed' }
  }

Transactions remain in the pending queue, unfortunately.

11

Finally, I managed to clear the pending queue by running replaceTransactionById for each transaction in ascending order of nonce. However, the reason for such failure remains unclear. Apparently, there are no tools available to troubleshoot the relay service.

12

Sorry for replying late.
Sending a new transaction with the same nonce, but with a higher gas price will effectively replace the old transaction. I initially suggested you do so, you probably didn't pay attention to the writeups above.
I'm glad it works finally.

13

Yeah, probably I missed something, sorry. Even now I can't see any thing abou higher gas price value :slight_smile: Anyway, I appreciate your help.

1 Like