For the past few days, we seem unable to get any transactions successfully relayed on Arbitrum Goerli Testnet. We are testing a feature which will use Relay on Arbitrum mainnet, so we are curious what the issue is here.
Relay ID: fa91880c-96dc-47d7-8a76-213060298868
Code to reproduce
Transactions are being sent using DefenderRelaySigner from the defender-relay-client/lib/ethers package
const signer = new DefenderRelaySigner(relayerCredentials[sdkNetwork], provider, {
speed: 'fast'
});
Could you let us know more about what you are perceiving as being an issue so we can better understand it? maybe it does not have the behavior you were expected?
The transactions you see on arbiscan are the last successful txs we could get relayed. Wednesday through Friday last week we were unable to get any through. Each tx we attempt to relay shows up in the unconfirmed transaction queue but never gets confirmed. They expire and are removed from the queue after 8 hours however.
I just attempted two new transactions and am still experiencing the issue. Here's a screenshot of the txs in the unconfirmed queue:
Just to confirm, those txs have sat in the queue without executing. They are about to expire though. I can create more if it helps to debug the issue. Let me know.
Hello again. Just confirming this is still an issue today with new txs that I've tried sending. I noticed the gas price is different, but the tx still just sits in the queue.
At MakerDAO, we are planning to use Defender Relay as part of our "gasless" poll voting feature. We're close to being ready to release but recently ran into this issue. Until we have more information on why these txs can not get through the queue, we're unlikely able to release this feature.
Bump. We still can't get any transactions through on this relayer. It doesn't seem to occur on our mainnet relayer but this is our relayer for testing/goerli and we're wondering what the issue is. Are there any known issues with the Arbitrum Goerli testnet? Anyone from the Open Zeppelin team who can help look into why all our txs sit in the queue until they expire?
I've been researching this issue all day and here is what I have found:
There were 2 transaction nonces that caused issues: One had nonce 90 and one had nonce 91.
The one with nonce 90 was repeatedly attempted to be submitted but the node rejected it each time with an error like:
gas required exceeds allowance (2975150)
I traced that error to the Arbitrum geth implementation. It indicates that there is a max cap on gas and this transaction exceeded it. However, in researching Arbitrum gas limits it seems like this should have been accepted, so I don't have a great explanation for the node returning this error other than "the testnet node was acting funny". Let me know if you have any other context here though.
Eventually this transaction was submitted as a noop (and with a lower gas limit of 961326)
Meanwhile, the transaction with nonce 91 had been submitted. It had not yet been mined despite appearing to be close the current nonce for this account. We made an assumption that the transaction with nonce 90 had been submitted successfully and attempted to keep submitting this one, but the node responded with an error related to the Arbitrum sequencer not expecting this nonce:
nonce too high
We kept attempting to resubmit nonce 91 while nonce 90 was stuck and eventually tried it enough times that it was eventually considered "failed". I think we have some optimizations we can make to prevent a tx from entering the failed state in such a scenario, which should prevent this Relayer from getting stuck.
For the moment, I can clear this Relayer's pending transactions which will reset it and make it available for use again, please let me know if you would like me to do so.
Appreciate the detailed investigation and response. It's good to know this appears to be an edge case on a testnet node. All the txs we're sending through this relayer should be calling the same function on the same contract, so I'm curious why the node interpreted the gas required as being so high. The function does loop over an array parameter, so one possibility is that maybe too many items were included in the array. We can look into it a little bit on our end.
If you can clear the pending txs for this relayer so it can be used again, that'd be great.
Environment
Details
Code to reproduce
