Hey, we have been facing an issue with our Defender relayers lately. After sending a transaction to Relay, the transactions are just queued and not broadcasted to the blockchain.
We tried to replace transactions in order to clear the queue, but that did not work either.
We ultimately had to create new relayers and use them instead. Our funds are still stuck in the old relayers and we are unable to withdraw it due to queued txs. Here are the IDs of relayers we faced this issue with :
Relayer IDs :
Arbitrum : 32e1ff10-9b44-498b-8ff9-f26c3c2fde75
Arbitrum : 27519e24-12f2-42ff-904e-ec56dbcf3ba5
Optimism : d3f2f064-65bc-4980-8a62-4526d4c3077f
Aurora : 20044977-8419-4f7f-adb1-d520456882f4
We would really appreciate if you can help us in understanding and fixing this issue. And, also in recovering our funds stuck on the relayers.
Apologies for the inconvenience. We are investigating this issue. To help you as fast as possible, could you provide us with the lowest nonce (for each relayer), from the point it got stuck? You should be able to see this in the first column of the "unconfirmed transactions" table. This will help us track down the transaction that's causing the subsequent transactions to get stuck.
We're still investigating why these transactions got stuck. Meanwhile, to unblock your relayer, are you happy with me cancelling the stuck transactions for each of your relayers?
We'd really appreciate if we can set up a group chat between our teams ASAP. I can share my TG / Discord if you DM-ed me here.
We've also reached out via the defender email & filled in the inquiry form. So, just escalating it here as well
We're heavily relying on Defender Relay and running into issues quite frequently lately. We'd really appreciate dedicated support to resolve these immediately!
Hey @nami, we are experiencing the same issue again. Would appreciate if you can help us debug. Sharing the relayer ID and nonce at which tx is stuck below.
Relayer ID : 8241a40a-0e93-4955-bd12-d504a7df5b0d
Nonce : 3460
We just released a fix for the stuck transactions. Previously it was an error on our resubmission logic. However, this time seems like the arbitrum nodes are behaving weird, look, this is the rawTransaction you got stuck:
If you use a tx decoder you'll see that gasPrice and gasLimit are both reasonable (2000000000 and 60000), however, when sending the tx to a public Arbitrum RPC endpoint shows that you don't have enough funds (which you actually have).
curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["0xf901ce820d84847735940082ea6094c0e02aa55d10e38855e13b64a8e1387a04681a0080b901640ddedd8400000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000012000000000000000000000000000000000000000000000000000000000000075300000000000000000000000000000000000000000000000000058d15e1762800000000000000000000000000000000000000000000000000000000000000000010000000000000000000000002f37d0a7427cd4746a0b5f31a875ad5c5e976b780000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000002f87a98fe7ed730000000000000000000000000000000000000000000000000000000000000001dd7db2783f21d053a4916769d884929768ed70955020655a5913836707b9bcc883014985a0af70f8f1b3e47df9372b86f473243ed8a390a66385871180e23af3ebb384ec05a0542db329ef2ea452ec242d89af1d081e176092c5a8ded5db2c896844c3f86ccf"],"id":1}' https://rpc.ankr.com/arbitrum
returns:
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"not enough funds for gas"}}
Before resetting your relayer and sending NOOPs, I'll investigate further why is Arbitrum doing this, please validate the information I'm showing you. And if you want me to reset your relayer anyway, please let me know
Getting back to you, seems like Arbitrum sends the not enough funds message whenever the gasLimit is low according to this thread, and also it was spotted by us in the past.
Makes sense since when I run an estimation on your transaction it returns me 839774 when you're using 60000
Thanks for the prompt response. I see why the transactions weren't going through. We had hardcoded the gasLimit temporarily as Arbitrum was giving inconsistent estimates a couple of hours ago, but they've normalised now.
Can you please kill these txs for us? (Option 2) We'll re-initiate them later on our end.
Sounds amazing @ShreyKeny
Happy to help as always, please report again if you have any issue, I'm particularly interested on seeing if the hotfix that I sent for stuck transactions on rollups has been a success.
We'll be monitoring anyhow. Have a nice day as well!
