Open Zeppelin Defender

Hey, we have been facing an issue with our Defender relayers lately. After sending a transaction to Relay, the transactions are just queued and not broadcasted to the blockchain.

We tried to replace transactions in order to clear the queue, but that did not work either.

We ultimately had to create new relayers and use them instead. Our funds are still stuck in the old relayers and we are unable to withdraw it due to queued txs. Here are the IDs of relayers we faced this issue with :

Relayer IDs :

  1. Arbitrum : 32e1ff10-9b44-498b-8ff9-f26c3c2fde75
  2. Arbitrum : 27519e24-12f2-42ff-904e-ec56dbcf3ba5
  3. Optimism : d3f2f064-65bc-4980-8a62-4526d4c3077f
  4. Aurora : 20044977-8419-4f7f-adb1-d520456882f4

We would really appreciate if you can help us in understanding and fixing this issue. And, also in recovering our funds stuck on the relayers.

1 Like

Hi @ShreyKeny

Apologies for the inconvenience. We are investigating this issue. To help you as fast as possible, could you provide us with the lowest nonce (for each relayer), from the point it got stuck? You should be able to see this in the first column of the "unconfirmed transactions" table. This will help us track down the transaction that's causing the subsequent transactions to get stuck.

Thank you!

1 Like

Hey, Arth here from same team. Here are the nonces for all our stuck relayers.






Hey @nami, thanks for the fast reply!

Can you please check the nonce numbers shared by @arthcp?

Hi @ShreyKeny @arthcp,

We're still investigating why these transactions got stuck. Meanwhile, to unblock your relayer, are you happy with me cancelling the stuck transactions for each of your relayers?

Hey @nami,

That would be great! Please do cancel all the transactions in the queue on each of these relayers.

Please do let us know once it's done. Thank you!

Also @nami,

We'd really appreciate if we can set up a group chat between our teams ASAP. I can share my TG / Discord if you DM-ed me here.

We've also reached out via the defender email & filled in the inquiry form. So, just escalating it here as well :slight_smile:

We're heavily relying on Defender Relay and running into issues quite frequently lately. We'd really appreciate dedicated support to resolve these immediately!

Hi @ShreyKeny

I have deleted all pending transactions on all relayers. Let me know if you run into any more issues.

As for a group chat, I will loop in Jack on the email thread who can further assist you with that :slight_smile:

Hey @nami, really appreciate the help!

Hoping to hear from you guys soon about the group chat. Thanks a lot!

Hey @nami, we are experiencing the same issue again. Would appreciate if you can help us debug. Sharing the relayer ID and nonce at which tx is stuck below.

Relayer ID : 8241a40a-0e93-4955-bd12-d504a7df5b0d
Nonce : 3460

cc : @ernestognw

Hey @ShreyKeny,

We just released a fix for the stuck transactions. Previously it was an error on our resubmission logic. However, this time seems like the arbitrum nodes are behaving weird, look, this is the rawTransaction you got stuck:


If you use a tx decoder you'll see that gasPrice and gasLimit are both reasonable (2000000000 and 60000), however, when sending the tx to a public Arbitrum RPC endpoint shows that you don't have enough funds (which you actually have).

curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["0xf901ce820d84847735940082ea6094c0e02aa55d10e38855e13b64a8e1387a04681a0080b901640ddedd8400000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000012000000000000000000000000000000000000000000000000000000000000075300000000000000000000000000000000000000000000000000058d15e1762800000000000000000000000000000000000000000000000000000000000000000010000000000000000000000002f37d0a7427cd4746a0b5f31a875ad5c5e976b780000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000002f87a98fe7ed730000000000000000000000000000000000000000000000000000000000000001dd7db2783f21d053a4916769d884929768ed70955020655a5913836707b9bcc883014985a0af70f8f1b3e47df9372b86f473243ed8a390a66385871180e23af3ebb384ec05a0542db329ef2ea452ec242d89af1d081e176092c5a8ded5db2c896844c3f86ccf"],"id":1}'


{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"not enough funds for gas"}}

Before resetting your relayer and sending NOOPs, I'll investigate further why is Arbitrum doing this, please validate the information I'm showing you. And if you want me to reset your relayer anyway, please let me know

Hey @ernestognw, we tried sending the tx to our RPC endpoint as well and it oddly returns the same message -- "not enough funds for gas"

We'd appreciate if you'd investigate further why this is happening. In the meantime, we'll disable support for Arbitrum on our application.

Hoping to hear from you ASAP. Thank you!

1 Like

Hey guys,

Getting back to you, seems like Arbitrum sends the not enough funds message whenever the gasLimit is low according to this thread, and also it was spotted by us in the past.

Makes sense since when I run an estimation on your transaction it returns me 839774 when you're using 60000

curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"eth_estimateGas","params":[{"gasPrice": "0x77359400","gasLimit": "0xea60","to": "0xc0e02aa55d10e38855e13b64a8e1387a04681a00","value": "0x0","data": "0x0ddedd8400000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000012000000000000000000000000000000000000000000000000000000000000075300000000000000000000000000000000000000000000000000058d15e1762800000000000000000000000000000000000000000000000000000000000000000010000000000000000000000002f37d0a7427cd4746a0b5f31a875ad5c5e976b780000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000002f87a98fe7ed730000000000000000000000000000000000000000000000000000000000000001dd7db2783f21d053a4916769d884929768ed70955020655a5913836707b9bcc8","from": "0x95e655674c6889f80fa024eba86cde29d69028a6"}],"id":1}'

Right now there are 3 options you can follow:

  1. You can send a replacement for those transactions using the defender-relay-client via API and set a higher gasLimit so they can go through
  2. You can kill the txs (or let me do that) by NOOPing
  3. You can wait until the validUntil kicks in and NOOP your transactions.

I'd go for number 1, but let me know if you need help

Hey @ernestognw,

Thanks for the prompt response. I see why the transactions weren't going through. We had hardcoded the gasLimit temporarily as Arbitrum was giving inconsistent estimates a couple of hours ago, but they've normalised now.

Can you please kill these txs for us? (Option 2) We'll re-initiate them later on our end.

Please do let us know once you do :pray:

1 Like

It's done @ShreyKeny, I replaced them with NOOPs so I'm expecting the relayer to get cleared soon.

Please let us know if you need anything else.


1 Like

Hey @ernestognw, thanks a lot for your help! We were able to re-initiate these transactions after the relayer was cleared.

We have everything sorted for the time being. Really appreciate your time & help!

Have a nice day!

1 Like

Sounds amazing @ShreyKeny
Happy to help as always, please report again if you have any issue, I'm particularly interested on seeing if the hotfix that I sent for stuck transactions on rollups has been a success.

We'll be monitoring anyhow. Have a nice day as well!

1 Like