We are aware of issues users are experiencing with Defender Relayers and are investigating we'll provide updates on this thread.
2 Likes
Just want to provide an update that we have a team investigating this incident, but unfortunately we do not yet have a resolution. We'll provide updates on this channel as we have more information.
We've identified one of the causes of this incident related to a SQS queue being overwhelmed with transactions, which is impacting Defender Relayers. We've implemented a fix for that acute issue but are not yet considering this issue resolved. Will update as we have more information.
We've implemented a couple of fixes related to this incident, but we are still seeing relayers with pending transactions and are continuing to investigate the issue.
We've now implemented additional fixes and Defender relayers appear to be operating as expected. Apologies all for issues this incident has caused.
1 Like
Hey @dan_oz , this is still happening to us. Transactions are not going through and they are accumulating
For context: we have an autotask that runs every 5 minutes, with a 'vaild until' value of 2 minutes. But the transactions is not being executed, and neither is the cancellation. So they just accumulate
This issue is still not fixed for me, transactions sitting in unconfirmed transactions on arbitrum.
Hi all,
We've been having the same issue on Arbitrum Goerli (Arbitrum Nitro Testnet) since Friday 7th October afternoon
Thanks
1 Like
Hi
@elias_taz @12345
Sorry to hear about the issues.
Could you please send relayerIds(visible in the url when opening relayer page) for relayers that are still experiencing issues to defender-support @ openzeppelin.com so we can check them.
Thanks!
Best,
Zeljko
@zeljko: 3662f092-cb21-4f27-8107-a3a5bbedbbf7
Thanks for following up
Hi Zeljko!
Thanks for your help,
Here is my open zeppelin relay id: 73eb24cf-2e83-43b9-a73e-d5ea41934f78
Have a great day,
Elias
Hi @12345
We have checked and unblocked relayer.
Thanks for your patience.
Feel free to reach us in case of additional issues.
Best
Zeljko
Hi @elias_taz
Thanks for sharing details.
I see that transactions are passing to the chain, everything seems ok.
Feel free to reach in case of new issues.
Also, have a great day!
Zeljko
Hi,
Thank you for your reply,
Could we have explanations as to why this bug happened:)?
We are prototyping on Arbitrum testnets but will move to prod in less than 3 weeks.
- Can these kind of bugs happen in production?
- How likely are these errors to occur again?
- Do you have guidelines to handle these errors cases ?
- Did this bug occur on Arbitrum mainnet as well?
Thank you for your help and have a good day too;)!!
Elias
Hi
Post mortem on this incident should be published by end of the week.
More details will be there about the issue, impact and steps that we have taken to prevent it in the future.
Best,
Zeljko
Hi,
Thank you for your reply,
Where will the postmortem be posted?
Thanks!
Have a good day,
Elias
Hi,
We post them here, on Forum.
Thank you!
Have a great day also!
Thanks!
So to be clear: Open Zeppelin Relayer was unavailable on Arbitrum (and Arbitrum Goerli) not due to an Arbitrum Provider but rather due to Open Zeppelin's Queue service being hard spammed and unavailable.
Have a good day,
1 Like