Defender Relayer Issues October 9, 2022

We are aware of issues users are experiencing with Defender Relayers and are investigating we'll provide updates on this thread.

2 Likes

Just want to provide an update that we have a team investigating this incident, but unfortunately we do not yet have a resolution. We'll provide updates on this channel as we have more information.

We've identified one of the causes of this incident related to a SQS queue being overwhelmed with transactions, which is impacting Defender Relayers. We've implemented a fix for that acute issue but are not yet considering this issue resolved. Will update as we have more information.

We've implemented a couple of fixes related to this incident, but we are still seeing relayers with pending transactions and are continuing to investigate the issue.

We've now implemented additional fixes and Defender relayers appear to be operating as expected. Apologies all for issues this incident has caused.

1 Like

Hey @dan_oz , this is still happening to us. Transactions are not going through and they are accumulating

For context: we have an autotask that runs every 5 minutes, with a 'vaild until' value of 2 minutes. But the transactions is not being executed, and neither is the cancellation. So they just accumulate

This issue is still not fixed for me, transactions sitting in unconfirmed transactions on arbitrum.

Hi all,
We've been having the same issue on Arbitrum Goerli (Arbitrum Nitro Testnet) since Friday 7th October afternoon
Thanks

1 Like

We are now seeing the same problem on Polygon also

Hi
@elias_taz @12345

Sorry to hear about the issues.

Could you please send relayerIds(visible in the url when opening relayer page) for relayers that are still experiencing issues to defender-support @ openzeppelin.com so we can check them.

Thanks!

Best,
Zeljko

@zeljko: 3662f092-cb21-4f27-8107-a3a5bbedbbf7

Thanks for following up

Hi Zeljko!

Thanks for your help,

Here is my open zeppelin relay id: 73eb24cf-2e83-43b9-a73e-d5ea41934f78

Have a great day,

Elias

Hi @12345

We have checked and unblocked relayer.
Thanks for your patience.
Feel free to reach us in case of additional issues.

Best
Zeljko

Hi @elias_taz

Thanks for sharing details.

I see that transactions are passing to the chain, everything seems ok.

Feel free to reach in case of new issues.

Also, have a great day!

Zeljko

Hi,

Thank you for your reply,
Could we have explanations as to why this bug happened:)?
We are prototyping on Arbitrum testnets but will move to prod in less than 3 weeks.

  • Can these kind of bugs happen in production?
  • How likely are these errors to occur again?
  • Do you have guidelines to handle these errors cases ?
  • Did this bug occur on Arbitrum mainnet as well?

Thank you for your help and have a good day too;)!!

Elias

Hi

Post mortem on this incident should be published by end of the week.

More details will be there about the issue, impact and steps that we have taken to prevent it in the future.

Best,
Zeljko

Hi,

Thank you for your reply,
Where will the postmortem be posted?

Thanks!

Have a good day,

Elias

Hi,

We post them here, on Forum.

Thank you!
Have a great day also!

Hi

In case you missed it, here is a link to post mortem.

Best,
Zeljko

Thanks!

So to be clear: Open Zeppelin Relayer was unavailable on Arbitrum (and Arbitrum Goerli) not due to an Arbitrum Provider but rather due to Open Zeppelin's Queue service being hard spammed and unavailable.

Have a good day,

1 Like