Hello team,
We are experiencing an issue across all of our relayers where in a transaction is submitted to Defender Relay but is not broadcasted to the blockchain.
On the Defender UI logging section, we see that these transactions are marked stale and Defender is trying to re-send them.
This issue emerged ~ 1.5 hours ago. Can you please help us ASAP? We're using relayers on Optimism, Arbitrum, Gnosis Chain, Avalanche, Aurora, Polygon, Fantom and BSC
cc : @ernestognw @ericglau @bjmrq
It seems we have a similar problem on Arbitrum testnet.
Hi @ShreyKeny
Very sorry for these issues.
We identified an issue that impacted Relayers earlier today regarding our providers. It should now be resolved.
@Archethect_Archethec - I saw the other forum issues you submitted. If there are persistent issues with an Arbitrum relayer, could you please send more information about the relayer impacted such as:
URL of the page on Defender for that Relayer
or
The Relayer's address
Re: RPC endpoints, we use multiple providers to handle failover and the public JSON RPC endpoint you referenced is one of them
Dan McKeon
OpenZeppelin
Hey Dan,
Thanks for the follow up. The relaying seems to be fixed, but we still get a very strange error on one of our transactions. Will take this up in the other ticket.
Kind regards,
Archethect
Hi, we're experiencing the same issue here.
Our BSC relay is unable to send transactions as it got 120 stuck transactions due to this issue.
I'm unable to cancel the transactions and also unable to withdraw the funds from this relayer, probably due to an issue with the nonces.
Same issue here
Relayers...
50a60c70-9ca2-47c6-9bb4-c417c99dfbe3
95d16afa-ccc8-4fd3-936f-5ed7683257a4
My relay address is: 0xb10899410f73bc2516371e4a524441c936b45c49
Hey @Mr.Peanutz, thanks for letting us know. I've checked on your relayer and seems like you hit a condition that's very particular:
gasPrice you're sending is 5gwei, which is also the gasPriceCap you seem to have set on the relayer policiesWhat I can do is run a script to clear your relayer avoiding the gasPrice cap, or you can remove it. However, we understand this is not the expected behavior that a relayer with a cap should have, so, I have to get back to the team and discuss.
Curious, do you think we should avoid the cap when we're sending a NOOP? (expired or cancelled transaction).
Best
Hey @amaurer-rise.
I notice both of your relayers are already cleared.
Is everything correct? I'd like to know how much the relayer was stuck, so I can send a fix if there's some misbehavior.
Best!
Hi @ernestognw, thanks for explaining, so isn't there any way to cancel the transactions without broadcasting them to the blockchain?
Anyways, if possible i'd like you to run that script so i wouldnt have to clear it out manually.
And yes, I think it should be ignored since it would be replacing a previous transaction
Yeah, there's a way to try that. I can delete every pending transaction we have on Defender and reset your Relayer's nonce, so you can start sending transactions again.
It's worth noting that it's almost certain that the nodes have dropped previous transactions, but unfortunately we can't guarantee that. In such case, you'll have your relayer stuck trying to send cheaper transactions than the ones already sent. But, to be honest, I really doubt any node kept the old txs.
Are you okay with me deleting all of the stuck transactions and reset your Relayer's nonce?
Best
Done, I deleted every transaction on our side, reset your Relayer's nonce and set pending costs to 0, so your relayer should behave normally now.
Best!
@ernestognw, we seem to have the same problem for relayer d610b378-01ca-4844-88ed-4aa7981b605d. Can you rest it as well please?
Kind regards
Just reset, failed transactions were deleted and nonce got back to what's on chain.
Best!