Hello team,
We are experiencing an issue across all of our relayers where in a transaction is submitted to Defender Relay but is not broadcasted to the blockchain.
On the Defender UI logging section, we see that these transactions are marked stale and Defender is trying to re-send them.
This issue emerged ~ 1.5 hours ago. Can you please help us ASAP? We're using relayers on Optimism, Arbitrum, Gnosis Chain, Avalanche, Aurora, Polygon, Fantom and BSC
cc : @ernestognw @ericglau @bjmrq
It seems we have a similar problem on Arbitrum testnet.
Hi @ShreyKeny
Very sorry for these issues.
We identified an issue that impacted Relayers earlier today regarding our providers. It should now be resolved.
@Archethect_Archethec - I saw the other forum issues you submitted. If there are persistent issues with an Arbitrum relayer, could you please send more information about the relayer impacted such as:
URL of the page on Defender for that Relayer
or
The Relayer's address
Re: RPC endpoints, we use multiple providers to handle failover and the public JSON RPC endpoint you referenced is one of them
Dan McKeon
OpenZeppelin
Hey Dan,
Thanks for the follow up. The relaying seems to be fixed, but we still get a very strange error on one of our transactions. Will take this up in the other ticket.
Kind regards,
Archethect
Hey @dan_oz,
Thanks for your help! All of our relayers are working normally
Hi, we're experiencing the same issue here.
Our BSC relay is unable to send transactions as it got 120 stuck transactions due to this issue.
I'm unable to cancel the transactions and also unable to withdraw the funds from this relayer, probably due to an issue with the nonces.
Hey @dan_oz any idea on how to fix it?
Same issue here
Relayers...
50a60c70-9ca2-47c6-9bb4-c417c99dfbe3
95d16afa-ccc8-4fd3-936f-5ed7683257a4
My relay address is: 0xb10899410f73bc2516371e4a524441c936b45c49
Hey @Mr.Peanutz, thanks for letting us know. I've checked on your relayer and seems like you hit a condition that's very particular:
gasPrice you're sending is 5gwei, which is also the gasPriceCap you seem to have set on the relayer policiesWhat I can do is run a script to clear your relayer avoiding the gasPrice cap, or you can remove it. However, we understand this is not the expected behavior that a relayer with a cap should have, so, I have to get back to the team and discuss.
Curious, do you think we should avoid the cap when we're sending a NOOP? (expired or cancelled transaction).
Best
Hey @amaurer-rise.
I notice both of your relayers are already cleared.
Is everything correct? I'd like to know how much the relayer was stuck, so I can send a fix if there's some misbehavior.
Best!
Hi @ernestognw, thanks for explaining, so isn't there any way to cancel the transactions without broadcasting them to the blockchain?
Anyways, if possible i'd like you to run that script so i wouldnt have to clear it out manually.
And yes, I think it should be ignored since it would be replacing a previous transaction
Yeah, there's a way to try that. I can delete every pending transaction we have on Defender and reset your Relayer's nonce, so you can start sending transactions again.
It's worth noting that it's almost certain that the nodes have dropped previous transactions, but unfortunately we can't guarantee that. In such case, you'll have your relayer stuck trying to send cheaper transactions than the ones already sent. But, to be honest, I really doubt any node kept the old txs.
Are you okay with me deleting all of the stuck transactions and reset your Relayer's nonce?
Best
Yes I'm ok with deleting the transactions, thanks @ernestognw!
Done, I deleted every transaction on our side, reset your Relayer's nonce and set pending costs to 0, so your relayer should behave normally now.
Best!
@ernestognw, we seem to have the same problem for relayer d610b378-01ca-4844-88ed-4aa7981b605d. Can you rest it as well please?
Kind regards
Just reset, failed transactions were deleted and nonce got back to what's on chain.
Best!
hi @ernestognw we're experiencing that same issue again and i'm unable to withdraw funds from that same relayer.
It has sent hundreds of cancel transactions as you can see in this link:
And I'm unable to withdraw funds as it shows this message:
When there are no pending transactions:
Hi @ernestognw and @dan_oz, can you take a look at my previous comment?
Hey @Mr.Peanutz
Sorry for the late response, we didn't get a notification since it was on an already solved thread.
Although we should also put an eye on these, next time it'll be more visible to us if you post it as a new topic.
Regarding your issue, seems like your relayer got stuck and started to send NOOPs because of the validUntil set (default to 8 hours). As you said, there are no pending transaction so I reset your relayer's nonce to the current address nonce and pending costs to 0.
You should be able to withdraw your funds without problem. Let me know if you need anything else.
Best
