Defender Relay not broadcasting txs

Support Defender
#1

Hello team,

We are experiencing an issue across all of our relayers where in a transaction is submitted to Defender Relay but is not broadcasted to the blockchain.

On the Defender UI logging section, we see that these transactions are marked stale and Defender is trying to re-send them.

This issue emerged ~ 1.5 hours ago. Can you please help us ASAP? We're using relayers on Optimism, Arbitrum, Gnosis Chain, Avalanche, Aurora, Polygon, Fantom and BSC

cc : @ernestognw @ericglau @bjmrq

Arbitrum testnet relayer transactions not relayed to blockchain
#2

It seems we have a similar problem on Arbitrum testnet.

#3

Hi @ShreyKeny

Very sorry for these issues.

We identified an issue that impacted Relayers earlier today regarding our providers. It should now be resolved.

@Archethect_Archethec - I saw the other forum issues you submitted. If there are persistent issues with an Arbitrum relayer, could you please send more information about the relayer impacted such as:

URL of the page on Defender for that Relayer

or

The Relayer's address

Re: RPC endpoints, we use multiple providers to handle failover and the public JSON RPC endpoint you referenced is one of them

Dan McKeon
OpenZeppelin

1 Like
#4

Hey Dan,

Thanks for the follow up. The relaying seems to be fixed, but we still get a very strange error on one of our transactions. Will take this up in the other ticket.

Kind regards,

Archethect

1 Like
#5

Hey @dan_oz,

Thanks for your help! All of our relayers are working normally

#6

Hi, we're experiencing the same issue here.

Our BSC relay is unable to send transactions as it got 120 stuck transactions due to this issue.

image
image1380×546 23.5 KB

I'm unable to cancel the transactions and also unable to withdraw the funds from this relayer, probably due to an issue with the nonces.

#7

Hey @dan_oz any idea on how to fix it?

#8

Same issue here
Relayers...
50a60c70-9ca2-47c6-9bb4-c417c99dfbe3
95d16afa-ccc8-4fd3-936f-5ed7683257a4

#9

My relay address is: 0xb10899410f73bc2516371e4a524441c936b45c49

#10

Hey @Mr.Peanutz, thanks for letting us know. I've checked on your relayer and seems like you hit a condition that's very particular:

  • The transactions gasPrice you're sending is 5gwei, which is also the gasPriceCap you seem to have set on the relayer policies
  • Any resubmission attempt on our end ends up surpassing the gas price cap you set, so we don't resubmit it.

What I can do is run a script to clear your relayer avoiding the gasPrice cap, or you can remove it. However, we understand this is not the expected behavior that a relayer with a cap should have, so, I have to get back to the team and discuss.

Curious, do you think we should avoid the cap when we're sending a NOOP? (expired or cancelled transaction).

Best

#11

Hey @amaurer-rise.
I notice both of your relayers are already cleared.
Is everything correct? I'd like to know how much the relayer was stuck, so I can send a fix if there's some misbehavior.

Best!

#12

Hi @ernestognw, thanks for explaining, so isn't there any way to cancel the transactions without broadcasting them to the blockchain?

Anyways, if possible i'd like you to run that script so i wouldnt have to clear it out manually.

And yes, I think it should be ignored since it would be replacing a previous transaction

#13

Yeah, there's a way to try that. I can delete every pending transaction we have on Defender and reset your Relayer's nonce, so you can start sending transactions again.

It's worth noting that it's almost certain that the nodes have dropped previous transactions, but unfortunately we can't guarantee that. In such case, you'll have your relayer stuck trying to send cheaper transactions than the ones already sent. But, to be honest, I really doubt any node kept the old txs.

Are you okay with me deleting all of the stuck transactions and reset your Relayer's nonce?

Best

#14

Yes I'm ok with deleting the transactions, thanks @ernestognw!

#15

Done, I deleted every transaction on our side, reset your Relayer's nonce and set pending costs to 0, so your relayer should behave normally now.

Best!

1 Like
#16

@ernestognw, we seem to have the same problem for relayer d610b378-01ca-4844-88ed-4aa7981b605d. Can you rest it as well please?

Kind regards

#17

Hey @Archethect_Archethec.

Just reset, failed transactions were deleted and nonce got back to what's on chain.
Best!

#18

hi @ernestognw we're experiencing that same issue again and i'm unable to withdraw funds from that same relayer.

It has sent hundreds of cancel transactions as you can see in this link:

And I'm unable to withdraw funds as it shows this message:

image
image1092×661 23.6 KB

When there are no pending transactions:

image
image1632×811 36.1 KB

#19

Hi @ernestognw and @dan_oz, can you take a look at my previous comment?

#20

Hey @Mr.Peanutz

Sorry for the late response, we didn't get a notification since it was on an already solved thread.
Although we should also put an eye on these, next time it'll be more visible to us if you post it as a new topic.

Regarding your issue, seems like your relayer got stuck and started to send NOOPs because of the validUntil set (default to 8 hours). As you said, there are no pending transaction so I reset your relayer's nonce to the current address nonce and pending costs to 0.

You should be able to withdraw your funds without problem. Let me know if you need anything else.

Best