BSC Relayer/Autotask not writing to chain

Hi, we're having issues with our BSC relayer and autotasks. We're not 100% sure if it's solely related to BSC but relayers on other chains running the same code works fine. Below are the events that occurred:

  1. Our cron autotask failed with this weird error:
{"errorType":"Error","errorMessage":"Error while attempting request: error processing the request: Error parsing relay: LoadBalancer not found","stack":["Error: Error while attempting request: error processing the request: Error parsing relay: LoadBalancer not found"," .

Unfortunately I don't have the exact error message and logs because I deleted and recreated the autotask in an attempt to fix this as described in the next steps.

  1. Deleting and recreating the autotask did not work, it resulted in the same error. So I deleted and recreated the relayer, which was created using the "Clone to another network" feature. I then recreated the autotask again to have a completely clean start.

  2. Now the errors are gone, and the autotasks complete with example logs shown below:

AUTOTASK START
<truncated>
END RequestId: 3c910dd4-7309-4e06-b942-f2dc52f0443f
AUTOTASK COMPLETE

However, the weird thing is the defender on-chain never actually writes anything. Perhaps there is some issue with this last leg where the lambda tries to write to chain and is silently failing? Faced the same issues for multiple autotasks so any help would be appreciated, and happy to provide any additional info needed.

1 Like

Hi there - I am having the exact same issue also. Have you found a resolution to this?

Unfortunately not, going to try again to see if anything has resolved over the weekend. Will try to think of more things to try but I think the issue might be on the OZ side so not sure what else we can do actually :slightly_frowning_face:. Would anyone from the team mind looking into or providing an update for this, perhaps @emnul?

Another update:

Tried writing to chain but still having the same the same issue. I logged the tx itself and got the following:

However, when I enter the tx 0xc8ca14e1c98224359dfac14160d22a73c7d6e8ef4815373f56747fd5f92b4790 based on the screenshot into bscscan, it is unable to find the transaction.

I also tested changing the autotask to just read a value from the chain, and it was able to do so successfully.

1 Like

Hi @Sahil_Gathani I'm sorry you're experiencing these issues. Could you please provide your tenantId and autotask ID so we can investigate further?

I'm having the same issue on bsc, I can't even send a transaction from the relayer dashboard

Hi emnul, thanks for your reply! The autotask ID is 4ee85d9a-2e59-4473-99d0-f2086eb00051, and the tenant ID is 6a48486c-bc44-410a-85af-4b03c9394a1f.

1 Like

@emnul I'm also seeing it with autotask ID: 5aa44823-23eb-4980-8297-c303df4d745e and tenant ID: 42343d55-60ec-4eab-8246-b1002c9451ee. All my BSC autotasks are having this issue. 90% of the time they fail with this error. The occasional one does successfully run and nothing has changed in our code.

Thank you for the additional information! We're working hard to figure out the root cause issue :muscle:

Hi all - any updates here? This has been actively disrupting our project for nearly a week now.

Really appreciate any help here.

We apologize for the slow response. We had a large backlog of other support requests which were blocking us, but I will personally investigate this issue today and try to find a fix

1 Like

@Sahil_Gathani I'm seeing in at least one Autotask run that tenantId: 6a48486c-bc44-410a-85af-4b03c9394a1f and autotaskId: 4ee85d9a-2e59-4473-99d0-f2086eb00051 throws the following error

{ "errorType": "Error", "errorMessage": "overflow [ See: https://links.ethers.org/v5-errors-NUMERIC_FAULT-overflow ] (fault=\"overflow\", operation=\"BigNumber.from\", value=36893488147421470000, code=NUMERIC_FAULT, version=bignumber/5.6.2)", "code": "NUMERIC_FAULT", "reason": "overflow", "fault": "overflow", "operation": "BigNumber.from", "value": 36893488147421470000, "stack": [ "Error: overflow [ See: https://links.ethers.org/v5-errors-NUMERIC_FAULT-overflow ] (fault=\"overflow\", operation=\"BigNumber.from\", value=36893488147421470000, code=NUMERIC_FAULT, version=bignumber/5.6.2)", " at c.makeError (/var/task/index.js:2:36559)", " at c.throwError (/var/task/index.js:2:36679)", " at y (/var/task/index.js:2:12767)", " at Function.from (/var/task/index.js:2:11649)", " at C.encode (/var/task/index.js:2:1114242)", " at /var/task/index.js:2:1111154", " at Array.forEach (<anonymous>)", " at v (/var/task/index.js:2:1111024)", " at k.encode (/var/task/index.js:2:1115415)", " at Object.encode (/var/task/index.js:2:1130022)" ] }

@Sahil_Gathani @hotpocket we discovered that our internal pokt JSON-RPC provider was down and have taken action to pause it. This should resolve both of your issues with the Load Balancer error (it's a pokt network request error). Please let me know if this has resolved your issues.

Thanks for this. I can confirm all my regular tasks are running again as expected.

Hi, still not working for me unfortunately. Below are the logs of the autotask I just ran, seems fine with no issues but nothing appears on-chain.

AUTOTASK START

END RequestId: 3143de6c-da38-40bd-b9dd-26d0e6f02937
AUTOTASK COMPLETE

@Sahil_Gathani could you please try to rerun the autotask? I believe I discovered the root cause of your issue and pushed a fix.

Hi, still not working. I tried it the first time and got the same issue as before where defender says the autotask was successful but nothing is written to chain. However, trying it a second time resulted in a new error and the autotask failing.

Error

Error while attempting request: Quota of 120 for unconfirmed transactions exceeded

Logs

AUTOTASK START
(Truncated function logs)
END RequestId: 70254021-b729-41ae-81c5-4644de16df56
AUTOTASK COMPLETE
2023-07-17T06:50:05.484Z ERROR Invoke Error {"errorType":"Error","errorMessage":"Error while attempting request: Quota of 120 for unconfirmed transactions exceeded","stack":["Error: Error while attempting request: Quota of 120 for unconfirmed transactions exceeded"," at i.execute (/var/task/index.js:2:483148)"," at processTicksAndRejections (node:internal/process/task_queues:96:5)"," at async m.sendTransaction (/var/task/index.js:2:513116)"]}

I also saw some very weird logs on the BNB chain itself from my Relayer when the autotask failed - a whole bunch of self-transfers as per the screenshot below:

Seems like something executed, but not the autotask as intended.

@emnul, Hi any update on this?

Hi @emnul, bumping this again. Been almost two weeks now since the original incident without a resolution :frowning:

Hi @emnul, any update on this? Really would appreciate a response as it's currently breaking our workflow.