Hi, we're having issues with our BSC relayer and autotasks. We're not 100% sure if it's solely related to BSC but relayers on other chains running the same code works fine. Below are the events that occurred:
Our cron autotask failed with this weird error:
{"errorType":"Error","errorMessage":"Error while attempting request: error processing the request: Error parsing relay: LoadBalancer not found","stack":["Error: Error while attempting request: error processing the request: Error parsing relay: LoadBalancer not found"," .
Unfortunately I don't have the exact error message and logs because I deleted and recreated the autotask in an attempt to fix this as described in the next steps.
Deleting and recreating the autotask did not work, it resulted in the same error. So I deleted and recreated the relayer, which was created using the "Clone to another network" feature. I then recreated the autotask again to have a completely clean start.
Now the errors are gone, and the autotasks complete with example logs shown below:
AUTOTASK START
<truncated>
END RequestId: 3c910dd4-7309-4e06-b942-f2dc52f0443f
AUTOTASK COMPLETE
However, the weird thing is the defender on-chain never actually writes anything. Perhaps there is some issue with this last leg where the lambda tries to write to chain and is silently failing? Faced the same issues for multiple autotasks so any help would be appreciated, and happy to provide any additional info needed.
Unfortunately not, going to try again to see if anything has resolved over the weekend. Will try to think of more things to try but I think the issue might be on the OZ side so not sure what else we can do actually . Would anyone from the team mind looking into or providing an update for this, perhaps @emnul?
However, when I enter the tx 0xc8ca14e1c98224359dfac14160d22a73c7d6e8ef4815373f56747fd5f92b4790 based on the screenshot into bscscan, it is unable to find the transaction.
I also tested changing the autotask to just read a value from the chain, and it was able to do so successfully.
@emnul I'm also seeing it with autotask ID: 5aa44823-23eb-4980-8297-c303df4d745e and tenant ID: 42343d55-60ec-4eab-8246-b1002c9451ee. All my BSC autotasks are having this issue. 90% of the time they fail with this error. The occasional one does successfully run and nothing has changed in our code.
We apologize for the slow response. We had a large backlog of other support requests which were blocking us, but I will personally investigate this issue today and try to find a fix
@Sahil_Gathani I'm seeing in at least one Autotask run that tenantId: 6a48486c-bc44-410a-85af-4b03c9394a1f and autotaskId: 4ee85d9a-2e59-4473-99d0-f2086eb00051 throws the following error
{ "errorType": "Error", "errorMessage": "overflow [ See: https://links.ethers.org/v5-errors-NUMERIC_FAULT-overflow ] (fault=\"overflow\", operation=\"BigNumber.from\", value=36893488147421470000, code=NUMERIC_FAULT, version=bignumber/5.6.2)", "code": "NUMERIC_FAULT", "reason": "overflow", "fault": "overflow", "operation": "BigNumber.from", "value": 36893488147421470000, "stack": [ "Error: overflow [ See: https://links.ethers.org/v5-errors-NUMERIC_FAULT-overflow ] (fault=\"overflow\", operation=\"BigNumber.from\", value=36893488147421470000, code=NUMERIC_FAULT, version=bignumber/5.6.2)", " at c.makeError (/var/task/index.js:2:36559)", " at c.throwError (/var/task/index.js:2:36679)", " at y (/var/task/index.js:2:12767)", " at Function.from (/var/task/index.js:2:11649)", " at C.encode (/var/task/index.js:2:1114242)", " at /var/task/index.js:2:1111154", " at Array.forEach (<anonymous>)", " at v (/var/task/index.js:2:1111024)", " at k.encode (/var/task/index.js:2:1115415)", " at Object.encode (/var/task/index.js:2:1130022)" ] }
@Sahil_Gathani@hotpocket we discovered that our internal pokt JSON-RPC provider was down and have taken action to pause it. This should resolve both of your issues with the Load Balancer error (it's a pokt network request error). Please let me know if this has resolved your issues.
Hi, still not working. I tried it the first time and got the same issue as before where defender says the autotask was successful but nothing is written to chain. However, trying it a second time resulted in a new error and the autotask failing.
Error
Error while attempting request: Quota of 120 for unconfirmed transactions exceeded
Logs
AUTOTASK START
(Truncated function logs)
END RequestId: 70254021-b729-41ae-81c5-4644de16df56
AUTOTASK COMPLETE
2023-07-17T06:50:05.484Z ERROR Invoke Error {"errorType":"Error","errorMessage":"Error while attempting request: Quota of 120 for unconfirmed transactions exceeded","stack":["Error: Error while attempting request: Quota of 120 for unconfirmed transactions exceeded"," at i.execute (/var/task/index.js:2:483148)"," at processTicksAndRejections (node:internal/process/task_queues:96:5)"," at async m.sendTransaction (/var/task/index.js:2:513116)"]}
I also saw some very weird logs on the BNB chain itself from my Relayer when the autotask failed - a whole bunch of self-transfers as per the screenshot below: