Hardhat deploy on xDAI "TransactionMinedTimeout [Error]: Timed out waiting for transaction"

jacobrosenthal · January 31, 2021, 12:40am

Looks like Ive got a similar problem as

Were migrating into hardhat. Im using xdai if it matters.

local deploys work every time.
mainnet deploys get to same place before timing out each time.

The deploy script deploys some other contracts as part of the project, 2 of them also upgradable, and pre deployed 4 libraries, which are given to the third contract which fails with a txhash that doesnt exist....


  const DC = await ethers.getContractFactory("DC", {
    libraries: {
      a: a.address,
      b: b.address,
      c: c.address,
      d: d.address,
    },
  });




  const dC = await upgrades.deployProxy(
    DC,
    [init1, init2, init3, init4],
    { unsafeAllowCustomTypes: true, unsafeAllowLinkedLibraries: true }
  );// <-time outs here

    You are using the `unsafeAllowLinkedLibraries` flag to include external libraries.
    Make sure you have manually checked that the linked libraries are upgrade safe.

TransactionMinedTimeout [Error]: Timed out waiting for transaction 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49
    at Object.waitAndValidateDeployment (/contracts/eth/node_modules/@openzeppelin/upgrades-core/src/deployment.ts:61:15)
    at fetchOrDeployGeneric (/contracts/eth/node_modules/@openzeppelin/upgrades-core/src/impl-store.ts:42:5)
    at Proxy.deployProxy (/contracts/eth/node_modules/@openzeppelin/hardhat-upgrades/src/deploy-proxy.ts:45:18) {
  deployment: {
    address: '0xE86FB472D997bD489EEd806f6b138518d5473De0',
    txHash: '0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49',
    layout: { storage: [Array], types: [Object] }
  }
}

    "@nomiclabs/hardhat-ethers": "^2.0.1",
    "@nomiclabs/hardhat-waffle": "^2.0.1",
    "@openzeppelin/contracts-upgradeable": "^3.3.0",
    "@openzeppelin/hardhat-upgrades": "^1.5.0",
    "hardhat": "^2.0.7",
    "hardhat-typechain": "^0.3.4",

jacobrosenthal · February 1, 2021, 7:25pm

Got the DEBUG enabled but mostly says same thing, its looking for txhash that it thinks it submitted but doesnt exist?

    You are using the `unsafeAllowLinkedLibraries` flag to include external libraries.
    Make sure you have manually checked that the linked libraries are upgrade safe.

  @openzeppelin:upgrades:core fetching deployment of implementation f3b04f614a3a1e805085ed0d5729854e07fa045ae4fbaf7bd7a23d8ca77e0e3a +53s
  @openzeppelin:upgrades:core deployment of implementation f3b04f614a3a1e805085ed0d5729854e07fa045ae4fbaf7bd7a23d8ca77e0e3a not found +2ms

  @openzeppelin:upgrades:core initiated deployment 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49 +1m
  @openzeppelin:upgrades:core verifying deployment tx mined 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49 +4ms
  @openzeppelin:upgrades:core waiting for deployment tx mined 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49 +250ms
  
  @openzeppelin:upgrades:core verifying deployment tx mined 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49 +5s
  @openzeppelin:upgrades:core waiting for deployment tx mined 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49 +223ms
...
  @openzeppelin:upgrades:core verifying deployment tx mined 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49 +5s
  @openzeppelin:upgrades:core waiting for deployment tx mined 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49 +221ms
TransactionMinedTimeout [Error]: Timed out waiting for transaction 0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49
    at Object.waitAndValidateDeployment (/contracts/eth/node_modules/@openzeppelin/upgrades-core/src/deployment.ts:61:15)
    at fetchOrDeployGeneric (/contracts/eth/node_modules/@openzeppelin/upgrades-core/src/impl-store.ts:42:5)
    at Proxy.deployProxy (/contracts/eth/node_modules/@openzeppelin/hardhat-upgrades/src/deploy-proxy.ts:45:18) {
  deployment: {
    address: '0xE86FB472D997bD489EEd806f6b138518d5473De0',
    txHash: '0x2035336a41b178ec16035e93621acd3f0107ec20e877ff4fc2215d909c138f49',
    layout: { storage: [Array], types: [Object] }
  }
}

abcoathup · February 2, 2021, 5:53am

Hi @jacobrosenthal,

Welcome to the community

I was able to deploy a simple upgradeable contract to xDAI: OpenZeppelin Upgrades: Step by Step Tutorial for Hardhat so assume it is the linked libraries which results in the timeout of 60 seconds.

github.com

OpenZeppelin/openzeppelin-upgrades/blob/master/packages/core/src/deployment.ts#L61


// Poll for 60 seconds with a 5 second poll interval.
// TODO: Make these parameters configurable.
const pollTimeout = 60e3;
const pollInterval = 5e3;
// eslint-disable-next-line no-constant-condition
while (true) {
  const elapsedTime = Date.now() - startTime;
  if (elapsedTime >= pollTimeout) {
    // A timeout is NOT an InvalidDeployment
    throw new TransactionMinedTimeout(deployment);
  }
  debug('verifying deployment tx mined', txHash);
  const tx = await getTransactionByHash(provider, txHash);
  if (tx?.blockHash !== null && tx?.blockHash !== undefined) {
    debug('succeeded verifying deployment tx mined', txHash);
    break;
  }
  debug('waiting for deployment tx mined', txHash);
  await sleep(pollInterval);
}

Can you try updating to the latest version of @openzeppelin/hardhat-upgrades in case that resolves the issue. (You are using 1.5.0 and the latest is 1.6.0).

jacobrosenthal · February 2, 2021, 4:40pm

Tried. no change.

Seems to be a bug in the “deployment of implementation” “not found” path.
Thats the only time that path is hit in the logs, and the one that has a nonexistant txhash.

martriay · February 2, 2021, 4:45pm

How are you connecting to xdai?

jacobrosenthal · February 2, 2021, 5:01pm

https://rpc.xdaichain.com

frangio · February 2, 2021, 6:46pm

The contract at the address seems to have been created, but the transaction hash is a different one.

https://blockscout.com/poa/xdai/address/0xE86FB472D997bD489EEd806f6b138518d5473De0

I don’t really know what could be going wrong here. We obtain the deployment transaction hash from ethers.js:

const instance = await Contract.deploy(...);
instance.deployTransaction;

abcoathup · February 3, 2021, 4:30am

Hi @jacobrosenthal,

I’d like to be able to reproduce the issue. I am not familiar with how libraries are linked in ethers.js and I couldn’t find an example in the ethers.js documentation.

Are you able to create a simple example that reproduces the problem?

jacobrosenthal · February 3, 2021, 5:08pm

Thanks. I wrote up a github issue. Let me know if that shows linking enough.

martriay · February 3, 2021, 7:07pm

I believe @abcoathup was asking for code to reproduce the issue. The code you shared in github issue is not executable as is, there’s no contracts and the script code is incomplete. Bear in mind that “reproducible code” is meant to be executed, not read.

In order for you to get the best help in the shortest time possible, I suggest you make this as simple for other people as possible: either by simplifying your script and adding dummy contracts or directly setting up a full working repo.

abcoathup · February 5, 2021, 3:25am

Hi @jacobrosenthal,

I wasn’t able to reproduce with a simple example.
I found how to link libraries in Hardhat: https://hardhat.org/plugins/nomiclabs-hardhat-ethers.html#library-linking

I got the same timeout using your code.

I updated the issue with this information.

frangio · February 5, 2021, 11:45pm

@jacobrosenthal I was able to deploy your contracts using your deploy:prod script.

I did get a timeout when deploying DarkForestTokens, but in my case it was apparently just a timeout, and I worked around it by rerunning the script, with modifications to avoid redeploying what was already on chain. The key is that if you rerun the same deployProxy or upgradeProxy function call, it will resume where it had previously timed out. You can wrap deployProxy around a try..catch block to rerun it if it times out until it succeeds. (We can make the timeout configurable but this is a quicker workaround you can implement.)

I’m still really surprised that the contract address that you shared was found on chain but deployed with a different transaction hash than the plugin had observed. Since we are seeing all of these other deployments on xDai succeed, I’m thinking this was an outlier error that must have been caused by an unrelated reason, maybe the transaction was replaced by one with a higher fee or something?

To sum up, I believe the timeout that you are seeing can be worked around by using try..catch to rerun deployProxy, which will resume waiting for the already sent transaction that caused the timeout.

jacobrosenthal · February 8, 2021, 12:17am

Ive never seen a timeout on DarkForestTokens, mine was always the same error on the final Core deploy.

However trying again now (without try catches or any changes) I actually see it complete successfully-- but only 1 out of 4 times. I havent had the timeout with no tx hash anymore. now I usually get one of

hangs forever after deploying whitelist
ProviderError: Transaction gas price supplied is too low
ProviderError: Transaction nonce is too low

frangio · February 8, 2021, 6:28pm

The first two sound like they could be related to a low gas price. You should check what gas price your transactions are using.

I’m not sure what could be causing the third error. Did this show up in the middle of your deployment script or was it when you reran the script after a failed run? Perhaps there was a transaction pending in the mempool that cause the nonce to become out of sync.

jacobrosenthal · February 9, 2021, 7:28pm

I’m not specifying gas or gasLimit in hardhat config so they should be ‘auto’ assuming hardhat-upgrades is using those values. Ive also explicitly set those values to ‘auto’. I wonder about the code path for redeploy. Is it correctly looking at last nonce and gas and adjusting appropriately?

Im actaully trying to do a fresh deploy in this case so I dont care to recover some old deploy, so I completely deleted the .openzeppelin directory and was getting the timeouts now again all of a sudden, so Im using npm edit @openzeppelin/upgrades-core to edit src/deployment.ts.

-    const pollTimeout = 60e3;
+    const pollTimeout = 160e3;
-    const pollInterval = 5e3;
+   const pollInterval = 1e3;

And now Ive seen my deploy mostly work.

Others trying it seem to be at 100% without any changes. But I randomly Ill still see it hang forever. I’m known to have pretty bad internet here so I think theres definitely something that could be done around retry logic, timeouts.

romans · November 1, 2021, 10:53am

Same issue here. But the problem was mainly, because my node I was connecting to was slow in respond. So, the first TX on Binance Smart Chain went through and then the poll timeout hit. 20 USd down the drain.

IT would be nice to have configurable timeouts

Onigiri · January 5, 2022, 6:41pm

Hi there following up on this

@jacobrosenthal if you could share more how you used npm edit to solve this? If you can suggest instructions in yarn I would appreciate that. I think the 60 second limit is too short in most cases.

Can this be made configurable?

I am having issues right now putting any upgrades through on the polygon network that has recently been congested by bots. Upgrades have been fine for me always in the past.

Thanks

frangio · January 5, 2022, 7:52pm

The issue reported originally in this thread was not a true timeout, it was that the transaction hash the plugin was looking at was somehow wrong.

You can check if the transaction hash is correct by running in debug mode with env DEBUG='*' <your script>.

If the transaction exists but it just hasn't mined yet, you can re-run the script or the deployProxy function call to resume waiting for the transaction to mine.

Currently the timeout delay is not configurable but it is in the roadmap for soon.

ericglau · January 31, 2022, 7:29pm

Following up on the above, the timeout delay is now configurable in the latest release of the Hardhat Upgrades plugin by using the timeout and pollingInterval options. For more details, see Common Options.

Topic		Replies	Views
Deployment tx timed out, deployed implementation but not proxy Upgrades	4	3108	September 15, 2021
TransactionMinedTimeout [Error]: Timed out waiting for transaction Upgrades proxies , hardhat	4	2103	October 4, 2022
Timed out waiting for implementation contract deployment [RINKEBY] Upgrades etherscan-verify	2	2891	April 15, 2022
Deploying Upgradable Contract Polygon TransactionMinedTimeout [Error]: Timed out waiting for transaction Upgrades	2	2006	October 14, 2021
Timed out waiting for existing implementation Upgrades	11	1307	June 20, 2024

Hardhat deploy on xDAI "TransactionMinedTimeout [Error]: Timed out waiting for transaction"

Related topics