On August 21st, Immunefi informed our team of a critical vulnerability in the TimelockController contract in OpenZeppelin Contracts, our open source smart contracts library. In this document, we will share an overview of the affected smart contract, the technical details of the vulnerability, and our assessment of the issue. We will also provide transparency regarding the development process for all OpenZeppelin Contracts, lessons learned, and next steps to further harden the security of the library. Of the projects potentially exposed, we determined — to the best of our ability — that no critical instance of the vulnerability contained assets at risk.
We want to thank the whitehat Zb3 who identified the issue, as well as the Immunefi team, who reached out to us with the disclosure. OpenZeppelin rewarded the whitehat $25,000 for the disclosure. We encourage any affected projects to contact Immunefi and reward Zb3 for their work. We also want to thank the Dedaub and Tenderly teams, who assisted us in identifying affected projects, as well as Etherscan and TheGraph, which proved key to navigating the deployed instances of the TimelockController smart contract.
TimelockController
The TimelockController (Timelock), introduced in OpenZeppelin Contracts 3.3 last November, is a smart contract that enforces a delay on all actions directed towards an owned contract. A typical setup is to position the TimelockController as the admin of an application smart contract so, whenever a privileged action is to be executed, it has to wait for a certain time specified by the Timelock.
The security benefits of the Timelock are twofold. Firstly, it provides an extra layer of security to a project’s team by giving a heads up on every privileged action anticipated in the system. This allows the team to detect and react to malicious calls by compromised admin accounts. Secondly, it protects the community from the project’s governance itself, allowing members to exit the protocol if they disagree with any impending changes.
At its core, the TimelockController has two main public functions: First, a schedule function, callable by accounts with the PROPOSER role where a new action is enqueued; Second, an execute function, callable by accounts with the EXECUTOR role where after an enforced delay period, a previously scheduled action will run. The code below helps illustrate the point:
function schedule(
address target, uint256 value, bytes calldata data, bytes32 predecessor, bytes32 salt, uint256 delay
) public virtual onlyRole(PROPOSER_ROLE) {
bytes32 id = hashOperation(target, value, data, predecessor, salt);
require(!isOperation(id));
require(delay >= getMinDelay());
_timestamps[id] = block.timestamp + delay;
emit CallScheduled(id, 0, target, value, data, predecessor, delay);
}
function execute(
address target, uint256 value, bytes calldata data, bytes32 predecessor, bytes32 salt
) public payable virtual onlyRoleOrOpenRole(EXECUTOR_ROLE) {
bytes32 id = hashOperation(target, value, data, predecessor, salt);
require(predecessor == bytes32(0) || isOperationDone(predecessor));
(bool success, ) = target.call{value: value}(data);
require(success);
emit CallExecuted(id, 0, target, value, data);
require(isOperationReady(id));
_timestamps[id] = _DONE_TIMESTAMP;
}
Additionally, these functions come in a scheduleBatch and executeBatch flavor, allowing the caller to enqueue and execute proposals that run multiple calls in sequence.
Finally, the Timelock also has an ADMIN role, used for managing the other roles. This role is usually assigned to the Timelock smart contract itself, making it self-governed. Self-governance also applies to tweaking the minimum enforced delay. Thus, for any administrative change on the Timelock itself, users need to schedule a proposal and then execute it.
In a typical setup, the Timelock is set as an owner of a set of application contracts, and a multisig or governor is set as the PROPOSER, since it is the proposer who dictates the privileged actions to be run through the Timelock. The EXECUTOR role can be set to either the same account as the PROPOSER or to a set of EOAs, as its sole responsibility is to trigger the actions already defined by the PROPOSER. The Timelock also provides the option to leave the EXECUTOR role open, allowing anyone to trigger a scheduled action after its delay has passed.
Vulnerability
When combined with the fact that the Timelock is self-governed, the vulnerability is enabled by a reentrancy in the executeBatch method allowing any account with the EXECUTOR role to set themselves as PROPOSER and ADMIN of the Timelock, and to set the minimum delay to zero. This allows the attacker to execute instantaneous changes to the protocol and grants the attacker immediate access to all assets controlled by the TimelockController.
For context, the executeBatch method looks like the following (modified for clarity):
function executeBatch(
address[] calldata targets, uint256[] calldata values, bytes[] calldata datas, bytes32 predecessor, bytes32 salt
) public payable virtual onlyRoleOrOpenRole(EXECUTOR_ROLE) {
require(targets.length == values.length);
require(targets.length == datas.length);
bytes32 id = hashOperationBatch(targets, values, datas, predecessor, salt);
require(predecessor == bytes32(0) || isOperationDone(predecessor));
for (uint256 i = 0; i < targets.length; ++i) {
_call(id, i, targets[i], values[i], datas[i]);
}
require(isOperationReady(id));
_timestamps[id] = _DONE_TIMESTAMP;
}
Note, the isOperationReady check is done after the calls in the batch, so if someone tries to execute a batch that is not ready, the require statement would just revert all calls done in the batch. However, the logic was flawed: as part of its execution, an unprepared batch could schedule itself and appear ready by the end.
A malicious EXECUTOR could submit a batch that calls the Timelock itself to clear the minimum delay and grant PROPOSER rights to an address under their control. The EXECUTOR could then use this new PROPOSER to schedule the batch being executed with a delay of zero.
In this scenario, the isOperationReady check passes when the set of calls finishes since the last operation scheduled the batch to be available for immediate execution. The malicious EXECUTOR can extend additional grants in this batch, such as giving ADMIN and PROPOSER rights to themselves, allowing the attacker to revoke the rights of other members of the Timelock and instantly pass any proposals, effectively taking full control of the smart contract, as well as any assets owned by it.
The fix is simple: Add an additional isOperationReady check before the batch begins execution. Note, this check is still needed after the batch ends execution, otherwise an EXECUTOR could execute a batch multiple times via reentrancy.
Threat Assessment
The TimelockController, as mentioned, fulfils two roles in a system’s security: Protecting the team from compromised admins and protecting the community from actions performed by the project’s team.
Regarding the latter, since this vulnerability reduces the minimum delay to zero without a waiting period, communities can no longer trust this version of the Timelock to protect them since a malicious project could exploit the vulnerability, using it against their community.
Regarding the threat to each project and their corresponding teams: We identified four categories of varying severity that depend on roles assignment in the Timelock contract:
-
Critical. If the
EXECUTORrole is open, then anyone can use this vulnerability to hijack the TimelockController and have control over the project assets. -
High. If there is an untrusted address with
EXECUTORrights, such as an externally owned account with a private key that is not properly secured, anyone with access to that account can hijack the Timelock. For the purpose of this assessment, we consider anEXECUTORto be untrusted if they don’t havePROPOSERrights as well, sincePROPOSERs are typically closely guarded accounts. We then consider a Timelock to be in this tier if there is anyEXECUTORaddress that doesn’t also have PROPOSER rights. -
Medium. Even if all
EXECUTORaccounts are trusted, the vulnerability allows a single one of them to escalate toADMINprivileges and revoke access to the others. As such, we consider all Timelocks with more than oneEXECUTORto be at least in this tier. - Low. Timelocks who have only a single proposer/executor are in the lowest risk tier. The vulnerability here poses no risk to project owners, only to the community members.
We used this categorization to guide our course of action during the disclosure process. Additionally, our assessment clarified necessary and immediate mitigation for the vulnerability: Revoke access to all untrusted EXECUTOR accounts in the Timelock contract.
Events Timeline
We received first notice of the vulnerability from the Immunefi team on August 21st, 2021 at 20:00 UTC. Immunefi was initially alerted of the vulnerability via a whitehat, who identified one project in the critical tier. Immunefi reached out to them because this project had an open bounty with them. The project’s team then used the vulnerability to immediately revoke access to the public EXECUTOR without going through the minimum enforced delay. Later, after surveying multiple networks in the following days, we confirmed that the whitehat’s exploit was the only use of the vulnerability.
Following Immunefi’s alert, we proceeded to identify all instances of the TimelockController deployed in production networks. The Dedaub and Tenderly teams also provided assistance. Our team managed to identify 316 instances deployed across Ethereum Mainnet, Polygon, Binance Smart Chain, Fantom, and Avalanche C-Chain. We found no instances on xDAI. We then categorized the findings into the four severity tiers, yielding 9 criticals, 38 highs, 33 mediums, and 236 lows.
We reviewed the Timelocks for potential assets at risk for each of the 47 critical and high instances. We found no instances of critical Timelocks with assets at risk, so we discarded the need to exploit the vulnerability ourselves to forcefully revoke public executor rights.
Next, we proceeded to manually link the 47 high and critical instances to known projects by inspecting deployer accounts, proposers, and owned assets. We successfully identified 20 projects by name.
On August 24th at 18:00 UTC, we reached out to the identified projects to alert them of a vulnerability in the Timelock smart contract and recommend they revoke access to all untrusted EXECUTORS. We also provided a guide for Defender users and a script for facilitating this task.
At this time, we did not disclose the vulnerability nor its fix. We wanted to ensure as many projects as possible moved to the low-risk tier before disclosing the fix in order to prevent anyone from reverse engineering the vulnerability.
On August 26th at 21:00 UTC, we made the fix public along with a security advisory. The fix was applied to all major versions of OpenZeppelin Contracts affected (4.3.x, 3.4.x, and a custom build of 3.4.x for Solidity 0.7), as well as the upgradeable fork of the library.
The entire Contracts development team reviewed the fix, in addition to multiple OpenZeppelin security researchers. As an added precaution, we had our audit team review other smart contracts in the library to determine if we missed similar vulnerabilities.
Development Process
We invest in OpenZeppelin Contracts, our open source smart contract library, as a public good for the open economy. We maintain it and provide details about our development process here below to improve the security of the ecosystem and transparency in our methods.
Every change to our smart contract library requires a review by at least one other maintainer in our team. All changes must also be thoroughly tested, and line coverage is tracked for each PR. The project stands at 98.52% coverage, and the TimelockController is at 100%. Any major changes undergo an internal review by our security auditors. Furthermore, we encourage our community to review our work by sharing a release candidate for each new version a few weeks before release.
Lessons Learned & Next Steps
While we dedicate considerable effort to security in our open source smart contract libraries, the industry has shown vulnerabilities happen. In this case, the Timelock vulnerability gave us an opportunity to strengthen aspects of our internal review of Contracts.
-
Going forward, we will increase the number of required reviewers for major changes in the library. Additionally, we will pay special attention to any complex functions that do not strictly follow the check-effects-interaction pattern, as in the case of Timelock.
-
We will also review event emissions across the library to ensure we can identify specific smart contracts if necessary. Recognizable emissions from TimelockController used in deployment allowed us to identify instances of the specific smart contract in production.
-
The ecosystem might consider mapping deployed addresses to projects in a way that can be consumed programmatically so appropriate teams can be alerted to issues when they arise. We also encourage all projects to set up and clearly communicate a security email address.