Test smart contracts like a rockstar

One does not simply test your dapp.

Among many other things while developing a decentralized application, testing it is something you always need to do. What most people still donโ€™t do is automate those tests, which leads to a huge loss of time and drains the energy to code.

Automated tests have been around for a while and they have been gaining lots of traction in the last decade. Whilst blockchain and smart contract development are fairly new, we can still automate tests for these. We can, so we actually should. Letโ€™s do it!

Tools

This tutorial is supposed to teach you the techniques, not to โ€œsellโ€ you on any particular tool, meaning, we are going to see how to test smart contracts with some specific tools, but itโ€™s up to you to decide what tools you use and how to use them.

In this tutorial we are going to use ganache-cli, Truffle, solidity-coverage, GitHub, Travis and coveralls.

Basic testing

Consider the following contract:

pragma solidity ^0.5.0;

contract SimpleStorage {
    uint256 private _storedData;

    function set(uint256 x) public {
        _storedData = x;
    }

    function get() public view returns (uint256) {
        return _storedData;
    }
}

This SimpleStorage contract has two functions, set and get. For someone who has never written tests before, it might sound confusing, but testing a function is actually as simple as calling it. So imagine, you are building a UI and connecting it to this contract. It should be as simple as:

SimpleStorage storage = SimpleStorage.at("0x989fedfsfsd...");
await storage.get();

Writing tests should be as simple as that too, just calling functions.

But in order to make all of this work, you need to start a private network (private testnet for running your tests), setup the connection to that network and then test. As a starting point, we are going to use Truffle, which allows us to build smart contracts and tests much faster and simpler. Truffle will connect to a network, inject web3 and the compiled contracts (called artifacts) and then run the tests.

We will only use a local instance of Truffle (no global installs), so all commands will start with npx. First, create a folder (name it your projectโ€™s name e.g. test-tutorial) then, within that folder initialize your project by running npm init -y then install Truffle locally with npm install --save-dev truffle and finally initialize truffle for your project by running npx truffle init.

This will create a few files and folders. You will now have truffle-config.js, a contracts folder, a migrations folder and a test folder (if you donโ€™t see a test folder, create one). To learn more about this folder/file structure, please see the Truffle documentation: https://www.trufflesuite.com/docs/truffle/getting-started/creating-a-project.

Now, create the contract shown above in the contracts folder, then create a migrations script to deploy the contract. Name the migrations script 2_deploy.js and save it in the migrations folder.

const SimpleStorage = artifacts.require('SimpleStorage');
module.exports = function (deployer) {
  deployer.deploy(SimpleStorage);
};

Once you have created the contract and the migrations script then create a file named SimpleStorage.test.js inside the test folder with the code below and then run npx truffle test.

const SimpleStorage = artifacts.require('SimpleStorage');
contract('SimpleStorage', (accounts) => {
    it('should store a value', async () => {
        const simpleStorageInstance = await SimpleStorage.deployed();
        // Set value of 89
        await simpleStorageInstance.set(89, { from: accounts[0] });
        // Get stored value
        const storedData = await simpleStorageInstance.get();
        assert.equal(storedData, 89, 'The value 89 was not stored.');
    });
});

This is the simplest test for a smart contract. The test gets the artifact for the SimpleStorage contract, then starts the contract test, which injects the web3 environment, and then runs the tests.

Note that you donโ€™t need to specify the contract address, thanks to the .deployed() function, which returns an instance of the Truffle deployed contract, so you can just use it like an object, calling the functions.

Coverage

Automated testing makes you feel better, no matter if you are a smart contract or a frontend developer. But when the codebase gets bigger, how do you know how much code was tested?

The trick is to have a coverage tool, but donโ€™t get me wrong, remember, this should not be a guidance tool used before you write your tests. What I mean is, use the tool to help you to know what conditions were not tested. For example, some of the time, we have some ifโ€™s in our code that might not make sense, and we only realize that when looking at the coverage report, or when we already have 30 tests for a function, but we forgot to test one else condition. Going deeper, coverage helps us to know how many times a given line of code was executed in the tests.

The package used in this tutorial is solidity-coverage (https://github.com/sc-forks/solidity-coverage). First install the package (npm install --save-dev solidity-coverage@beta) and add solidity-coverage as a truffle plugin to truffle-config.js

plugins: ["solidity-coverage"]

Then run npx truffle run coverage to generate a test coverage report.

Configure solidity-coverage

Looking at the solidity-coverage repository options section (https://github.com/sc-forks/solidity-coverage#options), itโ€™s possible to create a .solcover.js file to configure solidity-coverage, such as ignoring some smart contracts.

Itโ€™s recommended to have the same configuration across all machines, so tests or coverage arenโ€™t failing just because a command is running with incorrect parameters.

You donโ€™t need to include coverage reports in version control, so you can add the following to your .gitignore file.

# Coverage directory used by tools like istanbul
coverage
coverage.json
coverageEnv

Continuous Integration

Whilst local test coverage reports are great, it becomes really powerful when used with a Continuous Integration (CI) system. A CI system can run things like tests and coverage when you push commits or merge branches.

An example of a CI tool is https://travis-ci.org/. Travis is very easy to configure, you only need a .travis.yml and everytime you push to your remote GitHub branch it will start running.

The example below is a possible setup for .travis.yml.

dist: xenial
    language: node_js
node_js:
    - "lts/dubnium"
install:
    - npm install
script:
    - npm run test
    - npm run coverage

Finally, add two new npm scripts to your package.json, one for test and another for coverage, just like the example below:

"scripts": {
    "coverage": "npx truffle run coverage",
    "test": "truffle test"
},

Before pushing the code to your remote repository, remember to sign in to Travis (using GitHub), setup Travis and activate your repository, allowing Travis to run the tests and coverage on your repository. Then push your changes (including your .travis.yml file).

Wait for Travis to build your repository.

Cool, you now have a working CI system. You can also add a build status badge to your repository README.md, Travis provides markdown for the build status that you can copy to your README.md.

The only thing is, the results of the tests and coverage remain in the Travis console. It would be great if you can have those results in a tool like coveralls.

Export coverage

Coveralls (https://coveralls.io/) is a web service that helps you track your code coverage over time.

Itโ€™s actually very simple to export the coverage results from solidity-coverage to coveralls. Coveralls accepts lcov reports as input for coverage data, and solidity-coverage actually generates lcov reports. Coveralls integration with Travis is almost automatic, it is as simple as providing the coverage reports to coveralls.

Install the package (https://www.npmjs.com/package/coveralls) with npm install --save-dev coveralls

In your package.json change the coverage command from โ€œnpx truffle run coverageโ€ to โ€œnpx truffle run coverage && cat coverage/lcov.info | coverallsโ€.

Before pushing your changes, sign in to coveralls, setup your account, and add your repository to allow coveralls to publish reports about that repository.

You can add a coverage status badge to your repository README.md, coveralls provides markdown for the status that you can copy to your README.md.

Test Helpers (Optional)

OpenZeppelin includes JavaScript testing helpers for smart contract development, openzeppelin-test-helpers (https://github.com/OpenZeppelin/openzeppelin-test-helpers).

These test helpers include helpers such as expectRevert, expectEvent and time which can make testing for reverts, events and time much easier. These test helpers are used in the OpenZeppelin tests (https://github.com/OpenZeppelin/openzeppelin-solidity/tree/master/test)

Alternatively, I created my own Truffle test helpers called girino that you can use with chai (https://github.com/obernardovieira/girino) with syntax inspired by waffle.

Reporters (Optional)

You can use eth-gas-reporter to print an estimate of gas used per function, and the gas costs of deploying each contract.

Install npm install --save-dev eth-gas-reporter and add the following to your truffle-config.js:

 mocha: {
   reporter: 'eth-gas-reporter',
   reporterOptions : { excludeContracts: ['Migrations'] }
 },

When you run npx truffle test you get a gas report in your test results such as the following:

ยท---------------------------------------|----------------------------|-------------|----------------------------ยท
|  Solc version: 0.5.8+commit.23d335f2  ยท  Optimizer enabled: false  ยท  Runs: 200  ยท  Block limit: 6721975 gas  โ”‚
ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท
|  Methods                                                                                                      โ”‚
ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท
|  Contract             ยท  Method       ยท  Min         ยท  Max        ยท  Avg        ยท  # calls     ยท  eur (avg)  โ”‚
ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท
|  SimpleStorage        ยท  set          ยท           -  ยท          -  ยท      41684  ยท           1  ยท          -  โ”‚
ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท
|  Deployments                          ยท                                          ยท  % of limit  ยท             โ”‚
ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท|ยทยทยทยทยทยทยทยทยทยทยทยทยทยท
|  SimpleStorage                        ยท           -  ยท          -  ยท     105015  ยท       1.6 %  ยท          -  โ”‚
ยท---------------------------------------|--------------|-------------|-------------|--------------|-------------ยท

Controlled tests (Optional)

As you may know, Ethereum is deterministic, and sometimes during tests we want to use this to our advantage. This is the last step of this tutorial and it is not a requirement at all, use it only if you want or you need it.

By setting up specific accounts with specific addresses and a given amount of Ether and initial mnemonic, as Ethereum is deterministic, all tests should then always have exactly the same result. If you want to make use of it, this section is for you.

OpenZeppelin has a very good example of this approach. Looking at https://github.com/OpenZeppelin/openzeppelin-solidity/blob/master/scripts/test.sh we can see that the script defines accounts always with the same addresses and amounts of Ether, as seen after the comment

# We define 10 accounts with balance 1M ether, needed for high-value tests.

It also starts a local testnet (with ganache-cli) using specific flags for things like the specific addresses.

ganache-cli-coverage --emitFreeLogs true --allowUnlimitedContractSize true --gasLimit 0xfffffffffff --port "$ganache_port" "${accounts[@]}"

If you look down a bit more in test.sh, you will see that when running coverage in a CI system, it will upload the results to coveralls, as shown in the example before

if [ "$SOLIDITY_COVERAGE" = true ]; then
    npx solidity-coverage
    if [ "$CONTINUOUS_INTEGRATION" = true ]; then
        cat coverage/lcov.info | npx coveralls
    fi
else
    npx truffle test "$@"
fi

Conclusion

Weโ€™ve seen some of the commonly used tools for smart contract development, how to use them locally and how to setup a CI system. Again, there are alternate tools, use the ones that are right for you.

I hope you now have a better idea on how to test smart contracts like a rockstar.

Thanks to the zeppelin community and especially to @martriay who invited to write in the first place and @abcoathup to suggest the topic and improve my writing. Thank you.

Cheers.

7 Likes

Great article! It would be cool mentioning https://github.com/cgewecke/eth-gas-reporter as well.

3 Likes

Thank you. Itโ€™s actually a good idea, and I use it. Iโ€™ll update.

2 Likes

@krzkaczor check out the new section on Reporters:

1 Like

3 posts were split to a new topic: Testing with forked state from live network