One does not simply test your dapp.
Among many other things while developing a decentralized application, testing it is something you always need to do. What most people still don’t do is automate those tests, which leads to a huge loss of time and drains the energy to code.
Automated tests have been around for a while and they have been gaining lots of traction in the last decade. Whilst blockchain and smart contract development are fairly new, we can still automate tests for these. We can, so we actually should. Let’s do it!
Tools
This tutorial is supposed to teach you the techniques, not to “sell” you on any particular tool, meaning, we are going to see how to test smart contracts with some specific tools, but it’s up to you to decide what tools you use and how to use them.
In this tutorial we are going to use ganache-cli, Truffle, solidity-coverage, GitHub, Travis and coveralls.
As an alternative to truffle with fast test runner, see the OpenZeppelinTest Environment section.
There are also other test framework tools, such as waffle with buidler.
Basic testing
Consider the following contract:
pragma solidity ^0.5.0;
contract SimpleStorage {
uint256 private _storedData;
function set(uint256 x) public {
_storedData = x;
}
function get() public view returns (uint256) {
return _storedData;
}
}
This SimpleStorage contract has two functions, set and get. For someone who has never written tests before, it might sound confusing, but testing a function is actually as simple as calling it. So imagine, you are building a UI and connecting it to this contract. It should be as simple as:
SimpleStorage storage = SimpleStorage.at("0x989fedfsfsd...");
await storage.get();
Writing tests should be as simple as that too, just calling functions.
But in order to make all of this work, you need to start a private network (private testnet for running your tests), setup the connection to that network and then test. As a starting point, we are going to use Truffle, which allows us to build smart contracts and tests much faster and simpler. Truffle will connect to a network, inject web3 and the compiled contracts (called artifacts) and then run the tests.
We will only use a local instance of Truffle (no global installs), so all commands will start with npx. First, create a folder (name it your project’s name e.g. test-tutorial) then, within that folder initialize your project by running npm init -y then install Truffle locally with npm install --save-dev truffle and finally initialize truffle for your project by running npx truffle init (A warning will be thrown, about the non empty folder. Select ‘yes’).
This will create a few files and folders. You will now have truffle-config.js, a contracts folder, a migrations folder and a test folder (if you don’t see a test folder, create one). To learn more about this folder/file structure, please see the Truffle documentation: https://www.trufflesuite.com/docs/truffle/getting-started/creating-a-project.
Now, create the contract shown above in the contracts folder, then create a migrations script to deploy the contract. Name the migrations script 2_deploy.js and save it in the migrations folder.
const SimpleStorage = artifacts.require('SimpleStorage');
module.exports = function (deployer) {
deployer.deploy(SimpleStorage);
};
Once you have created the contract and the migrations script then create a file named SimpleStorage.test.js inside the test folder with the code below and then run npx truffle test. (IMPORTANT NOTE: truffle test only works if there’s no defined networks in truffle-config.js)
const SimpleStorage = artifacts.require('SimpleStorage');
contract('SimpleStorage', (accounts) => {
it('should store a value', async () => {
const simpleStorageInstance = await SimpleStorage.deployed();
// Set value of 89
await simpleStorageInstance.set(89, { from: accounts[0] });
// Get stored value
const storedData = await simpleStorageInstance.get();
assert.equal(storedData, 89, 'The value 89 was not stored.');
});
});
This is the simplest test for a smart contract. The test gets the artifact for the SimpleStorage contract, then starts the contract test, which injects the web3 environment, and then runs the tests.
Note that you don’t need to specify the contract address, thanks to the .deployed() function, which returns an instance of the Truffle deployed contract, so you can just use it like an object, calling the functions.
It’s also good practice to use clean contracts to test, so we have better control over their state. It’s possible to deploy a new one using new instead of deployed, like await SimpleStorage.new();. Note that if the contract has constructor fields, you need to write them here.
Test Helpers (Optional)
OpenZeppelin includes JavaScript testing helpers for smart contract development, openzeppelin-test-helpers (https://github.com/OpenZeppelin/openzeppelin-test-helpers).
These test helpers include helpers such as expectRevert, expectEvent and time which can make testing for reverts, events and time much easier. These test helpers are used in the OpenZeppelin tests (https://github.com/OpenZeppelin/openzeppelin-solidity/tree/master/test)
OpenZeppelin TestEnvironment
Tests take time, it’s a fact. And time is the only thing that nobody can buy. As the test set grows, they will take longer and longer to run. And the question arises “how can I make this faster?”. You might have thought about parallel testing among other options.
One simple and already available option is to use OpenZeppelin Test Environment. If you’ve used other test frameworks before, such as Mocha, you will find yourself at home. Yes, it’s test runner agnostic. This means that, you can use Mocha, Jest or even AVA.
Let’s look at our example updated to use OpenZeppelin Test Environment, OpenZeppelin Test Helpers, Mocha and Chai.
$ npm install --save-dev @openzeppelin/test-environment mocha chai
$ npm install --save-dev @openzeppelin/test-helpers
const { accounts, contract } = require('@openzeppelin/test-environment');
const { BN } = require('@openzeppelin/test-helpers');
const { expect } = require('chai');
const SimpleStorage = contract.fromArtifact('SimpleStorage'); // Loads a compiled contract
describe('SimpleStorage', () => {
it('should store a value', async () => {
const simpleStorageInstance = await SimpleStorage.new();
// Set value of 89
await simpleStorageInstance.set(89, { from: accounts[0] });
// Get stored value
expect(await simpleStorageInstance.get()).to.be.bignumber.equal('89');
});
});
That’s all. One line setup! Everything else is part of the tests. No global variables, tricks, nothing.
As you can see from the documentation, you can also add a test-environment.config.js file which allow you to set some extra configuration, such as block gas limit and accounts.
The example above uses Mocha. So, you will run it like any other Mocha test, with npx mocha --exit --recursive test.
Extra Notes: If you already have a test set setup using truffle, you can migrate it, by following this tutorial.
Coverage
Automated testing makes you feel better, no matter if you are a smart contract or a frontend developer. But when the codebase gets bigger, how do you know how much code was tested?
The trick is to have a coverage tool, but don’t get me wrong, remember, this should not be a guidance tool used before you write your tests. What I mean is, use the tool to help you to know what conditions were not tested. For example, some of the time, we have some if’s in our code that might not make sense, and we only realize that when looking at the coverage report, or when we already have 30 tests for a function, but we forgot to test one else condition. Going deeper, coverage helps us to know how many times a given line of code was executed in the tests.
The package used in this tutorial is solidity-coverage (https://github.com/sc-forks/solidity-coverage). First install the package (npm install --save-dev solidity-coverage) and add solidity-coverage as a truffle plugin to truffle-config.js
plugins: ["solidity-coverage"]
Then run npx truffle run coverage to generate a test coverage report.
Configure solidity-coverage
Looking at the solidity-coverage repository options section (https://github.com/sc-forks/solidity-coverage#options), it’s possible to create a .solcover.js file to configure solidity-coverage, such as ignoring some smart contracts.
It’s recommended to have the same configuration across all machines, so tests or coverage aren’t failing just because a command is running with incorrect parameters.
You don’t need to include coverage reports in version control, so you can add the following to your .gitignore file.
# Coverage directory used by tools like istanbul
coverage
coverage.json
coverageEnv
Continuous Integration
Whilst local test coverage reports are great, it becomes really powerful when used with a Continuous Integration (CI) system. A CI system can run things like tests and coverage when you push commits or merge branches.
An example of a CI tool is https://travis-ci.org/. Travis is very easy to configure, you only need a .travis.yml and everytime you push to your remote GitHub branch it will start running.
The example below is a possible setup for .travis.yml.
dist: xenial
language: node_js
node_js:
- "lts/dubnium"
install:
- npm ci
script:
- npm run test
- npm run coverage
Finally, add two new npm scripts to your package.json, one for test and another for coverage, just like the example below:
"scripts": {
"coverage": "truffle run coverage",
"test": "truffle test"
},
Before pushing the code to your remote repository, remember to sign in to Travis (using GitHub), setup Travis and activate your repository, allowing Travis to run the tests and coverage on your repository. Then push your changes (including your .travis.yml file).
Wait for Travis to build your repository.
Cool, you now have a working CI system. You can also add a build status badge to your repository README.md, Travis provides markdown for the build status that you can copy to your README.md.
The only thing is, the results of the tests and coverage remain in the Travis console. It would be great if you can have those results in a tool like coveralls.
Export coverage
Coveralls (https://coveralls.io/) is a web service that helps you track your code coverage over time.
It’s actually very simple to export the coverage results from solidity-coverage to coveralls. Coveralls accepts lcov reports as input for coverage data, and solidity-coverage actually generates lcov reports. Coveralls integration with Travis is almost automatic, it is as simple as providing the coverage reports to coveralls.
Install the package (https://www.npmjs.com/package/coveralls) with npm install --save-dev coveralls
In your package.json change the coverage command from “npx truffle run coverage” to “npx truffle run coverage && cat coverage/lcov.info | coveralls”.
Before pushing your changes, sign in to coveralls, setup your account, and add your repository to allow coveralls to publish reports about that repository.
You can add a coverage status badge to your repository README.md, coveralls provides markdown for the status that you can copy to your README.md.
Reporters (Optional)
You can use eth-gas-reporter to print an estimate of gas used per function, and the gas costs of deploying each contract.
Install npm install --save-dev eth-gas-reporter and add the following to your truffle-config.js:
mocha: {
reporter: 'eth-gas-reporter',
reporterOptions : { excludeContracts: ['Migrations'] }
},
When you run npx truffle test you get a gas report in your test results such as the following:
·---------------------------------------|----------------------------|-------------|----------------------------·
| Solc version: 0.5.8+commit.23d335f2 · Optimizer enabled: false · Runs: 200 · Block limit: 6721975 gas │
········································|····························|·············|·····························
| Methods │
························|···············|··············|·············|·············|··············|··············
| Contract · Method · Min · Max · Avg · # calls · eur (avg) │
························|···············|··············|·············|·············|··············|··············
| SimpleStorage · set · - · - · 41684 · 1 · - │
························|···············|··············|·············|·············|··············|··············
| Deployments · · % of limit · │
········································|··············|·············|·············|··············|··············
| SimpleStorage · - · - · 105015 · 1.6 % · - │
·---------------------------------------|--------------|-------------|-------------|--------------|-------------·
NOTE: you can have codechecks reports regarding gas usage in deploys and function calls. See more here.
Controlled tests (Optional)
As you may know, Ethereum is deterministic, and sometimes during tests we want to use this to our advantage. This is the last step of this tutorial and it is not a requirement at all, use it only if you want or you need it.
By setting up specific accounts with specific addresses and a given amount of Ether and initial mnemonic, as Ethereum is deterministic, all tests should then always have exactly the same result. If you want to make use of it, this section is for you.
OpenZeppelin has a very good example of this approach. Looking at https://github.com/OpenZeppelin/openzeppelin-contracts/blob/release-v2.4.0/scripts/test.sh we can see that the script defines accounts always with the same addresses and amounts of Ether, as seen after the comment
# 10 accounts with balance 1M ether, needed for high-value tests.
It also starts a local testnet (with ganache-cli) using specific flags for things like the specific addresses.
ganache-cli-coverage --emitFreeLogs true --allowUnlimitedContractSize true --gasLimit 0xfffffffffff --port "$ganache_port" "${accounts[@]}"
If you look down a bit more in test.sh, you will see that when running coverage in a CI system, it will upload the results to coveralls, as shown in the example before
if [ "$SOLIDITY_COVERAGE" = true ]; then
npx truffle run coverage
if [ "$CONTINUOUS_INTEGRATION" = true ]; then
cat coverage/lcov.info | npx coveralls
fi
else
npx truffle test "$@"
fi
Specify the compiler
When using any of the test frameworks mentioned above, they all use a solc version by default. In case you are using truffle and want to specify a version, read more here.
Conclusion
We’ve seen some of the commonly used tools for smart contract development, how to use them locally and how to setup a CI system. Again, there are alternate tools, use the ones that are right for you.
I hope you now have a better idea on how to test smart contracts like a rockstar.
Thanks to the zeppelin community and especially to @martriay who invited to write in the first place and @abcoathup to suggest the topic and improve my writing. Thank you.
Cheers.
this tutorial and share it a lot.
