Mutation based testing

As part of our increased testing coverage/completeness effort, I’ve been thinking about implementing some mutation testing. @federicobond’s eth-mutants project is supposed to help in this regard, though I haven’t tried it out yet. There are some simple mutations we could introduce that would help identify gaps in our testing suite:

  • Remove a modifier from a function
  • Remove an emitted event
  • Edge cases: change > for >=, etc

The obvious downside of this approach is that we’d be going through the test suite many times per run to get proper coverage, so having this on every single PR probably won’t be feasible. We could try to only have related tests run (e.g. if a mutation occurs on Crowdsale, only run Crowdsale-related tests), but that may need some fine-tuning. That said, we can probably aim for having this be executed a couple times a week (daily, ideally).

We could also run more thorough tests on release candidates.