0.8.0 Additional Coverage Measurements & Restructure (Merge)

Author: cgewecke

.circleci/config.yml

@@ -27,10 +27,6 @@ jobs:
           name: Delete any old node_modules
           command: |
             rm -rf node_modules/
-      - run:
-          name: Install truffle (globally)
-          command: |
-            sudo npm install -g truffle
       - run:
           name: Install dependencies
           command: |
@@ -54,23 +50,6 @@ jobs:
           name: Zeppelin E2E
           command: |
             ./scripts/run-zeppelin.sh
-  e2e-metacoin:
-    machine: true
-    steps:
-      - checkout
-      - <<: *step_install_nvm
-      - run:
-          name: MetaCoin E2E
-          command: |
-            ./scripts/run-metacoin.sh
-  e2e-metacoin-windows:
-    executor: win/default
-    steps:
-      - checkout
-      - run:
-          name: Windows Metacoin E2E
-          command: |
-            bash ./scripts/run-metacoin.sh
   e2e-nomiclabs:
     machine: true
     steps:
@@ -86,6 +65,4 @@ workflows:
     jobs:
       - unit-test
       - e2e-zeppelin
-      - e2e-metacoin
-      - e2e-metacoin-windows
       - e2e-nomiclabs

BUIDLER_README.md

@@ -58,6 +58,8 @@ A working example can be found at [openzeppelin-contracts, here.][35]
 | solcoverjs | `--solcoverjs ./../.solcover.js` | Relative path from working directory to config. Useful for monorepo packages that share settings. (Path must be "./" prefixed) |
 | network    | `--network development` | Use network settings defined in the Hardhat config |
 | temp[<sup>*</sup>][14]       | `--temp build`   | :warning: **Caution** :warning:  Path to a *disposable* folder to store compilation artifacts in. Useful when your test setup scripts include hard-coded paths to a build directory. [More...][14] |
+| matrix   | `--matrix` | Generate a JSON object that maps which mocha tests hit which lines of code. (Useful
+as an input for some fuzzing, mutation testing and fault-localization algorithms.) [More...][39]|
 
 [<sup>*</sup> Advanced use][14]
 
@@ -76,11 +78,14 @@ module.exports = {
 | Option <img width=200/>| Type <img width=200/> | Default <img width=1300/> | Description <img width=800/> |
 | ------ | ---- | ------- | ----------- |
 | silent | *Boolean* | false | Suppress logging output |
-| client | *Object* | `require("ganache-core")` | Useful if you need a specific ganache version. |
-| providerOptions | *Object* | `{ }` | [ganache-core options][1]  |
 | skipFiles | *Array* | `['Migrations.sol']` | Array of contracts or folders (with paths expressed relative to the `contracts` directory) that should be skipped when doing instrumentation. |
 | measureStatementCoverage | *boolean* | `true` | Computes statement (in addition to line) coverage. [More...][34] |
 | measureFunctionCoverage | *boolean* | `true` | Computes function coverage. [More...][34] |
+| measureModifierCoverage | *boolean* | `true` | Computes each modifier invocation as a code branch. [More...][34] |
+| modifierWhitelist | *String[]* | `[]` | List of modifier names (ex: "onlyOwner") to exclude from branch measurement. (Useful for modifiers which prepare something instead of acting as a gate.)) |
+| matrixOutputPath | *String* | `./testMatrix.json` | Relative path to write test matrix JSON object to. [More...][39]|
+| mochaJsonOutputPath | *String* | `./mochaOutput.json` | Relative path to write mocha JSON reporter object to. [More...][39]|
+| abiOutputPath | *String* | `./humanReadableAbis.json` | Relative path to write diff-able ABI data to |
 | istanbulFolder | *String* | `./coverage` |  Folder location for Istanbul coverage reports. |
 | istanbulReporter | *Array* | `['html', 'lcov', 'text', 'json']` | [Istanbul coverage reporters][2]  |
 | mocha | *Object* | `{ }` | [Mocha options][3] to merge into existing mocha config. `grep` and `invert` are useful for skipping certain tests under coverage using tags in the test descriptions.|
@@ -92,6 +97,9 @@ module.exports = {
 | onIstanbulComplete[<sup>*</sup>][14] | *Function* |  | Hook run *after* the Istanbul reports are generated, *before* the ganache server is shut down. Useful if you need to clean resources up. [More...][23]|
 | configureYulOptimizer | *Boolean* | false | (Experimental) Setting to `true` should resolve "stack too deep" compiler errors in large projects using ABIEncoderV2 |
 | solcOptimizerDetails | *Object* | `undefined` |(Experimental) Must be used in combination with `configureYulOptimizer`. Allows you configure solc's [optimizer details][1001]. Useful if the default remedy for stack-too-deep errors doesn't work in your case (See FAQ below). |
+| client | *Object* | `require("ganache-core")` | Ganache only: useful if you need a specific ganache version |
+| providerOptions | *Object* | `{ }` | Ganache only: [ganache-core options][1]  |
+
 
 [<sup>*</sup> Advanced use][14]
 
@@ -211,6 +219,7 @@ $ yarn
 [36]: https://hardhat.org/
 [37]: https://github.com/sc-forks/solidity-coverage/blob/master/HARDHAT_README.md
 [38]: https://github.com/sindresorhus/globby#globbing-patterns
+[39]: https://github.com/sc-forks/solidity-coverage/blob/master/docs/advanced.md#generating-a-test-matrix
 [1001]: https://docs.soliditylang.org/en/v0.8.0/using-the-compiler.html#input-description
 [1002]: https://github.com/sc-forks/solidity-coverage/blob/master/docs/faq.md#running-out-of-stack
 

README.md

@@ -106,3 +106,27 @@ Setting the `measureStatementCoverage` and/or `measureFunctionCoverage` options
 improve performance, lower the cost of execution and minimize complications that arise from `solc`'s
 limits on how large the compilation payload can be.
 
+## Generating a test matrix
+
+Some advanced testing strategies benefit from knowing which tests in a suite hit a
+specific line of code. Examples include:
++ [mutation testing][22], where this data lets you select the correct subset of tests to check
+a mutation with.
++ [fault localization techniques][23], where the complete data set is a key input to algorithms that try
+to guess where bugs might exist in a given codebase.
+
+Running the coverage command with `--matrix` will write [a JSON test matrix][25] which maps greppable
+test names to each line of code to a file named `testMatrix.json` in your project's root.
+
+It also generates a `mochaOutput.json` file which contains test run data similar to that
+generated by mocha's built-in [JSON reporter][27].
+
+In combination these data sets can be passed to Joram's Honig's [tarantula][29] tool which uses
+a fault localization algorithm to generate 'suspiciousness' ratings for each line of
+Solidity code in your project.
+
+[22]: https://github.com/JoranHonig/vertigo#vertigo
+[23]: http://spideruci.org/papers/jones05.pdf
+[25]: https://github.com/sc-forks/solidity-coverage/blob/master/docs/matrix.md
+[27]: https://mochajs.org/api/reporters_json.js.html
+[29]: https://github.com/JoranHonig/tarantula