Incorporating `solidity-docgen` into your project

I would like to share how we at Bancor have incorporated solidity-docgen in order to auto-generate our official documentation.

The full process consists of the following steps:

  1. Using solidity-docgen in order to generate a bunch of MarkDown files from the Natspec documentation in our Solidity contracts
  2. Rearranging the output of the previous step into the structure dictated by GitBook’s content-configuration

It assumes the following:

  1. Your entire project resides in a folder next to file package.json
  2. All contracts reside in a folder named contracts, which resides under your project’s folder

Here are the technical details:


Step 1:
Add this in file package.json:

  "scripts": {
    "docify": "node docify.js"
  },
  "dependencies": {
    "solidity-docgen": "0.3.11"
  }

Step 2:
Assuming that your entire project resides under folder project next to file package.json, add this in file docify.js next to file package.json:

const NODE_DIR     = "node_modules";
const INPUT_DIR    = "project/contracts";
const CONFIG_DIR   = "project/docgen";
const OUTPUT_DIR   = "project/docgen/docs";
const README_FILE  = "project/docgen/README.md";
const SUMMARY_FILE = "project/docgen/SUMMARY.md";
const EXCLUDE_FILE = "project/docgen/exclude.txt";

const fs        = require("fs");
const path      = require("path");
const spawnSync = require("child_process").spawnSync;

const excludeList  = lines(EXCLUDE_FILE).map(line => INPUT_DIR + "/" + line);
const relativePath = path.relative(path.dirname(SUMMARY_FILE), OUTPUT_DIR);

function lines(pathName) {
    return fs.readFileSync(pathName, {encoding: "utf8"}).split("\r").join("").split("\n");
}

function scan(pathName, indentation) {
    if (!excludeList.includes(pathName)) {
        if (fs.lstatSync(pathName).isDirectory()) {
            fs.appendFileSync(SUMMARY_FILE, indentation + "* " + path.basename(pathName) + "\n");
            for (const fileName of fs.readdirSync(pathName))
                scan(pathName + "/" + fileName, indentation + "  ");
        }
        else if (pathName.endsWith(".sol")) {
            const text = path.basename(pathName).slice(0, -4);
            const link = pathName.slice(INPUT_DIR.length, -4);
            fs.appendFileSync(SUMMARY_FILE, indentation + "* [" + text + "](" + relativePath + link + ".md)\n");
        }
    }
}

function fix(pathName) {
    if (fs.lstatSync(pathName).isDirectory()) {
        for (const fileName of fs.readdirSync(pathName))
            fix(pathName + "/" + fileName);
    }
    else if (pathName.endsWith(".md")) {
        fs.writeFileSync(pathName, lines(pathName).filter(line => line.trim().length > 0).join("\n\n") + "\n");
    }
}

fs.writeFileSync (SUMMARY_FILE, "# Summary\n");
fs.writeFileSync (".gitbook.yaml", "root: ./\n");
fs.appendFileSync(".gitbook.yaml", "structure:\n");
fs.appendFileSync(".gitbook.yaml", "  readme: " + README_FILE + "\n");
fs.appendFileSync(".gitbook.yaml", "  summary: " + SUMMARY_FILE + "\n");

scan(INPUT_DIR, "");

const args = [
    NODE_DIR + "/solidity-docgen/dist/cli.js",
    "--input="         + INPUT_DIR,
    "--output="        + OUTPUT_DIR,
    "--templates="     + CONFIG_DIR,
    "--solc-module="   + NODE_DIR + "/truffle/node_modules/solc",
    "--solc-settings=" + JSON.stringify({optimizer: {enabled: true, runs: 200}}),
    "--contract-pages"
];

const result = spawnSync("node", args, {stdio: ["inherit", "inherit", "pipe"]});
if (result.stderr.length > 0)
    throw new Error(result.stderr);

fix(OUTPUT_DIR);

Step 3:
Inside folder project, create folder docgen with files exclude.txt and contract.hbs.
File exclude.txt should contain a simple list of all files and folders which you want to exclude from your documentation. Since all of these files and folders reside under folder contracts, there is no need to specify it as part of the path of each file or folder.
File contract.hbs should contain a set of instructions for the desired structure of your documentation. These instructions should be written using the HandleBars language syntax, for example:

{{{natspec.devdoc}}}

{{#if ownFunctions}}
# Functions:
{{#ownFunctions}}
{{#if (or (eq visibility "public") (eq visibility "external"))}}
- [`{{name}}({{args}})`](#{{anchor}})
{{/if}}
{{/ownFunctions}}
{{/if}}

{{#if ownEvents}}
# Events:
{{#ownEvents}}
- [`{{name}}({{args}})`](#{{anchor}})
{{/ownEvents}}
{{/if}}

{{#ownFunctions}}
{{#if (or (eq visibility "public") (eq visibility "external"))}}
# Function `{{name}}({{args}}){{#if outputs}} → {{outputs}}{{/if}}` {#{{anchor~}} }
{{#if natspec.devdoc}}{{natspec.devdoc}}{{else}}No description{{/if}}
{{#if natspec.params}}
## Parameters:
{{#natspec.params}}
- `{{param}}`: {{description}}
{{/natspec.params}}
{{/if}}
{{#if natspec.returns}}
## Return Values:
{{#natspec.returns}}
- {{param}} {{description}}
{{/natspec.returns}}
{{/if}}
{{/if}}
{{/ownFunctions}}

{{#ownEvents}}
# Event `{{name}}({{args}})` {#{{anchor~}} }
{{#if natspec.devdoc}}{{natspec.devdoc}}{{else}}No description{{/if}}
{{#if natspec.params}}
## Parameters:
{{#natspec.params}}
- `{{param}}`: {{description}}
{{/natspec.params}}
{{/if}}
{{/ownEvents}}

Following that, you (and anyone else seeking to do so) can auto-generate the docs via npm run docify.
Then you can publish your project’s documentation on GitBook simply by registering your repository for this service.
Note that you will still need to add all the auto-generated files (.gitbook.yaml, project/docgen/SUMMARY.md and everything under project/docgen/docs) to your GitHub repository.

Thank you OZ!!!

7 Likes

Hi @barakman,

Thank you for this fantastic guide, and all your testing, feedback and contributions to improve solidity-docgen. It is greatly appreciated. :pray:

@barakman

I've made a few changes to your script to generalize it a bit.

Is now

  '--solc-module=' + NODE_DIR + '/solc',
  '--solc-settings=' +
    JSON.stringify({
      remappings: ['@openzeppelin/=./node_modules/@openzeppelin/'],
      optimizer: { enabled: true, runs: 200 },
    }),

I had to use the default solc, because I am just using oz without truffle and the truffle version was 5.0.0 so it didn't meet the minimum requirements.

The second, I had a compilation error if I didn't add the openzeppelin contracts remapping. This might have been caused because I was using the truffle solc, and my project wasn't using truffle to compile.

Great work!

4 Likes

@Emiller88: Thank you, glad to know you’ve found it useful.

At the time of posting this guide, I forgot that my script had been designed for a Truffle-structured project (though this fact is implicitly implied at the top of my post - “All contracts reside in a folder named contracts, which resides under your project’s folder”).

So yes, it does indeed configure --solc-module to the compiler used by Truffle, under the assumption that your project relies on Truffle.

I suppose that the general solution would be to place this path alongside all those paths at the top of the script (which I have placed there in order to make it clear that these are the constants which every user should configure specifically for his/her project).

Since we do not have any “external contract” dependency (such as openzeppelin contracts in your example), I did not face any issue which required adding remappings. Glad you were able to sort this out yourself, because as I recall, solc has some issues with contracts which reside outside of the project’s directory (resulting with a File outside of allowed directories error).

Thanks

1 Like

@barakman
I am confused about this project directory. Does the contract’s parent directory needs to be named project ?
I tried I replacing project with ./
Also tried creating a project subdirectory with all the elements listed. At the end both came back with the same error:

$ npm run docify                                                                                                                                                                                                                                    

> node docify.js

project/docify.js:69
  throw new Error(result.stderr)
  ^

Error: TypeError: this.solc.compileStandardWrapper is not a function

line 69 being throw new Error(result.stderr)

1 Like

Please note the following instruction:

Also, note that I've implemented this based on Truffle 4.x, meaning:

  1. All contracts are assumed to be under folder contracts
  2. Solidity compiler is assumed to be under <NODE_DIR>/truffle/node_modules/solc

If you change the project part to ., then I'm not sure what happens.
If anything, you should just remove the project/ part.
But that means that everything under it - the contracts folder, the docgen folder, the test folder (storing all Truffle tests) - must reside next to folder <NODE_DIR> and next to file package.json.

From the looks of it, your error seems to imply that docgen is unable to execute solc.
I suggest that you read the suggestion by Emiller88 above, it might be related to your problem.

Thanks

2 Likes

Thanks, its all sorted out now.

2 Likes

Thanks! Used your template as base for my own. Saved lots of time but fixed it a bit.
You can strip “fixing” script a lot if you just adjust template a little bit. For example one insterting newlines can just be done adding one bland line before ownFuncations

2 Likes