Showing Pull Request Build Status in Yaydoc

Yaydoc is integrated to various open source projects in FOSSASIA.  We have to make sure that the contributors PR should not break the build. So, I decided to check whether the PR is breaking the build or not. Then, I would notify the status of the build using GitHub status API.

exports.registerHook = function (data, accessToken) {
  return new Promise(function(resolve, reject) {
    var hookurl = 'http://' + process.env.HOSTNAME + '/ci/webhook';
    if (data.sub === true) {
      hookurl += `?sub=true`;
    }
    request({
      url: `https://api.github.com/repos/${data.name}/hooks`,
      headers: {
        'User-Agent': 'Yaydoc',
        'Authorization': 'token ' + crypter.decrypt(accessToken)
      },
      method: 'POST',
      json: {
        name: "web",
        active: true,
        events: [
          "push",
          "pull_request"
        ],
        config: {
          url: hookurl,
          content_type: "json"
        }
      }
    }, function(error, response, body) {
      if (response.statusCode !== 201) {
        console.log(response.statusCode + ': ' + response.statusMessage);
        resolve({status: false, body:body});
      } else {
        resolve({status: true, body: body});
      }
    });
  });
};

I’ll register the webhook, when user registers the repository to yaydoc for push and pull request event. Push event will be for building documentation and hosting the documentation to the GitHub pages. Pull_request event would be for checking the build of the pull request.

github.createStatus(commitId, req.body.repository.full_name, "pending", "Yaydoc is checking your build", repositoryData.accessToken, function(error, data) {
                    if (!error) {
                      var user = req.body.pull_request.head.label.split(":")[0];
                      var targetBranch = req.body.pull_request.head.label.split(":")[1];
                      var gitURL = `https://github.com/${user}/${req.body.repository.name}.git`;
                      var data = {
                        email: "admin@fossasia.org",
                        gitUrl: gitURL,
                        docTheme: "",
                        debug: true,
                        docPath: "",
                        buildStatus: true,
                        targetBranch: targetBranch
                      };
                      generator.executeScript({}, data, function(error, generatedData) {
                        var status, description;
                        if(error) {
                          status = "failure";
                          description = error.message;
                        } else {
                          status = "success";
                          description = generatedData.message;
                        }
                        github.createStatus(commitId, req.body.repository.full_name, status, description, repositoryData.accessToken, function(error, data) {
                          if (error) {
                            console.log(error);
                          } else {
                            console.log(data);
                          }
                       });
                 });
              }
        });

When anyone opens a new PR, GitHub will send  a request to yaydoc webhook. Then, I’ll send the status to GitHub saying that “Yaydoc is checking your build” with status `pending`. After, that I’ll documentation will be generated.Then, I’ll check the exit code. If the exit code is zero,  I’ll send the status `success` otherwise I’ll send `error` status.
Resources:

Adding Github buttons to Generated Documentation with Yaydoc

Many times repository owners would want to link to their github source code, issue tracker etc. from the documentation. This would also help to direct some users to become a potential contributor to the repository. As a step towards this feature, we added the ability to add automatically generated GitHub buttons to the top of the docs with Yaydoc.

To do so we created a custom sphinx extension which makes use of http://buttons.github.io/ which is an excellent service to embed GitHub buttons to any website. The extension takes multiple config values and using them generates the `html` which it adds to the top of the internal docutils tree using a raw node.

GITHUB_BUTTON_SPEC = {
    'watch': ('eye', 'https://github.com/{user}/{repo}/subscription'),
    'star': ('star', 'https://github.com/{user}/{repo}'),
    'fork': ('repo-forked', 'https://github.com/{user}/{repo}/fork'),
    'follow': ('', 'https://github.com/{user}'),
    'issues': ('issue-opened', 'https://github.com/{user}/{repo}/issues'),
}

def get_button_tag(user, repo, btn_type, show_count, size):
    spec = GITHUB_BUTTON_SPEC[btn_type]
    icon, href = spec[0], spec[1].format(user=user, repo=repo)
    tag_fmt = '<a class="github-button" href="{href}" data-size="{size}"'
    if icon:
        tag_fmt += ' data-icon="octicon-{icon}"'
    tag_fmt += ' data-show-count="{show_count}">{text}</a>'
    return tag_fmt.format(href=href,
                          icon=icon,
                          size=size,
                          show_count=show_count,
                          text=btn_type.title())

The above snippet shows how it takes various parameters such as the user name, name of the repository, the button type which can be one of fork, issues, watch, follow and star, whether to display counts beside the buttons and whether a large button should be used. Another method named get_button_tags is used to read the various configs and call the above method with appropriate parameters to generate each button.

The extension makes use of the doctree-resolved event emitted by sphinx to hook into the internal doctree. The following snippet shows how it is done.

def on_doctree_resolved(app, doctree, docname):
    if not app.config.github_user_name or not app.config.github_repo:
        return
    buttons = nodes.raw('', get_button_tags(app.config), format='html')
    doctree.insert(0, buttons)

Finally we add the custom javascript using the add_javascript method.

app.add_javascript('https://buttons.github.io/buttons.js')

To use this with yaydoc, users would just need to add the following to their .yaydoc.yml file.

build:
  github_button:
    buttons:
      watch: true
      star: true
      issues: true
      fork: true
      follow: true
    show_count: true
    large: true

Resources

  1.  Homepage of Github:buttons – http://buttons.github.io/
  2. Sphinx extension Tutorial – http://www.sphinx-doc.org/en/stable/extdev/tutorial.html

Deploying documentations generated by Yaydoc to Heroku

There are many web applications available online that generates static websites. Among these projects are two unique projects developed here at FOSSASIA. These are the Open Event WebApp Generator and Yaydoc (an automatic documentation generation and deployment project.). Since Yaydoc already supports the deployment of the generated documentations to Github pages, it was just a matter of time that the deployment to Heroku is also supported.

Heroku is an excellent cloud-based platform used as a web application deployment service. Heroku provides most of its services at free of cost to the users and is excellent to host static websites provided that a little bit of tweaking is done.

For this implementation, we use the `Platform API` provided by Heroku. Stating it’s description mentioned in the documentation,

The platform API empowers developers to automate, extend and combine Heroku with other services. You can use the platform API to programmatically create apps, provision add-ons and perform other tasks that could previously only be accomplished with Heroku toolbelt or dashboard.

In order to deploy the static websites to Heroku, we need to first prepare a bundle of source code that has been compiled and is ready for execution on the Heroku runtime. This bundle is known as a Slug.

cd temp/$EMAIL/${UNIQUE_ID}_preview
mkdir -p app
cd app

curl https://nodejs.org/dist/v6.11.0/node-v6.11.0-linux-x64.tar.gz | tar xzv > /dev/null

cp $BASE/web.js
rsync -av --progress ../ . --exclude app

cd ..
tar czfv slug.tgz ./app > /dev/null

We are using the files generated for preview to bundle them in a slug. Also, we download the NodeJS runtime files since we are deploying a static website to Heroku. Along with the static files, we require bundling a NodeJS server file (web.js) that will be used to reference the static files in the application.

After preparing the Slug, we publish the static web application to Heroku. For this, we start by creating a Heroku app using the command `heroku create <app-name>`. The app name is decided by the user when he or she fills the form in the Yaydoc Web App. Following that, we request Heroku to allocate a new slug for your app. After that, we upload the slug tar file to the platform.

# Create Heroku 
heroku create $APP_NAME

# Allocating new Slug
Arr=($(curl -u “:$API_KEY” -X \
-H ‘Content-Type:application/json’ \
-H ‘Accept: application/vnd.heroku+json;version=3’ \
-d ‘{“process_types”:{“web”:”node-v6.11.0-linux-x64/bin/node web.js”}}’ \
-n https://api.heroku.com/apps/${APP_NAME}/slugs | \
python -c “import sys,json; obj=json.load(sys.stdin);
print(obj[‘blob’][‘url’] + ‘\n’ + obj[‘id’])”))

# Upload the slug tar file
curl -X PUT \
-H “Content-Type:”\
--data-binary @slug.tgz \
“${Arr[0]}”

After uploading the slug to Heroku, we need to release the app. This is done using the following command.

curl -u “:$API_KEY” -X POST \
-H “Accept: application/vnd.heroku+json; version=3” \
-H “Content-Type: application/json” \
-d ‘{“slug”:”’${Arr[1]}’”}’ \
-n https://api.heroku.com/apps/$APP_NAME/releases

Releasing the application completes the process of deployment, making the documentation generated by Yaydoc up and running at the following URL: https://<app-name>.herokuapp.com/