Fabric CI process

This document explains the CI process for the fabric repository. The following steps explain the execution process when a patch set is submitted to the fabric repository.

At whatever point a patch set is submitted to the Fabric repository, Jenkins triggers the CI build process to test and validate the patch set. The Fabric CI verify and merge jobs are configured to test patch sets in the jenkins environment.

The Hyperledger Fabric (and associated) projects utilize various tools and workflows for continuous integration. Fabric CI is currently utilizing the following revisions in the master, release-1.4, release-1.3 branches…

Master:

  • GO version:(e.g. v1.11.1) https://github.com/hyperledger/fabric/blob/master/ci.properties
  • DOCKER version: 18.06.1-ce
  • baseimage version:(e.g. 0.4.14) https://github.com/hyperledger/fabric/blob/master/Makefile#L50

Release-1.4:

  • GO version:(e.g. v1.11.1) https://github.com/hyperledger/fabric/blob/release-1.4/ci.properties
  • DOCKER version: 18.06.1-ce
  • baseimage version:(e.g. 0.4.14) https://github.com/hyperledger/fabric/blob/release-1.4/Makefile#L50

Release-1.3:

  • GO version:(e.g. v1.10.4) https://github.com/hyperledger/fabric/blob/release-1.3/ci.properties
  • DOCKER version: 18.06.1-ce
  • baseimage version:(e.g. 0.4.13) https://github.com/hyperledger/fabric/blob/release-1.3/Makefile#L50

If you are interested to know more details about fabric CI process, Please click the link: Fabric,

There are several job types that are common across all Hyperledger Fabric projects. In some cases, all of the common job types are not in every project. This depends on the specific needs of the Hyperledger Fabric project being developed. The CI configuration is prepared using Jenkins Job Builder to create, update and modify each of the Jenkins Jobs.

As part of managing jobs ran by our Jenkins servers, we create JJBs (Jenkins Job Builder files) in YAML format to configure Jenkins jobs. JJBs have a flexible templating system, so creating multiple jobs with a common configuration is straightforward. You can find more details about Jenkins Job Builder on the JJB webpage.

The following steps explain what happens when a developer submits a patch set to the fabric repository.

Build Process

At the point when a patch set is submitted to the fabric repository, the Hyperledger Community CI server (Jenkins) triggers Verify jobs on x86_64 platform using the patch set’s parent commit which may or may not be the latest commit on fabric.

What does verify jobs do:

Gerrit trigger events that trigger relevant jobs based on the patch set:

  • fabric-verify-build-checks-x86_64 job is triggered when a patch set is created and it validates the patch sets git commit message.
  • If the commit message in the patch set contains ‘WIP’, the above build job ignores the build process and will not post a vote back to Gerrit. This means that this job is essentially skipping the build process. As a result, you will see “WIP - No build” in the patch set’s comment history.
  • If the patch set is not marked as ‘WIP’ in the commit message, and if it is a documentation change (which is defined as a patch set with changes solely on these file extensions (.rst, .md, .py, .png,.css,.html and .ini), the above job posts the Run DocBuild comment to the patch set in Gerrit and sends the following votes against the patch set: F1-VerifyBuild=+1 F3-IntegrationTest=+1 F3-UnitTest=+1.
Views

Views

  • Run DocBuild
    • This comment triggers fabric-docs-build-x86_64 CI job. Once the doc build is successfully executed, Jenkins sends Gerrit a vote of F2-DocsBuild=+1 on a successful build of the docs, or in the event of a failed doc build, a vote of F2-DocsBuild=-1 is sent.
  • If the patch set is not marked as ‘WIP’ in the commit message, and there are code and documentation changes, fabric-verify-build-checks-x86_64 executes the flow described below. This flow also applies to the code only patch set excluding documentation build process.

    • Executes make basic-checks, make native. If any of these make targets fail, the fabric-verify-build-checks-x86_64 job sends a F1-VerifyBuild=-1 vote to Gerrit on the patch set, otherwise it sends F1-VerifyBuild=+1, then triggers DocsBuild and IntegrationTest & UnitTest jobs in parallel by posting comments to the patch set:

    • Run DocsBuild
      • This comment triggers fabric-docs-build-x86_64 job and posts a vote of F2-DocsBuild=+1 to Gerrit if successful, or F2-DocsBuild=-1 in the event of a failure. You can review the RTD output from the doc build on the nexus log server.

        What happens with fabric-docs-build-x86_64 job

        Step 1: Builds the documentation changes:
        • Extracts the documentation files(.md, .rst, .txt, .py, .png, .css, .html & .ini) from the patch set submitted and builds the documentation after verification checks like syntax, and tox verification.This job is triggered only when a patch set contains documentation files.
        Step 2: Documented output is published to build logs:
        • Once the documentation build is successful, it is archived, and published on the Nexus repository.
      • Run IntegrationTest
        • This comment triggers fabric-verify-integration-tests-x86_64 job and posts F3-IntegrationTest=+1 on a successful run of the integration tests and a F3-IntegrationTest=-1 in the event of a failure.
      • Run UnitTest
        • This comment triggers fabric-verify-unit-tests-x86_64 job and posts F3-UnitTest=+1 vote against the patch set if successful, otherwise a vote of F3-UnitTest=-1 is submitted.
Views

Views

Conditions to merge the patch set

Maintainers have to look for +1 votes on all the labels before they merge the patch sets. The votes on the patch set will look like the following:

F1-VerifyBuild     +1 Hyperledger Jobbuilder
F2-DocBuild        +1 Hyperledger Jobbuilder
F3-IntegrationTest +1 Hyperledger Jobbuilder
F3-UnitTest        +1 Hyperledger Jobbuilder

patch set is not eligible to merge, if it even gets one -1.

Views

Views

Merge Jobs

The Fabric merge build process is split up into multiple jobs.

What does merge jobs do:

Once the patch set is approved by the maintainers ofthe fabric repository, they will merge the patch set which triggers the Merge jobs mentioned above on the latest Fabric commit (note: this process does not use the patch set’s parent commit).

Views

Views

fabric-merge-end-2-end-x86_64:

Step 1: Builds the fabric images and binaries and third-party images (Kafka, Zookeeper, CouchDB) by using make targets.
  • docker – builds the fabric images
  • release-clean – clean the binaries from the host platform
  • release – builds release packages for the host platform
  • docker-thirdparty – builds the third-party images

Step 2: Clone the fabric-ca repository:

  • Clone the latest commit from the fabric-ca repository and then checks out the branch associated with the patch set. If the patch set is triggered on the fabric-ca release-1.1 branch, the script will checkout to the release-1.1 branch.
  • After the fabric-ca repository is cloned in the above step, the CI script proceeds to build the docker images that will be used to run the end-to-end (e2e) tests.

Step 3: Executes the e2e tests:

Below are the tests triggers in Fabric e2e job:

  1. e2e-node - Runs the sdk-node e2e tests (Executes gulp test command).
    • Clones fabric-sdk-node repository and will checkout to Branch
    • Spins up network using the docker-compose file from test/fixtures folder
    • Install nodejs 8.11.3 version on master (varies from fabric branch to branch)
    • RUN istanbul cover --report cobertura test/integration/e2e.js
  2. e2e-java - Runs e2e java integration tests.
    • If the patch set is on release-1.0 branch, CI skips the java e2e tests for now. TODO# mention the JIRA FAB here.
    • If not, run the java e2e tests by executing source cirun.sh
  3. byfn and efyn - Runs byfn and eyfn tests with default, custom channel, couchdb and nodejs chaincode. See the CI script here: https://github.com/hyperledger/ci-management/blob/master/jjb/fabric-samples/shell/include-raw-fabric-samples-byfn-e2e.sh

fabric-merge-x86_64:

Step 1: Pulls the third party docker images:

  • Pulls the fabric-javaenv from nexus and fabric baseimage version third party docker images (kafka, zookeeper, couchdb). The image name is appended with ‘hyperledger’ and tagged with the latest tag.

Step 2: Executes Fabric tests using below two commands:

make linter See the make linter target in fabric/Makefile
( https://github.com/hyperledger/fabric/blob/master/Makefile#L207 )
make unit-test See the make unit-test target in fabric/Makefile
( https://github.com/hyperledger/fabric/blob/master/Makefile#L186 )

After the verify or merge tests are executed, It is time to archive the logs (artifacts). CI publishes the logs(artifacts) on Jenkins console.

ConsoleOutPut

ConsoleOutPut

Build Notifications

Jenkins sends the build failure email notifications to the CI team.

Trigger failed jobs through Gerrit comments

Re-triggering builds is possible with Jenkins by entering a specific comment on the Gerrit patch set, which can re-trigger a specific verify job. To do so, follow the process below:

Step 1: Open the Gerrit patch set for which you want to reverify the build

Step 2: Click on Reply, then type one of the below comments and click Post

VerifyBuild – Triggers fabric-verify-build-checks-x86_64 CI job, developers have to check the result of this job before posting the below comments on the patch set.

Run UnitTest - Triggers fabric-verify-unit-tests-x86_64.

Run DocsBuild - Triggers fabric-docs-build-x86_64

Run IntegrationTest - Triggers fabric-verify-integration-tests-x86_64.

These comments trigger specific Fabric verify jobs. Once the build is triggered, you can verify the Jenkins console output and go through the log messages if you are interested to know how the build is actively making progress.

Rebasing Patch Sets

With the existing Jenkins configuration, when rebasing a patchset from gerrit, Jenkins will not trigger the builds if it is a trivial rebase.

What is trivial rebase means? Conflict-free merge between the new parent and the prior patch set.

How to trigger manually from Gerrit?

  1. Click on rebase button from the patchset
  2. Post “Run VerifyBuild” or “VerifyBuild” comment phrase and click on post.

Build on Commit Message changes:

With the existing Jenkins job configuration, Jenkins will not trigger jobs on commit message changes.

Questions

Please reach out to us in https://chat.hyperledger.org/channel/ci-pipeline or https://chat.hyperledger.org/channel/fabric-ci RC channels for any questions.