Performing automation/smoking tests on a machine other than build machine

After my product has been built, it should be sent to another machine to run smoking/automation tests. If tests pass, mark current build as successful; otherwise, mark it as failed.

  1. Take "product2" as example, we create a configuration root.product2 at machine1 to build this product. In this configuration, set up the following things:

    • Create two repositories:

      repository1

      This repository is created to check out source code of product2 from CVS.

      repository2

      This repository is created to retrieve test log from latest build of configuration root.test-product2@machine2. In this repository, we define a module with source path be "..", and file name pattern be "build_log.txt". In this way, test log will be retrieved (Note that source path is relative to artifacts directory, so we use ".." to change to parent directory of artifacts, which includes the build log file).

    • Create two builders:

      builder1

      This builder is used to build product2.

      builder2

      This builder is used to publish test log into some sub directory of artifacts directory, so that it can be accessed from web interface.

    • Create five steps:

      check out from CVS

      This step uses repository1 to check out source code of product2.

      build with Ant

      This step uses builder1 to build product2.

      retrieve test results

      This step uses repository2 to check out test log of product2. At this point, you may curious about how build result of product2 is sent to machine2. The trick is at root.test-product2@machine2 side. In that configuration, we will define steps to pull latest build result of product2.

      publish test results

      This step uses builder2 to publish test log. Step necessary condition property of this step should be set to true in order to publish the test results for review even the step "retrieve test results" fails.

      default

      This step executes the above four steps one by one.

  2. Another configuration root.test-product2 needs to be created at machine2 to run test against product2. In this configuration, set up the following:

    • Set value of property build necessary condition as true, which means new build will always be generated upon triggering. This is needed because this configuration will be dependently triggered by root.product2@machine1, and for each dependent triggering, we want new build of root.test-product2 been generated so that tests can be run.

    • Create two repositories:

      repository1

      This repository is created to check out build result of product2 from latest build of configuration root.product2@machine1.

      repository2

      This repository is created to retrieve test script from CVS.

    • Create a builder builder1, which will be used to run test script.

    • Create four steps as below:

      check out test scripts

      This step uses repository2 to check out test scripts from CVS.

      check out build results of product2

      This step uses repository1 to check out build results of product2.

      run test against product2

      This step uses builder1 to run test.

      default

      This step executes the above steps one by one.

  3. In this way, build result of product1 will be tested on machine2, and test results will be collected back to machine1. If test fails (that is, build of root.test-product2@machine2 fails), step "retrieve test results" at root.product2@machine1 side will also fail, which causes build of product2 failed.

    [Note]Note

    As you may noticed, root.product2@machine1 depends on root.test-product2@machine2 to retrieve test results, and root.test-product2@machine2 depends on root.product2@machine1 to retrieve product2 build results. Obviously there is a dependency loop here. QuickBuild is clever enough to handle this correctly:

    • When a configuration is triggered actively (that is, triggered by user or schedule, instead of dependency resolver), other configurations detected in a dependency loop is considered as subordinate configurations which will not be taken into account when evaluating build necessary condition of the active configuration. In our case, root.product2@machine1 is triggered actively, and root.test-product2@machine2 plays the role as subordinate configuration.

    • At root.product2@machine1 side, when step "retrieve test results" executes, it will wait until the newly generated build at root.test-product2@machine2 side completes (either successful or failed), and then downloads test results from that build.

    • At root.test-product2@machine2 side, when step "check out build results of product2" executes, it knows that it is a subordinate configuration, and will download build results of product2 right away from root.product2@machine1 without waiting for build of root.product2@machine1 been finished. Otherwise, deadlock will occur. Consequently, you should make sure build results of product2 are available for retrieving before step "retrieve test results" gets running in configuration root.product2@machine1.

    [Warning]Warning

    You should never actively trigger a subordinate configuration. Otherwise, deadlock may occur.

  4. A live demo is available through QuickBuild's demo site http://livedemo.pmease.com:8081/. Within this live demo:

    • root.distributed-builds.product2 stands for configuration root.product2@machine1 we talked about.

    • root.distributed-builds.LinuxBox.test-product2 simulates configuration root.test-product2@machine2.