Set up of Windows ‘slave’ builds for Jenkins

Outline of procedure:

Set up a virtual machine. In our testing regieme we will be using:

  • Windows 8 64bit
  • Windows 8 32bit
  • Windows 7 64bit
  • Windows 7 32bit

We have purchased licenses of these OS’s specifically for deployment in our VirtualBox virtual machines. The setup and installation of windows and VirtualBox is not described here except as pertains to testing. We will be installing the following components on each windows slave:

  • Microsoft .net 3.5 (versions 4+ are not supported). Under windows 8 you need to do this as an explicit install as the OS ships with .net 4.5. .net is needed to run the Jenkins service.
  • Python 2.7. Although QGIS ships with a python interpreter, we need to install our own copy to support running unit tests and so on.
  • QGIS (1.8 at time of writing). The unit tests depend on QGIS being present to run.
  • GitHub for Windows. This is the convenient windows GIT client provided by GitHub - we will use it to clone and update our repositories.

With the base software installed, we will clone two GitHub repositories to the VM:

  • InaSAFE. The application that we will be testing.
  • inasafe_data. Test data required for unit testing the InaSAFE library.

Then we will configure the system to be able to run the unit tests, first independently of Jenkins, and then as a Jenkins job.

Lastly, we will configure the VM Jenkins instance to be a slave for our Jenkins master and fire off the InaSAFE test suite in the VM whenever a commit is made to the InaSAFE master branch.

We will try to keep things as simple as possible. The VM instances should not be used for any other purpose in order to keep them small, fast and to avoid configuration issues caused by conflicting libraries.

VM Configuration:

We used the following configuration options when creating our virtual machines:

  • 4 GB Ram (thus your host OS needs 8GB or more)
  • 25 GB Virtual Hard Disk
  • Installation of guest extensions.

All virtual machines were created using the free VirtualBox software.

Note that you may need to enable virtualisation in your PC bios too.

All Virtual Machines were created with a local user account with

Username: inasafe Password: XXXXXX (redacted)

We did not enable the new ‘cloud login’ options of Windows 8.

After the windows installation, we turned off all extraneous services to maximise performance and increase test times.

Software Installation

The following were the standard software installed on each Windows VM:

QGIS Install

GitHub for Windows Install

  • Install from http://windows.github.com
  • Do not log in with your account details (for security)
  • In options, set your default storage directory to C:Usersinasafe.qgispythonplugins

Copy over the repositories from the host system (quickest) or check them out from github anonymously. If you take the former route, after copying them in to the plugins directory use the GitHub Windows app’s options dialog to find them by clicking the ‘scan for repositories’ button.

Python Install

  • Download and install Python 32Bit (even if you are running a 64 bit windows!!!) to match the version of python shipped with QGIS (python 2.7 in the case of QGIS 1.8).
  • Follow the process described in windows_shell_launcher-label so that you can use the QGIS libraries from a python shell. Note that you probably need to change the second last line of that script to cd "%HOMEPATH%.qgispythonpluginsinasafe" (removing the ‘-dev’) at the end.
  • Follow the processed described in windows-nose-setup so that you have a working pip, nose etc.

Now run the tests and ensure that they can be run from the command line before attempting to run them via Jenkins:

C:\Users\inasafe\.qgis\python\plugins\inasafe>runtests.bat

.net 3.5 Install

To install Jenkins, you first need to ensure you have .net 3.5 on your system. Windows 8 ships with .net 4+ so you need to manually install the older version too. First visit http://www.microsoft.com/en-us/download/details.aspx?id=21 and either choose the .NET Framework Full Package (around 200mb, the option I took) or get the online installer. Note that the full package link is near the bottom of the page.

Run the installer and accept all the defaults to install the .net 3.5 framework.

Jenkins Install

Simply go to http://jenkins-ci.org/ and download the windows native package
and then install it, taking all the defaults.

Once Jenkins is set up, open your browser at http://localhost:8080 to access the Jenkins page.

Jenkins Configuration

Plugins

The first thing you need to do is install some jenkins plugins. To do this do Manage Jenkins ‣ Manage Plugins ‣ Available tab.

Now install at least these plugins:

  • Jenkins GIT plugin
  • GitHub API plugin
  • GitHub plugin

In addition these plugins should be available by default:

  • Jenkins mailer plugin
  • External Monitor Job Type Plugin
  • pam-auth

For simplicity, I also disabled the following plugins:

  • LDAP Plugin
  • ant
  • javadoc
  • Jenkins CVS Plug-in
  • Maven Integration plugin
  • Jenkins SSH Slaves plugins
  • Jenkins Subversion plugin
  • Jenkins Translation Assistance plugin

System configuration

7 We need to provide the path to git so that Jenkins can automatically make checkouts of each version.

Jenkins ‣ Manage Jenkins ‣ Configuration ‣ Git Installations ‣ Path to Git executable needs to be set. On my system I used the following path:

C:\Users\inasafe\AppData\Local\GitHub\PortableGit_93e8418133eb85e81a81e5e19c272776524496c6\bin\git.exe

The GitHub application’s git installer is a portable app and the path for you is going to look a little different - just lookin in your AppData dir and you should find it.

Note

The Jenkins system user will need to have read permissions on the above directory.

Next populate the options in:

  • Jenkins ‣ Manage Jenkins ‣ Configuration ‣ Git Plugins:
  • Global Config user.name Value : <your name>
  • Global Config user.email Value : :kbd:`<your@email.com>
  • Create new accounts base on author/committer’s email : no

Now click the :guiselection:`Save Button` to save your global configuration changes.

Job Configuration - Part 1

Next we create our build job with the following options:

  • Project name : inasafe-win8-64 (adjust the name as appropriate)
  • Build a free-style software project : select

On the job configuration page use the following options:

  • Description : Windows 8 64 bit build of InaSAFE
  • GitHub project : http://github.com/AIFDR/inasafe/
  • Source Code Management section
  • Git : Check
  • Repository URL : git://github.com/AIFDR/inasafe.git
  • Branches to build : version-1_1
  • Repository browser : githubweb
  • Url : http://github.com/AIFDR/inasafe/
  • Build triggers section
  • Poll SCM : check and set to * * * * * for minutely checks.

Save your changes at this point and make a commit, you should see the job produce output something like this the next time a commit takes place:

Started by timer
Building in workspace C:\Jenkins\jobs\inasafe-win8-64\workspace
Checkout:workspace / C:\Jenkins\jobs\inasafe-win8-64\workspace - hudson.remoting.LocalChannel@1fd5730
Using strategy: Default
Last Built Revision: Revision 5403e3ba45129b42edaa2bc0ebd12e8c9ead868e (origin/version-1_1)
Fetching changes from 1 remote Git repository
Fetching upstream changes from git://github.com/AIFDR/inasafe.git
Commencing build of Revision 5403e3ba45129b42edaa2bc0ebd12e8c9ead868e (origin/version-1_1)
Checking out Revision 5403e3ba45129b42edaa2bc0ebd12e8c9ead868e (origin/version-1_1)
Finished: SUCCESS

That validates that at least your git checkout is working as expected.

Job Configuration - Part 2

Now return to the configuration page for the job and lets set up the build process to run our unit tests.

Add build step ‣ Execute windows batch command and enter the following in to the command box. Note that you will need to adjust the path for PortableGit_93e8418133eb85e81a81e5e19c272776524496c6 again in the script below to match your system.:

rem '------------------------------------'
rem '  First update test data            '
rem '------------------------------------'

set PATH=%PATH%;"C:\Users\inasafe\AppData\Local\GitHub\PortableGit_93e8418133eb85e81a81e5e19c272776524496c6\bin\"
if not exist "C:\Jenkins\jobs\inasafe-win8-64\inasafe_data" (
    cd "C:\Jenkins\jobs\inasafe-win8-64\"
    git clone git://github.com/AIFDR/inasafe_data.git
)
cd "C:\Jenkins\jobs\inasafe-win8-64\inasafe_data"
git checkout master
git pull

rem '------------------------------------'
rem '  Now we run the tests!            '
rem '------------------------------------'

SET OSGEO4W_ROOT=C:\PROGRA~2\QUANTU~1
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
call "%OSGEO4W_ROOT%"\apps\grass\grass-6.4.2\etc\env.bat
SET GDAL_DRIVER_PATH=%OSGEO4W_ROOT%\bin\gdalplugins\1.9
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
path %PATH%;%OSGEO4W_ROOT%\apps\grass\grass-6.4.2\lib
path %PATH%;"%OSGEO4W_ROOT%\apps\Python27\Scripts\"
echo "OK"
set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python;
set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\Python27\Lib\site-packages
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT%\apps\qgis

set PATH=c:\python27;%PATH%

cd ..
cd workspace

nosetests -v --with-id --with-coverage --cover-package=storage,engine,impact_functions,gui safe safe_qgis

Now start a new job manually and watch its output using the console e.g.:

http://localhost:8080/job/inasafe-win8-64/10/console

Watch the job to make sure that it runs to completion and that all tests pass.