Development under MS Windows

In this document we will walk you through the different activities you will need to do as a windows developer wishing to work on the InaSAFE codebase.

Installation of version control tools

Setup msysgit

To check out the code for development, you first need to install a git client. We cover msysgit here, but you can also use tortoisegit if you prefer (although the tortoise git procedure is not covered here).

To install msysgit (which is a command line git client), download the latest version of the software from the msysgit web site. There is no need to get the ‘full install’ - just fetching the latest ‘preview’ is good enough. For example at the time of writing I downloaded Git-1.7.9-preview20120201.exe. The download is around 14mb in size.

Once the file is downloaded, run it and respond to the installer prompts as illustrated below:

../_images/msysgit-step1.jpg
../_images/msysgit-step2.jpg
../_images/msysgit-step3.jpg
../_images/msysgit-step4.jpg
../_images/msysgit-step5.jpg
../_images/msysgit-step6.jpg
../_images/msysgit-step7.jpg
../_images/msysgit-step8.jpg
../_images/msysgit-step9.jpg

Check out the code and the test data

In this section we actually check out the source code and the test data using the tools we installed above.

Clone the code repository

First open a GIT bash prompt as illustrated below:

../_images/msysgit-step10.jpg

The repository can now be cloned by issuing the commands listed below.:

cd  /c/Documents\ and\ Settings/<your username>/

mkdir -p .qgis/python/plugins

cd .qgis/python/plugins/

git clone https://<your username>@github.com/AIFDR/inasafe.git inasafe-dev

Note

The items in angle brackets above should be replaced with your personal details as required.

When the final command above runs, you should see something like this in the console when the clone process is completed:

$ git clone https://timlinux@github.com/AIFDR/inasafe.git inasafe-dev
Cloning into 'inasafe'...
remote: Counting objects: 5002, done.
remote: Compressing objects: 100% (1526/1526), done.
remote: Total 5002 (delta 3505), reused 4835 (delta 3338)
Receiving objects: 100% (5002/5002), 2.38 MiB | 7 KiB/s, done.
Resolving deltas: 100% (3505/3505), done.

Checkout the test data

To check out the test data from git, first open a GIT bash prompt as illustrated below:

../_images/msysgit-step10.jpg

The repository can now be cloned by issuing the commands listed below. (Already completed in previous step):

cd  /c/Documents\ and\ Settings/<your username>/.qgis/python/plugins/

git clone https://<your username>@github.com/AIFDR/inasafe_data.git inasafe_data

Note

The items in angle brackets above should be replaced with your personal details as required.

When the final command above runs, you should see something like this in the console when the clone process is completed:

$ git clone https://timlinux@github.com/AIFDR/inasafe_data.git inasafe_data
Cloning into 'inasafe_data'...
remote: Counting objects: 5002, done.
remote: Compressing objects: 100% (1526/1526), done.
remote: Total 5002 (delta 3505), reused 4835 (delta 3338)
Receiving objects: 100% (5002/5002), 2.38 MiB | 7 KiB/s, done.
Resolving deltas: 100% (3505/3505), done.

Install QGIS

Download the latest QGIS ‘standalone’ installer from http://download.qgis.org and install it by running the installation wizard and accepting the defaults throughout.

After opening QGIS (Start ‣ All Programs ‣ Quantum GIS Lisboa ‣ Quantum GIS Desktop (1.8.0)) you need to enable the plugin from the plugin menu by doing Plugins ‣ Manage Plugins and then search for the InaSAFE plugin in the list and enable it.

Windows Caveats

Our primary development platform is Linux (specifically Ubuntu Linux). Some features of the development environment - particularly the Make tools do not run on Windows. Some helper scripts have been written to substitute for make but they do not have feature parity with the make scripts.

Command line environment setup

Create a shell launcher

A command line environment is useful for running unit tests and for developing and testing standalone scripts written to use the InaSAFE libraries.

We will create a custom shell launcher that will give you a python shell environment using the python that comes bundled with QGIS, and that sets various paths and evironment variables so everything works as expected. Save the following listing in <QGIS Install Dir>/bin/python-shell.bat:

@echo off
SET OSGEO4W_ROOT=C:\PROGRA~1\QUANTU~1
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
call "%OSGEO4W_ROOT%"\apps\grass\grass-6.4.2\etc\env.bat
@echo off
SET GDAL_DRIVER_PATH=%OSGEO4W_ROOT%\bin\gdalplugins\1.9
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
path %PATH%;%OSGEO4W_ROOT%\apps\grass\grass-6.4.2\lib
path %PATH%;"%OSGEO4W_ROOT%\apps\Python27\Scripts\"

set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python;
set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\Python27\Lib\site-packages
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT%\apps\qgis
cd "%HOMEPATH%\.qgis\python\plugins\inasafe-dev"
start "Quantum GIS Shell" /B "cmd.exe" %*

Note

The QGIS_PREFIX_PATH environment variable should be unquoted!.

Note

You may need to replace PROGRA~1 above with PROGRA~2 if you are on 64bit windows.

Note

This script is for QGIS 1.8. You may need to do some adjustment if you are using another version of QGIS

For easy access to this shell launcher, right click on the qgis-shell.bat script and (without releasing your initial right click) drag with the file onto your start / windows button in the bottom left corner of the screen.

Verifying your system path

To verify your path, launch your python shell (by clicking the python-shell.bat) and then start a python shell. Now enter the follow simple script:

import sys
for item in sys.path:
    print item

Which should produce output like this:

C:\Users\inasafe\.qgis\python\plugins\inasafe-dev
C:\PROGRA~1\Quantum GIS Lisboa\apps\qgis\python
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\Lib\site-packages
C:\PROGRA~1\Quantum GIS Lisboa\bin\python27.zip
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\DLLs
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\plat-win
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\lib-tk
C:\PROGRA~1\Quantum GIS Lisboa\bin
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\PIL
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\win32
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\win32\lib
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\Pythonwin
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\wx-2.8-msw-unicode

It is particularly the second and third lines that you need to have in place so that the QGIS libs can found. Now dow a simple test to see if you can import the QGIS libs:

from qgis.core import *
exit()

Assuming you get no error messages, you have a functional python command line environment which you can use to test QGIS functionality with.

Nose testing tools

Installing pip

We need to install easy_install so that we can install pip so that we can install nosetests and other python tools. Under windows you need to run a little script to install easy_install and then use easy_install to install pypi. Download the script on this page called ez_setup.py and save it somewhere familiar e.g. c:temp.

Note

If you use windows 32bit, do not download the .exe file as said on the page, but just download the ez_setup.py

Special note for Win64 bit users

If your Windows is 64bit, you need to do a little trick to install pip. There are two options for doing this:

  • Add the following line near the end of your command shell launcher (as described in windows_shell_launcher-label):

    set PATH=c:python27;%PATH%

  • First you have to install python 32bit and add its path to PATH variable in environment variable (on Windows 7: System Properties ‣ Advanced ‣ Environment Variables).

To verify that you have the correct python in your path launch python and print sys.executable - is should show c:python27python.exe as per the demo session below:

C:\Users\inasafe\.qgis\python\plugins\inasafe>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.executable
c:\python27\python.exe
>>>

For both 32 and 64 bit

Next launch the shell (python-shell.bat as described in Command line environment setup) as administrator (by right clicking the file and choosing run as administrator). Then from the command line, launch ez_setup.py by typing this:

python c:\temp\ez_setup.py

Note

You will need to launch the shell as administrator whenever you need to install python packages by pypi.

Now in the same shell, use easy setup to install pip (make sure you have added the QGIS scripts dir to your shell launcher’s - which should be the case if you have followed the notes in Command line environment setup):

easy_install pip

If the installation goes successfully, you should see output like this:

Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Reading http://www.pip-installer.org
Best match: pip 1.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-1.1.tar.gz#md5=62a9f08dd5dc69d76734568a6c040508
Processing pip-1.1.tar.gz
Running pip-1.1\setup.py -q bdist_egg --dist-dir c:\users\timsut~1\appdata\local
\temp\easy_install--zkw-t\pip-1.1\egg-dist-tmp-mgb9he
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.txt' found under directory 'docs\_build'
no previously-included directories found matching 'docs\_build\_sources'
Adding pip 1.1 to easy-install.pth file
Installing pip-script.py script to C:\PROGRA~2\QUANTU~1\apps\Python25\Scripts
Installing pip.exe script to C:\PROGRA~2\QUANTU~1\apps\Python25\Scripts
Installing pip.exe.manifest script to C:\PROGRA~2\QUANTU~1\apps\Python25\Scripts
Installing pip-2.5-script.py script to C:\PROGRA~2\QUANTU~1\apps\Python25\Scripts
Installing pip-2.5.exe script to C:\PROGRA~2\QUANTU~1\apps\Python25\Scripts
Installing pip-2.5.exe.manifest script to C:\PROGRA~2\QUANTU~1\apps\Python25\Scripts

Installed c:\progra~2\quantu~1\apps\python25\lib\site-packages\pip-1.1-py2.5.egg
Processing dependencies for pip
Finished processing dependencies for pip

Installing nose

Nose is a tool for automation of running python unit tests. With nose you can run a whole batch of tests in one go. With the nosecover plugin you can also generate coverage reports which will indicate how many lines of your code actually have been tested.

To install these tools, launch your python prompt as administrator and then do:

pip install nose nose-cov

Running tests using nose

Once they are installed, you can run the nose tests from windows by going to the plugin directory/inasafe-dev folder (in your python-shell.bat shell session) and running:

runtests.bat

Building sphinx documentation

Sphinx is a tool for building documentation that has been written in the ReSTructured text markup language (a simple wiki like format). You can build the sphinx documentation under windows using a helper script provided in the docs directory of the InaSAFE source directory, but first you need to actually install sphinx.

Installing sphinx

Launch your QGIS python shell environment (see Installing pip) as administrator and then run the following command:

pip install sphinx

The cloud-sptheme package installs the sphinx theme we are using.

Building the documentation

To build the documentation, open a QGIS python shell (no need to be admin) and go into your inasafe-dev/docs directory. Now run the following command:

make.bat html

Note

Only the html make target has been tested. To use other make targets you may need to perform further system administrative tasks.

Viewing the documentation

The documentation can be viewed from withing QGIS by clicking the help button on the InaSAFE dock panel, or you can view it in your browser by opening a url similar to this one:

file:///C:/Users/Tim%20Sutton/.qgis/python/plugins/inasafe/docs/_build/html/index.html

Developing using Eclipse (Windows)

Note

This is optional - you can use any environment you like for editing python, or even a simple text editor.

If you wish to use an IDE for development, please refer to this article for detailed information on how to get the basic Eclipse with PyDev setup.

Installing Eclipse

You can download and install eclipse by getting the latest installer at eclipse.org. Just run the installer accepting all defaults.

Installing PyDev

With Eclipse running, click on Help ‣ Eclipse Marketplace and from the resulting dialog that appears, type PyDev into the search box and then click Go. On the search results page, choose PyDev and click the Install button next to it. Agree to the license terms and accept the aptana certificate, then restart Eclipse as requested.

Custom Eclipse Launcher

You need to create a custom Eclipse launcher in order to use Eclipse PyDev. The process is similar to Command line environment setup in that you need to create a custom batch file that launches eclipse only after the osgeo4w environment has been imported. Here are the typical contexts of the file:

@echo off

SET OSGEO4W_ROOT=C:\PROGRA~2\QUANTU~1
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
call "%OSGEO4W_ROOT%"\apps\grass\grass-6.4.2\etc\env.bat
@echo off
SET GDAL_DRIVER_PATH=%OSGEO4W_ROOT%\bin\gdalplugins\1.8
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin;%OSGEO4W_ROOT%\apps\grass\grass-6.4.2\lib
set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python;
set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\Python27\Lib\site-packages
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT%\apps\qgis
"C:\Progra~2\eclipse\eclipse.exe"

Note

Use the path where your eclipse was extracted. Also note that PROGRA~2 may be PROGRA~1 in 32bit windows.

Save this file under <QGIS Install Dir>/bin/python-shell.bat and then right-drag it from explorer to your Windows start button to create an easily accessible shortcut to eclipse.

Creating a project

The procedure for doing this is to do: File ‣ New ‣ Project... and then from the resulting dialog do PyDev ‣ PyDev Project.

In the resulting project dialog, set the following details:

  • Project name: : inasafe
  • Use default : uncheck
  • :guilabel (windows):Directory : C:\Users\<user>\.qgis\python\plugins\inasafe\
  • Choose project type : Python
  • Grammar Version : 2.7
  • Add project directory to PYTHONPATH? : check

Note

The python shipped with QGIS for windows is version 2.7 so you should avoid using any additions to the python spec introduced in later versions.

At this point you should should click the link entitled ‘Please configure an interpreter in related preferences before continuing.’ And on the resulting dialog do:

  • Python Interpreters: New... : click this button

In the dialog that appears do:

  • Interpreter Name : QGIS Python 2.7
  • Interpreter Executable : C:\Program Files (x86)\Quantum GIS Lisboa\bin\python.exe
  • OK Button : click this button

Another dialog will appear. Tick the first entry in the list that points to your:

C:\\users\\inasafe\\Downloads\\eclipse\\plugins\\org.python.pydev
_2.6.0.2012062818\\pysrc

The resulting list of python paths should look something like this:

C:\Program Files\eclipse\plugins\org.python.pydev_2.6.0.2012062818\pysrc
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\DLLs
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\plat-win
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\lib-tk
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\win32
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\win32\lib
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\Pythonwin
C:\PROGRA~1\Quantum GIS Lisboa\apps\Python27\lib\site-packages\wx-2.8-msw-unicode

Click on the New folder button and add the QGIS python dir:

C:\Program Files\Quantum GIS Lisboa\apps\qgis\python
  • OK Button : click this button

You will be returned to the Python Interpreters list and should see an entry for QGIS Python 2.7 listed there. Now do in the Environment tab:

New

In the dialog that appears

Name : QGIS_PREFIX_PATH Value : C:\PROGRA~1\QUANTU~1\apps\qgis

Then click ok to close the environment variable editor.

  • Ok : click this button

Then click finsih to finish the new project dialog .

  • Finish : click this button

Remote Debugging with Eclipse

For remote debugging, you should add pydevd to your PYTHONPATH before starting QGIS. Under Windows, the best way to do this is to add the following line to qgis.bat under C:Program Files (x86)Quantum GIS Wroclawbin:

SET PYTHONPATH=%PYTHONPATH%;C:\Progra~1\eclipse\plugins\org.python.pydev.debug_2.3.0.2011121518\pysrc

Note

(1) You need to add a settrace() line at the point in your code where you would like to initiate remote debugging. After that, you can insert eclipse debugger breakpoints as per normal.

(2) If you are running with remote debugging enabled, be sure to start the PyDev debug server first before launching the Risk-in-a-box QGIS plugin otherwise QGIS will likely crash when it can’t find the debug server.

(3) Place the above PYTHONPATH command before the final line that launches QGIS!

(4) The exact path used will vary on your system - check in your eclipse plugins folder for “org.python.pydev.debug_<some date> to identify the correct path.

To initiate a remote debugging session, add the settrace() directive to your source file and then start the python remote debugging service from the PyDev debug perspective. Then launch QGIS (or your command line application) and use the application until the settrace line is encountered. QGIS will appear to freeze - this is normal. Now switch to Eclipse and you should see the settrace line has been highlighted in green and you can step through the code using standard Eclipse debugging tools (done most easily from the debugging perspective).

Note

Always remove or comment out settrace() when are done debugging!

Running Unit tests from the IDE

Using PyDev’s build in test runner

Python has very good integrated support for unit testing. The first thing you should do after setting up the IDE project is to run the tests. You can run tests in the following ways:

  • For the entire inasafe package
  • For individual sub packages (e.g. engine, gui, storage, impact_functions)
  • for an individual test module within a package
  • for an class within a test module
  • for an individual method within a test class

You can view these individual entities by browsing and expanding nodes in the project panel in the left of the IDE.

Note

If you run the test suite for the entire inasafe package, it will mistakenly treat the sphinx documentation conf.py (docs.source.conf) as a test and fail for that test. This is ‘normal’ and can be ignored.

Setting PyDev to use the Nose test runner

You can also configure Eclipse to run the tests using nose (which is recommended). To do this first do:

Window ‣ Preferences ‣ PyDev – PyUnit

Now set TestRunner to Nosetests and set the following options:

-v --with-id --with-coverage --cover-package=storage,engine,impact_functions,gui

As with using Pydev’s built in test runner, you can also run any module, class etc. while using the nose test runner by right clicking on the item in the PyDev package explorer.

Note

Actually, we can run the test runner until this step. But, we got a problem, so you need to install python in your windows machine.