[gallery link="file"]

Here is what our course attendees were producing after a day on our course.

And here is what attendee Nyarobi Makuru produced today...

Tomorrow is openModeller day. More to follow...

I'm back in Dar es Salaam for a training workshop (2 days QGIS, 1 day openModeller). Sam Lee Pan and Chris Yesson have joined me to help with presentation. Sam introduced herself in an earlier blog post and Chris is an ex-colleague from my time working for the University of Reading, UK. Chris has excellent knowledge of ecological niche modelling and its a real pleasure to have him joining us to share his knowledge with the course attendees.

The course is being held at the Dar es Salaam University's Institute of Technology. We have around 30 attendees who come from academia and state parastatals.

It's always refreshing to present QGIS courses and see how well received the software is and how easy it is for new users to  pick up and get running with quickly.

During the course of the day we covered loading vector data, symbolisation, labelling and basic digitising.

The training course is part of the final stages of the work we have been doing for GBIF. Our work adds several new tools (non spatial table support, raster calculator, CSW client overhaul, darwin core data provider, openModeller plugin) - some of which have been added into the core of QGIS and will be in the upcoming QGIS 1.6 release.

We ended the day with a free form 'map a beautiful map' session and I was really impressed with the quality of work the attendees produced - all of them created nice neat projects with several layers, labels, customised symbology and so on.  I'm going to try to take some screenshots of a few of the better ones tomorrow and post them here for the world to see ... until then, happy QGIS'sing!

I know the little bash scripts I write and post here are popular so here is another one.  The script sequentially unzips worldclim future climate scenario datasets, and then clips them to the bounding box of Tanzania using gdal. After clipping, it removes the extracted files again so you are left with just your original downloaded zip files, and the clipped files in compressed geotif format.

#!/bin/bash
mkdir clip
for FILE in *.zip
do
  unzip $FILE
  cd 30s/
  for BIL in *.bil
  do
    CLIP=../clip/$(basename $BIL .bil).tif
    gdal_translate -of GTiff -co COMPRESS=LZW -co TILED=YES -projwin 29 0 40 -11 $BIL $CLIP
    rm $BIL
  done
  cd ..
  rm -rf 30s
done

A common activity in GIS is to mask off parts of an image such that it has transparent pixels in the masked area. We can do this in QGIS too with a bit of help from gdal. Lets look at this niche model I ran for Anopheles gambiae:

What I really want is all the parts of the model that fall outside of the borders of Tanzania to appear transparent. We will use a vector layer to delineate the masked area. I am going to make a mask for Tanzania. To make the mask from the vector layer we can use the gdal rasterize tool. But before we can do that we need to deal with the fact that rasterize is a bit awkward (in current version of GDAL) in that the image you are rasterizing must pre-exist. The simple work around is to just make a copy of my raster that I am trying to make a mask for using gdalwarp. I am using warp with the cutline option, in so doing assigning all cells outside of my country of interest to have a value of nodata. Unfortunately no gui tool exists for this yet so you need to run the command from the command line:

  gdalwarp -of GTiff -dstnodata 0 \
  -cutline Boundaries/Tanzania.shp Anopheles_gambiae__projection.tif maskbase.tif

|The first step to making our mask - all the non TZ areas are assigned a null value.|

Next I will load my vector layer containing the Tanzanian political boundary into QGIS:

Now I will use the Raster -> Rasterize tool to create the next phase mask - to fill the inside of the mask with the same value. Note that it will overwrite the contents of maskbase.tif - thats ok since we created maskbase specifically for that purpose. You can choose any attribute field for the mask, it doesn't really matter since we will be replacing all no data values in a few steps time:

gdal_rasterize -a POP_CNTRY \
-l Tanzania \
/home/timlinux/gisdata/Africa/Tanzania/MasterDataSet/Boundaries/Tanzania.shp \
/home/timlinux/gisdata/Africa/Tanzania/MasterDataSet/maskbase.tif

After it has run we should have something like the image below. Note how the areas outside the country borders are transparent, while the areas inside all have the same value (in my case 222).

Thats great, but I want a value of 1 inside all of the country so I use the shiny new raster calculator tool that will be in QGIS 1.6:

Layer -> Raster Calculator

And then create a simple boolean expression to assign all cells of value greate than 1 a value of 1.

Maskbase@1 > 1

I've called the resulting file mask.tif. I'll be throwing maskbase.tif away when I am done since is is a temporary working file. Note that the boolean (< and >) operators in the raster calculator are undocumented, and don't appear as buttons on the user interface - you will need to type the expression manually. Buttons for these will be added in QGIS 1.7. In the screen shot below if you look at the Value Tool dock window (ValueTool is a great plugin!), you will see that the final mask has a value of 1 where the cursor is placed while the maskbase layer has a value of 222.

Ok so now that we have a mask, we can apply it to arbitrary rasters to make areas of the raster transparent e.g. (Using Layer -> Raster Calculator again):

Mask@1 * Anopheles_gambiae__projection@1

Which produces this:

Some afterthoughts:

There are always many ways to do things with computers in general and with FOSSGIS in particular - and after writing this I thought of some ways to reduce the number of steps ... I'll leave it to my readers to post in the comments if they have other more efficient ways of going about this activity.

Also just a note that I am using the QGIS raster calculator from SVN with a patch applied from Marco for transparency support. The patch will make its way into SVN in the next day or two and the raster calculator in general will arrive in 1.6 release of QGIS (or grab a nightly test build).

Update:  The patch mentioned above is now in SVN trunk.

Just an update to let you know that I have recently joined Linfiniti. I was lucky enough to be an intern earlier in the year, and am happy to be part of the team again, and helping in bringing Open Source GIS to Africa and other parts of the world :)

A quick introduction - my name's Samantha Lee Pan (sam for short).  I have an undergrad in environmental & geographical science (with a second major in oceanography!)  I've done my honours in GIS.  I am also looking at doing a masters degree in the environmental field.

There is still a lot that I am learning here at Linfiniti and been very busy with a host of different projects. But I'm sure you will be hearing more from me, as I find useful bits of info or updates on the FOSS scene - and make time to post them here!

keep well

-sam

I spent quite a few frustrating hours trying to find a way to get TinyMCE to be used as the default control in textareas in my django admin interface. I know you can override individual admin forms to do this but there seems to be precious little written about how to do this globally for all admin forms.

I also (obviously) wanted to avoid any solution that forks django itself or satchmo (which I am using in this case too). Finally I found this nice way to do it by overloading the django admin template! Thanks Benshr!

Totally unrelated to GIS but a handy tip for linux users out there who want to merge several pdf's into a single document. You can do it from the command line like this:

`` convert -quality 90 -density 300x300 tim_sutton_cv_2010.pdf sam_cv_2010.pdf cvs.pdf``

The quality option is the jpeg compression level to use. The density option is important otherwise the resulting merged pdf will look crappy. Basically it determines the dpi of the output pdf (in my example above I'm putting out a 300dpi document). The merged file size will grow with higher density levels.

Then just give a list (in order of appearance) of pdf's to be merged and the final item in the list is the name of the output pdf.

One gotcha to note: convert rasterises the pdfs so you will no longer be able to select text in the pdf.

If you don't have imagemagick (which provides the convert command line tool) on your system (c'mon its an essential tool!) you can install it like this:

`` sudo apt-get install imagemagick``

Update: After posting this entry, Volker Mische wrote to me recommending pdftk (its in apt) which is easy to use and doesnt rasterise the pdfs when fusing them:

`` pdftk tim_sutton_cv_2010.pdf sam_cv_2010.pdf cat output cvs.pdf``

Some Background

A few months ago I received a nice email from Dr Serena Coetzee, University of Pretoria, who had read about some of our outreach activities to local universities:

Dear Tim,

Your email about FOSSGIS at the university prompted me to write this email.

As part of their year-long software engineering course, our third year students have to do a project. Prospective 'clients' submit project proposals (RFPs) to the lecturer, Morkel Theunissen, and the students have to write proposals for one or more projects. Clients receive these student proposal(s) and select the team with whom they are prepared to work. Morkel describes the whole process in the attached PDF. I think it would be an excellent idea to have an RFP for some ring-fenced functionality in QGIS.

Such a project would give students exposure to the 'insides' of FOSSGIS and promote FOSSGIS in the student community: apart from the team working on the project, the lecturers, tutors and the rest of the class will also see demonstrations of the project. It is also an opportunity to 'check out' students as potential future employees. On the research side, there are also opportunities: if students enjoyed the third year project and continue with Honours or even a Masters, it is possible to take the project further or to at least continue in that area of expertise.

For the students this would be a good opportunity to get real-life experience of working in an open source software development team. I discussed the idea with Morkel, who is busy with PhD research related to open source software engineering, and he was quite keen on the idea. In his call for project proposals, he puts a deadline of 25 January for the RFPs, but is happy to give us a few days extension.

Also attached is a proposal that I submitted this year - just to give you an idea of the length of proposal and the level of detail required. Some are even shorter than mine and the students have to fill in the details in their functional and technical specifications as part of the course.

Looking forward to hear from you.

Regards, Serena Coetzee

(with apologies to Serena for replicating her email verbatim).

My friend Wolfgang Luck and I then came up with a few proposals of interesting projects that could be undertaken in QGIS (there is an endless supply really :-P ). One of those, the implementation of an AGCP tool was selected by a group of students...

AGCP

Most FOSS GIS users I meet have not heard of AGCP or Automated Ground Control Point collection. GCP's are use to provide pixel-to-map referencing for an image so that it can be georeferenced or orthorectified. The process usually entails using a tool such as the QGIS georeferencing plugin to manually select a point in a georeferenced control image (e.g. a road crossing or other prominent feature). The same feature is then identified in the image being georeferenced. When you have enough of these Ground Control Points, you can perform some kind of transformation on the image being georeferenced, such that when loaded in your GIS software it appears in the correct geographical space.

The caveat with the aforementioned process is that it is a highly manual, time intensive and error prone process. In the commercial world software exists (such as PCI Geomatica) that can automate the collection of these GCP's, but as far as I know, no such algorithms are available to us FOSSGIS folks. Thats about to change soon...

A group of students from the University of Pretoria (South Africa) has been working on a project to provide just this capability in QGIS. They are being mentored by Wolfgang Luck and Rian van den Dool from the Hartebeeshoek Satellite Application Centre and myself (from Linfiniti). The students working on this (in no particular order) are:

James Meyer, Laurette Pretorius, Christoph Stallmann, Francois Maass, Dirk Ras

They have already posted a patch against QGIS trunk with a preview version of the work they are doing. The software basically composes two parts:

Their work plan includes support for orthorectification (as opposed to simply georeferencing) of the unreferenced image. Georectification provides a higher fidelity of the processed image to its surrounding geographical space since it does not simply perform an affine transform to place the image in geographic space. Rather, orthorectification takes into account things like the sensor model (including lens distortion, CCD characteristics and so on) and local topography to 'rubbersheet' the image into place. Georectification support is not yet implemented in their patch so look out for that in a future revision of their patch.

FOSSGIS is a great platform for an academic environment, not just for teaching end-users, but for providing a platform where technical students can delve deep into the inner workings of the software and gain fantastic experience and skills. Its doubly satisfying that the work that the students can actually become part of the software and they can provide real benefit to others from their work (instead of it just disappearing into the ether as many academic projects do).

I know there are many people out there who would benefit from their work, so I would like to urge anyone interested in their work to make comments, encouragement and suggestions to them via the trac ticket for their patch.

Here is a little taster of what you can expect in the GUI when their work is completed:

After some discussion and planning, Gavin Flemin (http://www.afrispatial.com), Zoltan Szecsei (http://geograph.co.za) and my company, Linfiniti Consulting have come together to create The FOSSGIS Academy.

FOSSGIS Academy was formed with the aim of creating a formalised FOSS GIS training curriculum to provide the needed injection of FOSSGIS skilled workers into the geospatial workplace. We aim to provide training courses in a professional environment at a reasonable price to show you how to make the best of all the great FOSS GIS tools out there.

We hope this will be an avenue into removing the perception that FOSSGIS is great but lacks the support and training infrastructure that commercial vendors provide. Over time we will build up a range of courses covering entry level desktop GIS, advanced desktop GIS (GRASS etc), geospatial data stores (PostGIS), web services (Mapserver etc), web mapping (OpenLayers etc) and FOSSGIS developer skills (geodjango, QGIS python, QGIS C++ etc).

We already have our first two courses lined up:

• QGIS & Postgis
• Mapserver, OpenLayers and Geodjango

We hope they will be the first of many! See The FOSSGIS Academy for more info!

Carson Farmer (of manageR fame :-) this week submitted a patch for QGIS for evaluation. His patch adds three more class breaks strategies for graduated symbology in vector data: Pretty, Jenks (Natural Breaks) and StdDev. He explains the rationale behind this patch in his blog post here. I really like this new functionality (which should make its way into trunk soon). I made a couple of screen shots that will make it easier to visualise the different classifiers he has created:

What's nice about the Jenks and Pretty breaks is that they create much more humanistic value ranges - it's easier to identify with a value range of 6450000 - 11100000 than it is for something like 15394734.4297 - 20055646.2193

Extracting Value from Consultants (subtitled: How to Hire, Control and Fire Them) is a book my sister Jenny and her husband Gordon recently wrote and published. The book is available on Amazon here. Usually I don't read 'business books' (I read tech stuff online or novels for pleasure), but since I was presented an autographed, free copy, I thought I would give it a read!

The book has two target audiences - those looking to hire consultants, and consultants looking to do a better job of servicing their customers. I kind of fall into both categories because I provide consulting services for others, and also make use of consultants to help me with my work. My context is a little different since in my context 'consultant' is really more 'IT Contractor' but the information in the book remains relevant regardless.

As I mentioned I don't usually read business books and so was expecting to be lapsed into a comatose state by a dry, jargon ridden tome. But I found the book to be quite the opposite - the book is matter of fact and extremely logically laid out and I found myself constantly nodding in agreement with the ideas presented. By the end of the book I was impressed by the easy narrative and the concise way in which the consulting process had been presented. It leads you through the process of deciding if it is appropriate to use a consultant, how to select a consultant, how to enter into a contract that will 'cover your butt', how to prevent the consultant engraining themself so deeply into your business that you can never be shod of them, and how to dissolve the partnership when the project comes to an end.

I learned a lot which I will try to apply to my own business albeit on a smaller scale than the multinational corporation perspectives presented in the book. Personal bias aside (given that the authors are family members), I heartily recommend the book to anyone who is planning on using consultants or contractors, or offering services as a consultant or contractor. It will give you plenty of food for thought and for the small investment in time required to read it will really help you to manage the process more effectively.

I have made previous posts about the download tallies for QGIS - here is an update:

Unfortunately we only have decent stats for the Windows standalone installer so Linux, Mac and Osgeo4W based installers are not factored in here. Also we only have stats since the Enceladus release. I'll post occasional updates as time goes by, but to me is looks like with each release we are increasing our user share (though we will only know for sure in 5 months time).

This week we have taken on our first employee! Samantha Booysen will be working as office manager. She will keep things running smoothly so that the geek in residence can focus on coding. She is all set up with an Ubuntu computer to do her work on and we will include her on our training courses so that she starts to learn about FOSS GIS too. Samantha doesn't have an IT background but she is very interested in social upliftment which is something close to our hearts here at Linfiniti so we are sure she will fit right in. Welcome Samantha, we hope you enjoy working here!

I spent most of last week in Dar es Salaam, Tanzania. A lovely tropical country in the heart of Africa. I was there as part of a project I am working to create tools for Biodiversity Informatics practitioners. Of course the tools are based on Free Software:Quantum GIS and openModeller.

The attendees at the workshop were entertained by my talk about what FOSS is and why it is important, an introduction to QGIS slideshow (superbly presented by Marco Hugentobler), and ending with a tour of openModellerDesktop. We also did some live demonstrations of QGIS and openModeller, before going on to discuss details about how these tools can be used to support their Biodiversity Informatics workflows.

The meeting was funded by the Global Biodiversity Information Facility (GBIF) with Juan Bello as their representitive, and hosted by the Tanzanian Commission for Science and Technology (COSTECH).

In case you are unfamiliar with the aims of GBIF, they are facilitating the digitisation (or digitization for our american readers) of the worlds biodiversity records - herbarium records, museum collections and so on. COSTECH provides the local infrastructure and staff for the 'TanBif' node in Tanzania.

The meeting also included 'in-country' experts in the fields of GIS, Meteorology, Ecology, IT and so on. I think for all of the attendees, the concept of FOSS was a real eye-opener. African economies can't compare with those in Europe and the USA and the capital outlay for proprietary software that presents an irritation in the Western world is a major burden in the third world. So just knowing that they could dive in and use QGIS was a great revelation.

We finished our workshop a little early on the Friday so Marco and I offered to go along to the COSTECH offices and geo-enable their PostgreSQL species occurrence database and install QGIS on their desktop PC's running Windows XP. In the space of a couple of hours we were done - the major part of which was spent showing the TanBif staff members how to bring up the PostGIS layer in QGIS, perform simple queries and make maps. Having spent days in the past trying to get proprietary software like Oracle and Arc*** configured, optimised, licensed and generally usable, I was struck by just how easy and quick it is to get someone up and running with a robust enterprise ready PostGIS geospatial datastore and a user friendly Free Software desktop GIS like QGIS.

Thanks to the friendly Tanzanian folks for their hospitality - I look forward to my next visit! Here are some piccies from the trip...

Juan Bello telling us about the cool things you can do with a good Biodiversity Information repository.

The workshop attendees (Marco and Juan out of shot)

Marco showing Godfrey how to use QGIS to bring up their PostGIS Biodiversity dataset.

Godfrey proudly showing off his first map (made with QGIS)!

Marco killing a mosquito - he became something of an expert!

Update 11 June 2011:I have updated this article to call what was previously referred to as 'QGIS Mapserver' by its new official name: QGIS Server. The name was changed to prevent confusion with the UMN Mapserver product.

Marco Hugentobler recently included his QGIS WMS Server work into QGIS trunk. You may have noticed QGIS being mentioned in Paul Ramsey's post about the WMS Shootout at OSGEO2010 and thought to yourself "huh? whats that?". Well let me introduce you....

The QGIS mapserver project implements a WMS server on top of the QGIS

libraries (funded by the EU projects Orchestra, Sany and the city of Uster).

• Marco Hugentobler

sudo apt-get install libfcgi-dev

/home/timlinux/apps/bin/qgis_mapserv.fcgi

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
  AllowOverride None
  Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
  Order allow,deny
  Allow from all
  # Next for QGIS Mapserver
  SetEnv LD_LIBRARY_PATH /home/timlinux/apps/lib
</Directory>

cd /usr/lib/cgi-bin/

sudo mkdir world
cd world

sudo ln -s /home/timlinux/apps/bin/qgis_mapserv.fcgi .

sudo cp /home/timlinux/gisdata/QGISProjects/world.qgs .

sudo cp /home/timlinux/dev/cpp/qgis/src/mapserver/wms_metadata.xml .

http://localhost/cgi-bin/world/qgis_mapserv.fcgi?SERVICE=WMS&
    VERSION=1.3.0&REQUEST=GetCapabilities

http://localhost/cgi-bin/world/qgis_mapserv.fcgi?SERVICE=WMS&VERSION=1.3.0&
REQUEST=GetMap&BBOX=-36.215710,6.817432,-9.700450,58.551008&
CRS=EPSG:4326&WIDTH=1077&HEIGHT=552&LAYERS=Continent,Country,Cities&
STYLES=,,&FORMAT=image/jpeg&DPI=96

<properties>
<WMSEpsgList type="QStringList">
<value>21781</value>
<value>4326</value>
</WMSEpsgList>
</properties>

<properties>
<WMSExtent type="QStringList">
<value>692000</value>
<value>241500</value>
<value>700100</value>
<value>249000</value>
</WMSExtent>
</properties>

http://yourserver/cgi/qgis_mapserv.fcgi?map=/home/qgisprojects/myproject.qgs&SERVICE=WMS&
REQUEST=GetLegendGraphics&FORMAT=image/png&LAYERS=hydrology&WIDTH=400

Horst and I are spending the week up in Johannesburg at the Satellite Applications Center in Hartebeeshoek. We are doing yet another week long training course (I hope I'm not working the poor guy too hard :-P ). This time we are doing:

Tomorrow we start with the PostGIS component. Horst and I have been compiling some course notes for the PostGIS module which we are making available to the world as per usual. The pdf still has some rendering issues - we are aware of that. The document tries to walk the reader through the basics of using SQL and then some basic activities with PostGIS and working with geometries.

I hope some of you out there find it useful - let us know if you do! Also if you have any improvements to make, we'd love to hear from you.

Here is a quick pic or two from the course:

I am always up for a little bit of eye-candy on my desktop. Here is a great theme and icon set:

sudo add-apt-repository ppa:tiheum/equinox
sudo apt-get update
sudo apt-get install gtk2-engines-equinox equinox-theme \
       equinox-ubuntu-theme faenza-icon-theme

Taken from the original article here.

Today was a great day. Gavin Fleming (http://www.afrispatial.co.za/), Horst Duester (visiting FOSSGIS guru in residence) and myself presented a one day mass QGIS workshop. There were around 110 people present! It was a feat of logistics on the part of Gavin, his wife Bridget and the IT staff and students from the Bridge House school (which played host for the day). 60 pcs running Windows were set up and had Jurgen Fischer's excellent all-in-one windows installer for QGIS installed on them. People sat 'two up' at each computer and watched us walk QGIS through the process of loading vector and raster data, digitising data, doing some simple analysis, and producing a print ready map.

A nominal fee or R150 (around USD15) was charged to cover lunch and tea. The remaining profits were donated to the school to sponsor the purchase of some sensors for monitoring river water quality in a local river. The presentors and organisers all donated their time. Those attendees that couldn't afford the R150 charge were admitted for free.

To be honest when Gavin proposed admitting around 80 people into the day workshop rather than the originally planned 20 Horst and I thought he was mad (little did we then know the actual turnout would hit 110). But I have to say Gavin and Bridget did a brilliant job and it was such a thrill to look out over a sea of interested and attentive faces all absorbing every morsel of knowledge about QGIS and FOSSGIS they could glean. It definitely left me feeling like it was a worthwhile experience and feeling positive about the demand and interest for FOSSGIS in the Western Cape.

I am off to crash for the night now after a long day, but I will leave you with these pics:

(above photo courtesy H. Duester)