So QGIS Server didn't win the performance shootout at FOSS4G2011...sniff. However there is no need to be alarmed - if you hide QGIS Server behind a tilecache you don't need to let your users wait around for their maps to render. I hadn't tried QGIS Server with TileCache and was curious to know how well it [...]

Lately I have been delving deeper into the (aweful) world of Microsoft Windows - specifically looking to be able to better support Windows users trying to get to grips with running FOSSGIS on that platform. I have mentioned before on this blog how dependent I am on GDAL for my daily work. GDAL is also [...]

I've just been chatting to Aaron Racicot and Pirmin Kalberer on IRC to hear how things went during their workshops at FOSS4G in Denver. Aaron and co-conspirator Greg Corradini have put together a really nice training resource on building python plugins in QGIS which is available at www.qgisworkshop.org. The source of their workshop notes can be forked [...]

Valgrind is a nifty too to do code instrumentation - tell you how much time gets spent in different parts of your code tree. KCachegrind is a visualisation tool for valgrind outputs. They are two separate tools that together can provide useful insight into where you code is slow or doing too much work. To [...]

There is quite an old ticket in the QGIS bug tracker relating to the need to be able to select which GDAL drivers to use. The issues is this: GDAL in some cases provides multiple drivers for the same image type. For example, JPEG2000 datasets can be opened using ECW's proprietary driver, KAKADU (gotta love that name!), OpenJPEG etc. The problem is that there isn't actually any way to choose which driver GDAL should use for a given image format - it works on a first come, first served bases. This is a bit of a problem sometimes when you actually want to be using a different driver. In some work I did recently for SANSA (the South African Space Agency), I built a custom version of QGIS for them. They are particularly keen on the JPEG2000 format since it is an open standard and provides good compression. However the custom build of QGIS I made for them includes the ECW JP2000 driver which unfortunately chokes and dies (no fault of GDAL, it happens down in the ECW code itself) when trying to open their JP2 imagery. Also we would prefer to use a FOSS driver for JP2 in keeping with the general ethos of being a FOSS project. So all this prompted me to implement the needed support to let you disable GDAL drivers at runtime in QGIS which is illustrated a little by the screenshots below.

By default GDAL want to use the ECW JPG2000 driver...(click to enlarge)

Using the options dialog, you can now disable a driver....(click to enlarge)

After the driver is disabled (and restarting QGIS), the alternate JP2 driver is available (click to enlarge)

There is quite an old ticket in the QGIS bug tracker relating to the need to be able to select which GDAL drivers to use. The issues is this: GDAL in some cases provides multiple drivers for the same image type. For example, JPEG2000 datasets can be opened using ECW's proprietary driver, KAKADU (gotta love that name!), OpenJPEG etc.

The problem is that there isn't actually any way to choose which driver GDAL should use for a given image format - it works on a first come, first served bases. This is a bit of a problem sometimes when you actually want to be using a different driver. In some work I did recently for SANSA (the South African Space Agency), I built a custom version of QGIS for them. They are particularly keen on the JPEG2000 format since it is an open standard and provides good compression.

However the custom build of QGIS I made for them includes the ECW JP2000 driver which unfortunately chokes and dies (no fault of GDAL, it happens down in the ECW code itself) when trying to open their JP2 imagery. Also we would prefer to use a FOSS driver for JP2 in keeping with the general ethos of being a FOSS project. So all this prompted me to implement the needed support to let you disable GDAL drivers at runtime in QGIS which is illustrated a little by the screenshots below.

Here is a handy tip if you are building QGIS under windows using our official INSTALL documentation. The document recommends that you open MSVC Express and then right click the BUILD_ALL and INSTALL targets. I find using a graphical user interface for this kind of thing tedious and so I looked around for a way to carry out the build from the command line. Here is how to do it:

cmake --build . --config RelWithDebInfo
cmake --build . --target install --config RelWithDebInfo

You should run those commands from within your build directory. I made a little batch file called INSTALL.bat containing the above two lines which I just call when I want to build in windows now.

I have blogged about the awesomeness that is QGIS Server before. Maybe you have seen some of the nice maps that Andreas Neumann has made in Uster, Switzerland. Over the past 3 months we were contracted to help prototyping a new mapping service for the Norwegian Trekking Association. We really got a chance to put QGIS Server (and the new QGIS Web Client project) through its paces. Why not take a little look for yourself. We had quite a lot of fun optimising the client's data set for good performance and generally making sure everything works smoothly.

Note:

- the site is still a test site, but the fine folks from Norway gave me the OK to let people take it for a whirl.
- we have not yet implemented search support.
- We made a bunch of different print layouts for you to try out!

It's now incredibly easy to get your QGIS cartography online. In the near future I plan to test out techniques for tilecaching with QGIS Server and once that works it will be the end of the line for my 11+ year love affair with UMN Mapserver for the most part I think.

Get your map on! QGIS Server in action (click for full image)

I have bloggedabout the awesomeness that is QGIS Server before. Maybe you have seen some of the nice maps that Andreas Neumann has madein Uster, Switzerland. Over the past 3 months we were contracted to help prototyping a new mapping service for the Norwegian Trekking Association. We really got a chance to put QGIS Server (and the new QGIS Web Client project) through its paces. Why not take a little look for yourself. We had quite a lot of fun optimising the client's data set for good performance and generally making sure everything works smoothly.

Note:

It's now incredibly easy to get your QGIS cartography online. In the near future I plan to test out techniques for tilecaching with QGIS Server and once that works it will be the end of the line for my 11+ year love affair with UMN Mapserver for the most part I think.

Otto Dassau and I recently wrote and article for GeoInformatics about the work we have done in in Tanzania and the QGIS project in general. The article is now online here.

Otto Dassau and I recently wrote and article for GeoInformatics about the work we have done in in Tanzania and the QGIS project in general. The article is now online here.

QGIS got another useful feature a few days ago, courtesy of Alexander Bruy. Here's the commit.

What this lets you do is to find out what projection your rasters are in. It then exports this to a .WLD file. It also allows you to create .PRJ files.

If you have GDAL Tools installed (and activated), you'll find this new tool under Raster > Projections > Extract Projection. It gives you this efficient-looking dialog to choose your options in:

The "extract projection" dialog.

The batch- and recursive modes are a great feature as well. You could easily find yourself in a situation where you have a whole project full of rasters, with undocumented projections, in countless subfolders, that got handed to you without comment. Whereas previously that would be a hassle, now you can just point this tool to the right directory and set it loose. Voila, instant projection information! And in your format of choice, no less. Ladies and gentlemen... a round of applause!

QGIS Server (the 'map' part of the name has been dropped) is an awesome way to publish maps and you can make really good looking cartographic products with QGIS. I have been building out a good looking online map for a client and I suddenly realised that we don't have any way to do road shields. This weekend I hacked together basic support for label shields. It integrates with the labelling-NG tool in QGIS master. Currently it provides only basic options - distance from label at which to start the shield and the shield colour. Only rectangular (with slightly rounded corners) shields are currently supported. In the future we could do lots of other fancy shield types and allow setting outline colour, use a gradient for the background, different shapes and so on. In it's current rather basic configuration though it suits my immediate needs and probably I will add to it as the need arises.

Shield options (click for larger view)

Yes the screenshot is taken in windows - something for a future blog post Here is a little sample of what you can do with shields  (click for larger image):

Some contrived examples of using the label shields

If you want to try it out, it is in my github clone here:

git remote add timlinux git://github.com/timlinux/Quantum-GIS.git
git fetch timlinux
git branch --track shield-labels timlinux/shield-labels

This week Marco Hugentobler (http://sourcepole.ch) committed a nice new feature to QGIS master branch: Line pattern fills. The default selection of hatches that can be used to fill polygons in QGIS is quite limited. Of course you can always use SVG symbols to extend that, but Marco has added another option. With the line pattern fill symbol style, you can create multiply symbol layers of pattern fills with different rotations, allowing you to create a wide range of fill styles. Rather than try to describe this, its easier to demonstrate with a few images.

Of course these new layer styles can be combined with existing layer styles (SVG Fill, Simple Fill etc.) to extend the already huge range of possibilities for symbolization in QGIS. Thanks Marco for another great feature!

Today I committed changes to QGIS master that I started working on at the Lisbon hackfest and finalised this week (along with a unit test yay!). The changes implement improvements to the way in which statistics are gathered for rasters. First a little history: the raster stats code in QGIS is some of the earliest work I have done in QGIS. The code was also hugely inefficient - basically it completely scans every pixel in the file twice! It does this because it extracts minima, maxima, stddev etc. from each band in the raster - and the algorithm used to calculate the stddev requires first calculating the sum of means, hence the second pass. Of course we use the underlying GDAL library to traverse the raster, but we completly bypassed the GDAL stats gathering functions. Now from my testing (see below) the difference of gathering stats in QGIS as opposed to directly in GDAL was not that great. However, the real killer is the fact that QGIS had no reasonable implementation to cache these statistics - either within a session or between sessions. This resulted in quite a lot of  complaints about the slow loading of rasters in QGIS and the spontaneous recollection of stats when making various changes to raster symbology etc.

With the release of 1.7, Radim Blazek implemented 'proper' support for raster providers so that they follow a similar model to the vector providers in QGIS. This opens up room for provider specific optitmisations and for implementing new providers in the future e.g. for PostGIS Raster, Terralib etc. At the Lisbon hackfest, I refactored the provider interface so that it provides a default implementation of stats gathering which can be overloaded by specific providers, each then being able to use best practice for that datasource when gathering stats. I then updated the GDAL Provider to use gdal native calls to fetch statistics.

When GDAl gathers stats, it stores them in a file named after the source raster but with a .aux.xml extension. As a bonus, GDAL also stores histogram data in this file. Thanks to some excellent tips from Etienne (Sky?) on the QGIS mailing list my implementation was further improved so that it actually uses these cached stats from GDAL properly. So let's see how performance is improved with these changes. I tested using a ~650mb tif hillshaded DEM from one of my clients.

 Added bonus: My changes also use a more efficient approach for obtaining min/max for a band (before all stats are gathered) and also histgram calculations are cached now. So after the initial gathering of stats and histograms, all activities when working with GDAL rasters should be near instant now. By the way the 1 in the graph above is rounded up - in my usage it was near instant in most cases.

I need to still evaluate whether it will be good to backport these changes to the 1.7 stable tree, so for now if you want to benefit from these changes, you need to be using a nightly build or build from source.

Every 6 months we hold a hackfest / QGIS meeting and many of our users generously donate towards it. I hope the above article gives those who do support our meetings a sense of the tangible benefits that arise from these meetings - there have been many other improvements coming into master that arose from the last hackfest that we all get to enjoy! If you would like to help towards the costs of the next hackfest, visit http://qgis.org and click the 'Donate!' button on the left hand side. Have fun!

I just got back from another week away, training people to use QGIS in Tanzania. There are few things more rewarding in life for a QGIS developer that looking out over a class of 30 trainees new to GIS (and QGIS) eagerly absorbing everything you have got to show them about your favourite FOSS GIS.  This was my second time running a training course in Dar Es Salaam and my third visit to Tanzania. What is really nice is that two of the attendees from the previous course I ran (Dr. Edward Kohi and Fadhili Bwagalilo) acted as training assistants this time around. It was nice on two counts. Firstly, the fact that they are now comfortable in a mentoring role for other users shows that the last training course was successful. Secondly, it was really great to have their help - anyone who has ever tried to train 30 demanding people in a computer lab for three days solid will attest to how tiring it is and how valuable it is to have roaming helpers in the room to absorb some of the queries that one gets bombarded with. Thanks Edward and Fadhili!

Hulda Gideon (TanBIF representitive), training assistant Fadhili Bwagalilo, myself and Dr. Edward Kohi (training assistant). (Click for larger image)

During the course we covered basic concepts of GIS, basic usage of QGIS and introduced the openModellerPlugins for QGIS that we wrote under contract to GBIF / TanBIF last year. Needless to say I also gave my introductory lecture on how wonderful the concept of Free Software and how bad proprietary software is. The latter really resonates with audiences in developing nations. The costs of proprietary software leave many intellectually  out in the cold. The ability to share FOSS GIS software freely and learn on an open system offers a real leg up in the world to someone who would otherwise be struggling to solve complex problems with an extremly limited set of tools.

A class full of QGIS converts! (click for full size image)

I really look forward to doing further work in Tanzania and other African countries in the future.  If anyone has need of my services in their country, please feel free to contact me. In the future I hope to hold some training workshops for previous course attendees with the idea of introducing more advanced concepts and getting their knowledge to the point where they are comfortable conducting their own training sessions. With this model, before we know it every little village in Africa that has a computer could be putting QGIS to good use to solving problems that have a spatial twist to them! A big thanks to my hosts COSTECH and TanBIF, and personally to Hulda Gideon for working all the behind the scenes magic! I leave you with something to ponder while you sit in your airconditioned office in some cold place....

Another beautiful sunset over the ocean near Dar Es Salaam...

I just got back from another week away, training people to use QGIS in Tanzania. There are few things more rewarding in life for a QGIS developer that looking out over a class of 30 trainees new to GIS (and QGIS) eagerly absorbing everything you have got to show them about your favourite FOSS GIS.  This was my second time running a training course in Dar Es Salaam and my third visit to Tanzania. What is really nice is that two of the attendees from the previous course I ran (Dr. Edward Kohi and Fadhili Bwagalilo) acted as training assistants this time around. It was nice on two counts. Firstly, the fact that they are now comfortable in a mentoring role for other users shows that the last training course was successful. Secondly, it was really great to have their help - anyone who has ever tried to train 30 demanding people in a computer lab for three days solid will attest to how tiring it is and how valuable it is to have roaming helpers in the room to absorb some of the queries that one gets bombarded with. Thanks Edward and Fadhili!

During the course we covered basic concepts of GIS, basic usage of QGIS and introduced the openModellerPlugins for QGIS that we wrote under contract to GBIF / TanBIF last year. Needless to say I also gave my introductory lecture on how wonderful the concept of Free Software and how bad proprietary software is. The latter really resonates with audiences in developing nations. The costs of proprietary software leave many intellectually  out in the cold. The ability to share FOSS GIS software freely and learn on an open system offers a real leg up in the world to someone who would otherwise be struggling to solve complex problems with an extremly limited set of tools.

I really look forward to doing further work in Tanzania and other African countries in the future.  If anyone has need of my services in their country, please feel free to contact me. In the future I hope to hold some training workshops for previous course attendees with the idea of introducing more advanced concepts and getting their knowledge to the point where they are comfortable conducting their own training sessions. With this model, before we know it every little village in Africa that has a computer could be putting QGIS to good use to solving problems that have a spatial twist to them! A big thanks to my hosts COSTECH and TanBIF, and personally to Hulda Gideon for working all the behind the scenes magic! I leave you with something to ponder while you sit in your airconditioned office in some cold place....

Courtesy of Martin Dobias, QGIS master/trunk now sports a snazzy new data browser! You'll find it under View > Panels > Browser. It does what you'd expect: allows you to navigate around on your hard drive and add any data you find. It even shows (by way of an icon, adjacent to the file name) what kind of dataset you're looking at. If you've got WMS layers set up, it seems to allow you to add those as well (although I haven't taken that for a spin yet). Never again will you need to Add Vector Layer, then Add Raster Layer, then Add WMS Layer, etc. Now you can do it all from one handy panel right in the main window!

There it is, on the left. Behold it in all its browserly glory

So, as seen in my previous post on customizing the QGIS UI, it's now possible to completely customize the QGIS interface, adding or removing items as you choose. You can save some layouts, and apply them as needed. It all comes built into QGIS, presented in a handy visual interface. This lets you check and uncheck boxes to activate and deactivate various items on the screen. Which is great, if your own machine is all you need to worry about.

But let's say you're in a situation where there's more than one user. Say, for example, that you're teaching a class on GIS concepts. You want all the students to keep up with what you're doing, so you'd like only the tools necessary for that class to be on the screen. In fact, you want everyone's QGIS to be configured exactly the same way as yours, and in more ways than just the UI.

In that case, there are two handy files to know about: QGIS.conf and QGISCUSTOMIZATION.conf. In Linux, you'll find them under /home/your_user_name/.config/QuantumGIS/. (If you've done something interesting with your install, or you're using another OS, you may have to search for those two files.)

When QGIS loads, it takes a look at those files to configure itself. What's great about this is that, for the most part, these files are written in plain English. The format goes more or less like this:

[Customizable Thing]
aspect_of_customizable_thing=some_property

Or, to name an example:

[locale]
userLocale=en_ZA

With some exceptions, these are all easily human-readable, and hence, modifiable.

QGISCUSTOMIZATION.conf contains only one set of customization options, called (surprisingly) Customization. This is the set of options that is used to activate or deactivate various parts of the UI. The options themselves are all in this format:

some\element\on\the\interface=state

For example:

StatusBar\mCoordsEdit=true
StatusBar\mCoordsLabel=false

... which will let your UI display your cursor's current geographic coordinates in the status bar, but not precede the coordinate display area with the (useless) "Coordinates:" label.

But here's something (arguably) even more interesting: the contents of QGIS.conf, the main configuration file. This goes beyond just the UI, and influences literally every customizable aspect of QGIS! Even including your database connections:

[PostgreSQL]
connections\foo\service=
connections\foo\host=localhost
connections\foo\port=5432
connections\foo\database=foo
connections\foo\username=foo_user
connections\foo\password=foo_password
connections\foo\publicOnly=false
connections\foo\geometryColumnsOnly=false
connections\foo\allowGeometrylessTables=true
connections\foo\sslmode=1
connections\foo\saveUsername=true
connections\foo\savePassword=true
connections\foo\estimatedMetadata=false

And your plugins:

[PythonPlugins]
plugin_installer=true
fTools=true
mapfile_tools=false
mapserver_export=false

And the list goes on. You can explore these files yourself, of course: what they contain depends greatly on what all you've done in your current install of QGIS.

But you don't even need to edit them by hand to gain a benefit from them. If you know where they are, and there's a standard setup on all the computers you'd like to customize, it's easy enough to throw a bash script out there that will configure all the machines en masse. For instance, say you've got a directory that looks like this:

batch_customize.sh .config/standard_qgis_install/QGIS.conf
.config/standard_qgis_install/QGISCUSTOMIZATION.conf

Then all that would need to be in batch_customize.sh is this:

#!/bin/bash
cp -r .config /home/*

... to be run on all the machines that need to be customized. Obviously, if you're using not-UNIX, this will have to look a bit different, but the same idea applies. You can set QGIS up in a standard way for a given purpose, and distribute this setup around easily. Following which, all the machines you have applied this process on will have a standardized QGIS setup. You might even want to put this folder on a network location somewhere, and configure all the computers to copy those files over on startup, which will minimize your maintenance.

Either way, if you want to apply a setup to a whole set of computers, there's no need to go from machine to machine to do it by hand. Just use the config files strategically, and you'll cut down on the effort required. And, since these files contain everything that can possibly be configured, you can be assured that the setup is identical on all machines. Problem solved!

There's a new development in QGIS Master: a customization tool which lets you (de)activate almost any element in the QGIS user interface. This can get very useful if you have a lot of plug-ins installed that you never use and that are filling your screen. Or maybe you just have a vendetta against clutter. Either way, there are many situations where it could prove to be useful.

Here's a default QGIS setup:

Enough room for everything, although you may start to feel lost among your own tools.

There are two rows of tool buttons running along the top, and this is not a problem on a large screen. But let's say your nice new 21" screen spontaneously combusts and you need to dig out the old CRT from the attic. Or, driven by urgency, you need to work on a 13" laptop. In either case, you would end up with something like this:

Not so great. Now you need to go tool-hunting.

Not very useful. If the tools you need are in one of those click-down lists, you'll need two clicks instead of one to access them. This can get annoying quite rapidly. There's only so much you can do by rearranging those menus, too, before you end up with three or four rows of them. Whereas you're trying to actually see the map you're working on.

Fortunately, thanks to developers Radim Blazek and Martin Dobias, there is now a way to prevent your screen filling up with menus. It's currently available in the development version of QGIS, so hopefully it will be a standard feature by the next release. You'll find it in the Settings menu as "Customization...". Here's a sample of its interface:

Deactivating a few things in the interface.

You can save your various different setups as well:

You can save various different settings for different uses.

Now, with all the tools except the ones you want to use right now deactivated, you've got a much nicer interface to work with:

Well, that's a lot neater!

Let me emphasize here that yes, you can deactivate any element of the user interface. You can even deactivate the File menu if you wanted to. In the screenshot above, notice the status bar. The labels reading "Coordinate:" and "Scale:" have been removed, giving more room for the status field at left, which would have been unable to display the full path otherwise.

The one downside to all this is that your settings don't apply immediately. You basically need to restart QGIS before your changes are applied, which discourages experimentation, especially on slower machines. There has been some talk about enabling different profiles (accessible from Settings > Options), but I haven't been able to find it in my version, which had all the latest updates from yesterday. It would be ideal, of course, if changes were applied immediately and different UI configurations could be applied at startup.

For now, though, it's already a very useful contribution, allowing you to customize your interface for a specific task, or just streamlining it overall. I have to confess that my personal favorite is the ability to remove the "Coordinate:" and "Scale:" labels, which serve no useful purpose if you're at all familiar with how a GIS application works. It's also great to be able to get rid of all the vector and database tools if you're just going to process rasters for the next few sessions, for example. Ultimately, with some better integration into the overall program, this could be a great addition to the overall QGIS user experience.

PS. Yes, the dataset used in the examples is from before the 9th of June, 2011.