Here is a nice story sent to me by Andrew Inglis where they are training people to use QGIS to map AIDS:

Mapping and Geographic Information System (GIS) have long been used in Rwanda for sectors such as agriculture and economic growth. The need for these innovative tools and skills, however, are just now being recognized in other fields, including health. As a monitoring and evaluation expert, I have seen how useful geography and maps can be to monitor and improve programs, and I was interested to learn more about how they were being used and enhanced in the field.

I always love seeing stories where QGIS is being used as a tool to help society!

In my previous blog post l mentioned that l would share on how to download OSM data, load it into Postgres, connect database to QGIS and view OSM data in QGIS. I will further explain how l created the styles in my swatch board using QGIS new symbology....but lets first download the data

OSM test dataset

The OSM project provides free spatial data which you can download and use for your own projects without any restrictions. Visit http://wiki.openstreetmap.org to learn more about OSM. My test dataset was Bayern in Germany therefore l downloaded OSM data for this region from CloudMade downloads site. You can find data for various places in different countries in the world from this site or alternatively use OSM data from Geofabrik. To download data do:

wget -c http://downloads.cloudmade.com/europe/germany/bayern.osm.bz2

Loading data into Postgres

I used osm2pgsql to import OSM data into Postgres. This article has general information on osm2pgsql: http://wiki.openstreetmap.org/wiki/Osm2pgsql. You can also use OSMOSIS to load OSM into Postgres. To install osm2pgsql do:

sudo apt-get install osm2pgsql

then,

time  osm2pgsql -s -v -d osm -C 3000 bayern.osm.bz2

to load the data into Postgres, where osm is the name of the database and bayern.osm.bz2 is the data to be iMported into Postgres. If import is successful, connect the database to QGIS.

Connecting PostGIS to QGIS

There are various GIS packages which one can use to connect to PostGIS but l personally prefer QGIS, mostly because of its clean user-friendly interface. To connect to PostGIS from QGIS, choose Add PostGIS Layer from Layer in the Menu bar. Alternatively click on the Add PostGIS Layer icon which appears in the toolbar. A dialog where you must enter your database credentials will pop-up.

After entering your database information, Click Test Connect. If connection is successful, Click OK to remove the dialog and Connect to view PostGIS tables. The tables should appear as shown below, with names starting with planet_osm. The type column in the dialog below shows the geometry of the feature.

Note that there are two linestring types. In my case, one of the linestrings (planet_osm_roads) represented roads only and the other (planet_osm_line) had both roads and other line features such as rivers. I noticed that some of the roads appeared in the lines table but not in the roads so you must check both tables to avoid omissions. Click on the tables in the list and ADD, to view the features in QGIS.

After connecting the OSM database to QGIS, I opened the attribute tables to analyze the tables. Unlike the normal QGIS shapefiles, OSM stores everything in one huge database. Features are made up of  nodes(basic unit, lat-long coordinates), ways(series of nodes) and relations. Since l had never used OSM data before l read up on the format of OSM data and how attributes were defined from the OSM wiki page http://wiki.openstreetmap.org/wiki/. For more detailed information of how to use OSM data in QGIS you can read this article http://www.qgis.org/wiki/Using_OpenStreetMap_data

OSM External Data - Coastlines

One of the problems l faced was on finding good quality large scale global coastline datasets. I decided to search external data for coastlines after realizing that the coastlines l had downloaded from OSM were not accurate (in most cases my roads would go over the coast). To make sure that my map wouldn't be spoiled by incorrect coastlines l thought having a set of coastlines at various scales (1: 10mil, 1: 2mi, 1: 1mil, 1: 250 000 and 1: 70 000) would be best. But this seemed to be an impossible mission since l only got 1:10mil and 1:1mil (VMap). The naturalearthdata site with coastlines at 10mil, 50 mil and 110 mil can be helpful but there are no 1:2mil, 250th and 70th coastlines. I found this site NOAA coastline extractor but the 1:70th medium resolution coastlines only partially covers the US and the rest of the world is omitted. I asked online but still didn't get the coastlines l wanted and ended up using administrative boundaries http://gadm.org/world at small scales.

Applying QGIS Styles to OSM data

To apply swatch board styles to OSM roads using QGIS right click the layer name (in my case planet_osm_roads) and click Properties. Click the New Symbology button from the Symbols tab and click OK when asked if you wish to use the new symbology implementation for the layer. The Properties dialog box will slightly change. The advantage of using new symbology is that you can create your own styles, save them within QGIS and quickly re-access them.

Click the dropdown next to Single Symbol and select Categorised. The Categorised symbol renderer renders all features from a layer using a single user-defined symbol, which color reflects the value of a selected feature’s attribute. Click the drop-down next to column and choose highway as the attribute. The highway tag in OSM is the primary tag used for any kind of road, street or way. Click classify and you will get a list of all the road classes, though they will have the same symbol.

Using Categorized Symbology in QGIS

To apply individual styles to road classes, double click the symbol to open the a Style Selector and click Change. A Symbol properties dialog which you can use to create your own symbol will appear.

Symbol properties dialog-box

To create a symbol with a fill and outline color, select Simple Line as your Symbol Type. Use Color to set color , Pen width to set the width of the road and Pen style to set the style of the line (solid, dash or dot). Click the Add symbol layer button at the left of the Symbol Properties dialog to add the outline color for the road. A new Simple Line will be added to the Symbol layers list. Use Color to set the color and increase the Pen width to a value slightly higher than the fill. Click the blue arrow to move the outline color below fill color. You will be able to see the symbol you created from the symbol preview.

Road Symbol Preview

If you're happy then click OK and select the Save as Style button to save the style you created. Create as many styles as you like in the same manner and save them. The Style manager stores all your styles and allows you to add, edit, remove, export and import styles. After creating and saving styles you should have a list of all your styles.

Saved QGIS styles

My Layer Properties dialog after categorizing and styling the roads is shown below.

Roads Styles

Part of my roads rendered using QGIS new symbology with categorized classes.

Roads with styles created using QGIS new symbology (click to enlarge)

During the Lisbon hackfest (between all those meetings!) I worked on updating the raster providers to be able to generate stats themselves. The raster provider base class has a default (and historically inefficient as we all know) implementation for collecting stats by walking over every cell in the raster twice. This method remains, but the providers can now supply stats themselves (heopfully using more efficient mechanisms). I have implemented gdal native stats already. Over the last few weeks I did some further testing and cleanups to this work. From my testing some of my worst case files opened in ~1minute rather than ~8+ minutes. It would be great if interested parties could test (details below) before I put it into master.

git remote add timlinux git://github.com/timlinux/Quantum-GIS.git
git fetch timlinux
git branch --track timlinux raster-stats
git checkout raster-stats

I look forward to your feedback.

Hi My name is Petty, I worked as an Intern at Linfiniti after completing my honours degree in GIS last year.

Petronella Tizora-Linfiniti member

A few months ago l did a project with Linfiniti, so I'll be sharing on the steps l took and a few cartography hints and tips which might come in handy if you're interested in making beautiful useful maps. The idea was to create an original, visually attractive and consistent cartographic scheme for representing OSM datasets using QGIS. The first step l took was to build a swatch board.

Swatch Board

The purpose of the swatch board was to help me plan and organize color schemes and feature scales. The advantage of planning your work before you start a project is that you have a basic picture of your output before you start working and you also save time by referring to your initial plan and keeping on track with predefined symbology. My swatch board was a table in Open Office with descriptions of features, colour values and classification fields.

I defined symbology bearing in mind that the map background may be raster (e.g. NASA blue marble or global DEM) or vector (like areas from OSM). I also looked at Google, Bing, OSM and Yahoo maps and websites such as ColourLovers www.colourlovers.com (cool site) to get inspiration on colours to use in my map. All attractive maps seem to have one thing in common....contrasting colours. A normal user should be able to distinguish between freeways and main roads without having a second look at the map. The choices of colors you select for you maps influence the message you send to any viewer. Deciding on which colours to use on a map can be difficult especially if you want to create a unique map. However there are various sites which can help you create themes for your map. Below are some of the sites which l found very useful.

Color Schemer

Color schemer allows you to pick a color or enter its RGB or Hex value and generates matching colors based on your first choice. You can also lighten or darken the scheme. Color Scheme Designer works in an almost similar manner and further allows you to adjust, export and even view colors as a color blind person would see them.

Color Brewer

Color brewer lets you view various color schemes based on the number of classes and scheme you choose for your map. It also provides some advice through "learn more" articles.

ColorLovers

Color lovers is a site where people create and share palettes, patterns and colors. You can also search for palettes and patterns and use them in your own projects.

I generated colors using Color Schemer and used some of them to make a palette in the ColourLovers site. Click on the image to see RGB values for roads in my map.

Roads colour palette generated using coulorlovers

Below is a screen-shot of part of my swatch board

Swatch board

I also created another table which defined scale breaks and line thicknesses for each feature.

Scale breaks and line thicknesses

After all colours, scales and road thicknesses were defined, it was time to get it working on the actual data. Read Part 2 of this document to learn how to download OSM data and load it into Postgres database and apply styles into QGIS .

Getting up and running with QtCreator and QGIS
Firstly let me preface this article by saying that I am not a huge fan of IDEs - I've been using VIM for more than a decade and it's a tough habit to break. That said, QtCreator has a lot going for it (including a special concession to VIM users) and it is a lot easier to use for debugging that crusty old gdb on the command line. So in the article below I am going to run through an optimal (from my point of view anyway) setup for using QtCreator to build, develop and debug QGIS with.
QtCreator is a newish IDE from the makers of the Qt library. With QtCreator you can build any C++ project, but it's really optimised for people working on Qt(4) based applications (including mobile apps). Everything I describe below assumes you are running Ubuntu 11.04 'Natty'.

Installing QtCreator

This part is easy:

sudo apt-get install qtcreator qtcreator-doc

After installing you should find it in your gnome menu.

Setting up your project

I'm assuming you have already got a local Quantum-GIS clone containing the source code, and have installed all needed build dependencies etc. There are detailed in instructions on doing that here:
http://github.com/qgis/Quantum-GIS/blob/master/CODING

On my system I have checked out the code into $HOME/dev/cpp/Quantum-GIS and the rest of the article is written assuming that, you should update these paths as appropriate for your local system.
On launching QtCreator do:

File->Open File or Project

Then use the resulting file selection dialog to browse to and open this file:

$HOME/dev/cpp/Quantum-GIS/CMakeLists.txt

Open the top level CMakeLists.txt

Next you will be prompted for a build location. I create a specific build dir for QtCreator to work in under:

$HOME/dev/cpp/Quantum-GIS/build-master-qtcreator

Its probably a good idea to create separate build directories for different branches if you can afford the disk space.

Build location

Next you will be asked if you have any CMake build options to pass to CMake. We will tell CMake that we want a debug build by adding this option:

-DCMAKE_BUILD_TYPE=Debug

CMake Parameters

Thats the basics of it. When you complete the Wizard, QtCreator will start scanning the source tree for autocompletion support and do some other housekeeping stuff in the background. We want to tweak a few things before we start to build though.

Setting up your build environment

Click on the 'Projects' icon on the left of the QtCreator window.

Project properties

Select the build settings tab (normally active by default).

Build settings

We now want to add a custom process step. Why? Because QGIS can currently only run from an install directory, not its build directory, so we need to ensure that it is installed whenever we build it. Under 'Build Steps', click on the 'Add Build  Step' combo button and choose 'Custom Process Step'.

Add custom step

Now we set the following details:

Enable custom process step [yes]
Command: make
Working directory: $HOME/dev/cpp/Quantum-GIS/build-master-qtcreator
Command arguments: install

Creating a custom process step

You are almost ready to build. Just one note: QtCreator will need write permissions on the install prefix. By default (which I am using here) QGIS is going to get installed to /usr/local. For my purposes on my development machine, I just gave myself write permissions to the /usr/local directory.

To start the build, click that big hammer icon on the bottom left of the window.

The build icon

Setting your run environment

As mentioned above, we cannot run QGIS from directly in the build directly, so we need to create a custom run target to tell QtCreator to run QGIS from the install dir (in my case /usr/local/). To do that, return to the projects configuration screen.

Project properties

Now select the 'Run Settings' tab

The run settings tab

We need to update the default run settings from using the 'qgis' run configuration (shown below) to using a custom one.

Default executable which we want to override

Do do that, click the 'Add v' combo button next to the Run configuration combo and choose 'Custom Executable' from the top of the list.

Add a custom executable

Now in the properties area set the following details:

Executable: /usr/local/bin/qgis
Arguments :
Working directory:
$HOME
Run in terminal: [no]
Debugger: C++ [yes]
          Qml [no]

Then click the 'Rename' button and give your custom executable a meaning full name e.g. 'Installed QGIS'

Custom executable properties

Running and debugging

Now you are ready to run and debug QGIS. To set a break point, simply open a source file and click in the left column.

Now launch QGIS under the debugger by clicking the icon with a bug on it in the bottom left of the window.

Run QGIS in debug mode

Conclusion

There are a few other handy tips and tricks you can do in QtCreator which I will try to cover in a future article. With the information presented above you should at least be on your way to having QGIS setup in QtCreator. Having an interactive debugger at your fingertips is a great way to learn to code in QGIS as it is easy to follow the logic flow from one class / to the next. It also makes it easy for casual developers to dive in and find a bug that is irritating them and either send us a comprehensive bug report or send use a fix .

One of the things we recently added to qgis.org is a new, django based, QGIS community map. So far we have 60 users registered and growing. If I get a chance, I will import the old users from the community map I used to run on QGIS.org some years ago. Today I made a few tweaks, including a 'meet a random user' option. Its fun to see the faces of our users appear on the map - some of which are now becoming familiar faces. In the future, I will write a little joomla module to put random users on the qgis.org home page too.

Our new random user feature - Hello Andreas! Click for larger view.

By the way, all of the code for the new web apps we have written is freely available here. I look forward to pull requests with customisations etc. - and to see others making use of these apps for their own projects.

Today we (the QGIS project) sent out the release announcement for QGIS 1.7. This release took much longer than we would have liked – it included many new features and we implemented many changes to our project infrastructure – all of which consumed time and slowed the release process. There was quite a bit of debate around the release, roughly divided into 3 camps:

- lets get the release out now, its overdue and we need to move on
- lets hold up until there are no more critical bugs
- lets put the release out but mark it as beta, release candidate or similar

Its always difficult to keep everyone happy, and some comments especially for example ones like this from Markus Neteler (of GRASS fame) are especially resonant:

If QGIS crashes several times in a row, I will tend to abandon it as a user.

 Recently I had to downgrade (like other colleagues I know) to QGIS 1.5
 since 1.6 was unusable for me. And if QGIS 1.7 is even worse (I stopped
 trying trunk which I used to do elsewhere), I will not even try to install it.
 Sorry if that sounds harsh but I start to no agree much with the apparent
 QGIS' policy of "we-release-with-most-features-possible-but-with-low-stability"
 which impacts more and more the recent QGIS releases.
 Please note that I (happily) follow QGIS since V 0.x.

 A plea: Please don't overload QGIS with new features at the cost of even
 destabilizing the existing good material.

Markus raises some good points and generally I would agree that we do chase features over stability. Personally I don’t experience too many stability issues with QGIS but it could be that I am in the habit of subconsciously working around issues may cause it to throw a wobbly. It’s a difficult trade off though – QGIS in many respects is still an immature GIS with many features that people expect from a GIS missing. This is especially true if you consider the ‘core’ QGIS application without the GRASS plugin (which introduces many tools from the excellent GRASS toolset). In order to meet the mainstream GIS user’s needs, we need to close this feature gap – and we have been making rapid strides in this direction over the last few years. But closing this feature GAP means that we don’t tend to have bug fix only releases and a strong focus on code stabilisation.

Much of these advances in functionality have been due to features directly funded by users (typically corporate or local government).  Usually they will contract a developer independently, the developer will build their requirements and if the features are of general interest, incorporate them into the general QGIS source code base. This is a great model and has really allowed the QGIS featureset to ramp up quickly – and to build up a small cottage industry of code development work around QGIS.

The downside to this model is that these same clients typically don’t exhibit much interest in funding some of the less sexy aspects of the project – things like writing unit tests, maintaining API documentation, maintaining overall GUI consistency and user experience often fall by the way side. We have soft policies for these things – new features to QGIS core should be accompanied by unit tests, all code should be documented properly and adhere to our coding standards, we have a basic Human Interface Guide (HIG) and so on. However, to really follow through with these QA aspects of the project is probably a full time job  for multiple developers and at the moment its achieved as a patchy effort as and when volunteer developers have time to do so.

In the short term we plan to try address the concerns of Markus and others by backporting fixes from the master branch into the 1.7 branch and putting out maintenance releases (i.e. 1.7.1, 1.7.2 and so on) which will hopefully provide a more stable platform into the future. Even this can be difficult though given the points I made above about feature ramp up. This is especially true of the upcoming development cycle where we are going to be paving the way for 2.0 – which will include removing redundant elements of the application. Redundant you may ask? Currently we have two labelling systems, two rendering systems, a growing number of deprecated API calls and so on. For 2.0 we will spring cleaning a lot of this away. This is good, but it also means that the delta between the master branch and the release branch will become rather large and backporting fixes may not always prove to be very easy.

In doing a bit of navel gazing around the 1.7 release, I updated the download stats for the 1.6 release. Here is a quick overview  (click the image for full size view):

Cumulative download chart for QGIS 1.6

Download decay chart for QGIS 1.6

As you can see from the charts, QGIS is becoming more and more popular, we are just shy of 190,000 downloads for the windows standalone installer for QGIS 1.6. (based on our awstats logs). These figures may not totally accurately reflect our user base though. On the one hand, a download does not necessarily represent a user since the internet is full of bots and spiders crawling web sites and so on. On the other hand, a single download may be used by multiple, tens or even hundreds of users. Now the interesting thing is that given that user base, the donations and sponsorship the project receives is probably equivalent to the purchase price of only one or two ESRI Arc* software licenses. When you think about the sheer number of people we are enabling to have access to GIS software at not cost, it is quite a stunning revelation.

At the same time, there are 190,000 users out there whose expectations need to be met by a very small and relatively unfunded group of active developers and community members. The QGIS developer mailing list has had quite a few active discussions about funding models, sponsored bug fixing and the like over the last few months. In the past we have had problems with ‘directed‘ funding for features or bug fixes not materialising or placing unreasonable expectations on a project that is essentially volunteer driven. It has long been by dream that we can create an undirected fund for QGIS that is there to fully sponsor several of us developers and community members to work on those ‘unsexy’ things I listed above. I think directed funding to target bugs and specific features can be a valuable source of improvement to the project, but I think we will only see real improvement gains to the project when we have sufficient undirected funding to afford us the luxury of not having day jobs that interfere with our QGIS hobby, and really focus on the things that we often don’t get time to do in our capacity as hobbyists.

Hopefully as our user base grows and the software reaches a critical mass in Government and the private sector, my dream of working on QGIS full time, and others who share this dream, will become a reality. In the mean time we will continue to try to make the best software that we can and improve the quality of the work we do so that everyone who tries QGIS enjoys the experience and never feels the need to shell out for a commercial software vendors license.

This turned into a bit of an essay….but it is probably useful to put these things out there so that folks get a better feel for how the project ticks, and where they can meaningfully contribute help. Many of the very active developers and community members of the QGIS project don’t ‘blog’ and otherwise keep a low profile, but I want to just say: “you guys and girls are awesome, I am really going to enjoy using your work in QGIS 1.7!”.

If you’re using QGIS and PostGIS, this article is for you. If not, it could still show you a useful approach, so don’t close the tab yet!

Consider this scenario: you have a large vector dataset which you have combined from several mapsheets (for example, the standard 1:50 000 topo sheets issued by the Surveyor-General). You have loaded it into a PostGIS database, some sort of mapserver is implemented, and all is well. You only need to set this up to render in a timely fashion when queried by a user.

Obviously, you will need to generalize the data to cut down on processing / data retrieval time. Imagine having land use data at 1:50 000 for the whole KZN province – if you host this on a server and someone decides to look at the whole lot at once, it will never render (and nothing else will happen on the server until that one user gets bored and quits).

But not to worry, I only need to generalize the data, right? For example:

SELECT
ST_SimplifyPreserveTopology(the_geom,1000) AS the_geom, landuse
INTO landuse_generalized_1000 FROM "1_50000_landuse_Union";

Chances are, however, that this won’t work as well as it should (if at all) straight off the bat. In my case, I encountered three problems:

1. The data had lots of tiny internal rings which would still get generalized, but would thereby become invisible polygons with two vertices and zero area (or very thin triangles, but either way). They obviously still need to be loaded, with all their attribute data, making the generalization less useful than it could have been.
2. The join boundaries from the earlier union (which turned umpteen map sheets into a single dataset) were still around and would need to be dissolved away before generalization, otherwise there’d be a bunch of slivers / gaps in between polygons from different map tiles.
3. ST_SimplifyPreserveTopology() only accepts Polygon geometries, but once you aggregate the polygons by some attribute to get rid of the boundaries between map tiles, you will have a Multipolygon geometry type.

Hunting around (mainly on http://postgis.reflections.net/) for functions that could be useful, and with many suggestions from Tim (who owns this blog), I eventually came up with a nice procedure.

First apply a small amount of generalization (case-dependent – leave it out if you need the data at full detail), then remove the interior rings. This turns your polygons into Linestrings, but you can just re-polygon-ize them in the same step:

SELECT
ST_MakePolygon(ST_ExteriorRing(ST_SimplifyPreserveTopology(the_geom,10)))
AS the_geom, "LAND_USE" AS usetype INTO landuse1
FROM "1_50000_landuse_Union";

This bit is fairly elementary: to get rid of the tile joins, run a Union, then dissolve by land cover type:

SELECT ST_Union(the_geom) AS the_geom, usetype
INTO landuse2 FROM landuse1
GROUP BY usetype;

(Note that this gets rid of all your other unmentioned attributes, so be sure to list them if you want to keep them!)

Last but not least, to get the data back into a type that can be accepted as input by ST_SimplifyPreserveTopology, you need to run a multipart to singlepart. Although PostGIS does not have a one-step, built-in Multipart to Singlepart function, here’s how to fake one:

SELECT (ST_Dump(the_geom)).geom AS the_geom, usetype
INTO landuse3 FROM landuse2;

It’s important to note that you have to specify (ST_Dump(the_geom)).geom AS the_geom, because the output data type of ST_Dump() is not a geometry. Instead, it’s a special data type called “geometry_dump” which is specfic to the “Dump” family of functions. It contains extra data which in this case we don’t need, but the content of “geom” (which is part of the geometry_dump data structure) is equivalent to plain old geometry, so just use that.

Now at last you’re free to generalize the data:

SELECT ST_SimplifyPreserveTopology(the_geom,250) AS the_geom, usetype
INTO landuse_generalized_250 FROM landuse3;

SELECT ST_SimplifyPreserveTopology(the_geom,500) AS the_geom, usetype
INTO landuse_generalized_500 FROM landuse_generalized_250;

SELECT ST_SimplifyPreserveTopology(the_geom,1000) AS the_geom, usetype
INTO landuse_generalized_1000 FROM landuse_generalized_500;

Now you’ve got four datasets: the slightly simplified one you’ve been working from, and the three you generalized. Come to think of it, “landuse3″ is a bad name, so just rename it to, say, “landuse_full”. Also, if your new datasets came out well, you can probably get rid of your intermediate tables at this point (landuse1 and landuse2, in this case).

Compared to the original (green), the generalized version (orange outlines) is highly simplified ...

... but at the appropriate zoom level, the differences are minor.

You can now delete those tiny, invisible polygons, too. Keep in mind at what scale you’d like to display which layer and view that layer in QGIS at that scale. For safety’s sake, create a test dataset:

SELECT * INTO test1000 FROM landuse_generalized_1000;

Calculate the area in a new column created for the purpose:

ALTER TABLE test1000 ADD area float;
UPDATE test1000 SET area = ST_Area(the_geom);

And get rid of polygons by area:

DELETE FROM test1000 WHERE area < 10;

The maximum area of the polygons that you can remove without making a visual or practical difference to the final map is something you’ll need to determine by trial and error. In the case of my most generalized dataset, I could get rid of about 70 000 entries by throwing out every polygon with an area smaller than 50 000, but obviously this will be highly case-dependent. Still, needing to render about 120 000 rather than 190 000 entries for a visually identical result did speed things up a bit.

Once you’re sure, run those same Alter, Update and Delete statements on the real dataset and remove the test. Even if something horrible happens, you can rest assured knowing that everything was done with SQL statements and getting it all back is as simple as rerunning some commands.

In its final form together with other map layers. Although the data is simplified, it's not noticeable.

As for the practical application of these various levels of generalized data, QGIS has a handy bit of functionality: scale dependent rendering. In your layer properties (once you’ve loaded all needed datasets into your map), go to the “General” tab and check the “Use scale dependent rendering” box, then set whatever scales seem appropriate. Have your layers switch out at the applicable scales for your map, and the user will never even realize that he’s looking at different datasets with different levels of detail. So in the end, the user gets to look at land use for the whole KZN, and your server gets to not crash. Everybody’s happy!

Let’s say you’ve got a lot of raster files you want to host on your map server, or even just one large backdrop. I needed to do the latter, but either way, you’re faced with the same problem: how can I reduce file size and access time? Whatever else I do – such as building pyramids – my raster file is going to be compressed some way or another. I can’t very well have visitors waiting for my server to run through an elaborate (de)compression method, but neither can I store images of the whole of South Africa and the world at several GB per file without perhaps running out of space.

Depending on the amount of space I have on the server, I might be prepared to sacrifice some speed to save space, or vice versa. So what’s the best compression method for my requirements?

I used GDAL’s translate utility (more info here) and went through several of its options to get a better idea of what’s on offer. For compression, my bash command basically looked like this:

  gdal_translate <compression options> -of GTiff input.tif output.tif

To test response times, I ran these commands to export the compressed file to thumbnails of various sizes:

  time gdal_translate -outsize 80 60 -of GTiff output.tif thumb-output.tif
  time gdal_translate -outsize 800 600 -of GTiff output.tif thumb-output.tif
  time gdal_translate -outsize 1280 1024 -of GTiff output.tif thumb-output.tif

(In retrospect, it may have been better to remove the thumbnail between each step rather than overwriting it, to better replicate conditions. Still, I don’t think it made much of a difference.)

My results are summarized in the graphs below, using the following labeling scheme:

Name: Compression options
N/A : -co COMPRESS=NONE
  A : -co COMPRESS=LZW -co PREDICTOR=1
  B : -co COMPRESS=LZW -co PREDICTOR=2
  C : -co COMPRESS=PACKBITS
  D : -co COMPRESS=DEFLATE -co PREDICTOR=1 -co ZLEVEL=1
  E : -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=1
  F : -co COMPRESS=DEFLATE -co PREDICTOR=1 -co ZLEVEL=6
  G : -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=6
  H : -co COMPRESS=DEFLATE -co PREDICTOR=1 -co ZLEVEL=9
  I : -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9

Fig. 1: File size relative to the uncompressed image. Shorter bars are better.

Fig. 2: Access times in seconds. Shorter bars are better.

Going by these results, you can choose whichever compression best suits your requirements. For the shortest access times, you could use “packbits”; however, this also gave the largest (compressed) file size. For the smallest files size, “deflate” with a predictor of 2 and zlevel of 9. For a compromise, you could consider “deflate” with a predictor of 1 and zlevel of either 6 or 9.

Obviously, there are more permutations of this; you could include pyramids as well (use gdaladdo), for example. Or you may be willing to consider JPEG compression if you don’t mind the quality. In any case, I hope this article gives you ideas on where to look!

I  previously blogged about how to use table spaces and showed a little script to automate updating your spatial indexes to use a given namespace. A common problem with PostGIS is that people forget to build spatial indexes on their tables (I'm guilty of it myself as much as anyone). Assuming you use 'the_geom' as the geometry column for your tables as standard, here is a little automation for creating spatial indexes on all your tables in all your schemas (run after logging in to your database at the psql prompt):

\t
\o /tmp/createindexes.sql
select 'CREATE INDEX idx_' || tablename || '_geom ON ' || fullpath || ' 
USING GIST ( the_geom ) TABLESPACE optspace ;'
FROM (select tablename, schemaname || '.' || tablename as fullpath
 FROM pg_catalog.pg_tables where schemaname not in
('pg_catalog','information_schema')) as tablelist;
\o
\i /tmp/createindexes.sql

Some notes:

• the \t psql option disables table headers and footers (it shows tuples only). Its a toggle, so enter it again to re-enable them.
• the \o option writes all query results to a text file on the file system (under /tmp). It is disabled again by doing \o with no parameters.
• The select query uses the system pg_catalogue.pg_tables table to pull out all user made tables and formulate a little sql expression to create the index. Example output of this is shown below.
• the \o disables outpu recording as mentioned above.
• the \i executes the commands in a file from the filesystem – in our case the file full of  ‘create index’ commands we created using the select statement above.

Example output:

CREATE INDEX idx_a_munic_geom ON za.a_munic
USING GIST ( the_geom ) TABLESPACE optspace ;
 CREATE INDEX idx_c_munic_geom ON za.c_munic
USING GIST ( the_geom ) TABLESPACE optspace ;
 CREATE INDEX idx_d_councl_geom ON za.d_councl
USING GIST ( the_geom ) TABLESPACE optspace ;

Note: You can see a list of your tablespaces like this:

select * from pg_catalog.pg_tablespace;
spcname   | spcowner |   spclocation    | spcacl | spcoptions
------------+----------+------------------+--------+------------
pg_default |       10 |                  |        |
pg_global  |       10 |                  |        |
homespace  |    16384 | /home/pg9-data   |        |
optspace   |    16384 | /opt/pg9-indexes |        |
(4 rows)

Here is a nice little article in PC World magazine that mentions the use of QGIS in computer centers in volunteer fire stations. If we have wormed our way into a PC World magazine, I guess we (the Quantum GIS project) have finally ‘made it’ as a popular desktop GIS eh? Thanks William Kinghorn for the pointer!

A client wanted their own custom QGIS build which included some patches and a selection of additional python plugins that were to be available from point of installation. Additionally we wanted to brand the installer with their logo and the splash screen too so that users realise they are using a custom build. The actual build process under windows using msvc is fairly
straightforward (see the INSTALL document in the root of QGIS source tree) so I won’t discuss that here. Rather I am going to show you an approach for building such a custom installer.

Creating a branch

Since we live in git land now, the first thing I am going to do is create a branch for my client’s customisations. If any changes in that branch are useful to the master branch,  I will cherry pick them across later using git cherry-pick.

git checkout -b sansa-branch

Now apply all your code level customisations to the sources in your branch and go ahead and compile everything to make sure it builds, links and installs.

Customising the installer sidebars

The installer sidebars can be altered in the gimp – simply edit :

ms-windows/Installer-Files/sidelogomaster.xcf.bz2

You can open it directly (no need to untar it first). Then use the gimp project to create new versions of:

ms-windows/Installer-Files/WelcomeFinishPage.bmp
 ms-windows/Installer-Files/UnWelcomeFinishPage.bmp

Make sure to keep the image dimensions the same. Those can be commited to your local branch:

git commit -m "Customised sidebar images" \
ms-windows/Installer-Files/WelcomeFinishPage.bmp \
ms-windows/Installer-Files/UnWelcomeFinishPage.bmp

Our custom installer running with new sidebar images

Customising the Splash Screen

The splash screen master can be found here:

images/splash/splash.xcf.bz2

Once again, you can open this directly in the gimp and edit it according to your needs. When you are ready to save the final splash, resize it to 600×287 pixels first. Be careful to also preserve the
transparency in the corners properly or otherwise you will see artifacts when you start QGIS which won’t look nice. The saved splash should be saved as:

images/splash/splash.png

Making a splash: Our custom QGIS build's splash screen

Creating a package tree

There are two ways you can go here…

Way 1:

You can take an existing build of QGIS, extract it under windows, copy your build outputs from msvc over the apps/qgis or apps/qgis-dev  directory and then copy the whole lot over to linux. When it is in your linux box, place it under:

ms-windows/osgeo4w/unpacked

In this case the contents under unpacked should be what was originally under your c:\Program Files\Quantum GIS directory.

One important thing to note is that there are various batch files that are run after QGIS installs. After they are run they are renamed to have a .done extension after them. We need to remove this extension so that they get run after your custom QGIS installer has run. Here are a few lines of bash that do just that:

cd ms-windows
cd osgeo4w
cd unpacked
mv postinstall.bat.done
postinstall.bat
cd etc
for FILE in *.done; do mv $FILE \
$(basename $FILE .done); done
cd ../../../..

Way 2:

You can run the creatensis.pl script like this and specify a shortname e.g. qgis or qgis-dev.

cd osgeo4w
./creatensis.pl --shortname=qgis

It will take a while to run if you have a slow connection like I do, After it is done, place the build outputs from msvc into the file tree at:

unpacked/apps/qgis

So in my case, msvc installed the QGIS I built into c:\program files\qgis1.7.0 – which now becomes the unpacked/apps/qgis directory.

Understanding the batch files

Another important thing to understand is how the various batch files are called within your package tree. Immediately after installation, the postinstall.bat program is run. Its main job is to set a bunch of  environment variables and then call a some subsidiary scripts living in the etc/postinstall directory. One common issue is that you create your installer, install it on a windows machine, and then nothing happens when you click the icon. Usually this will be caused by a mismatch between you qgis launcher script name and the shortcut that was created to call it. The naming convention used in osgeo4w is to install the current stable version of qgis under apps/qgis and the latest development build under apps/qgis-dev. Typically you will only want to ship one or the other to your users – that is an installer based on the current stable release, or one based on the current QGIS trunk. Either way what is important is that everything gets named consistently. For this article, I will use a shortname of ‘qgis’ throughout. So you should check your postinstall.bat to make sure that its calling an appropriate qgis postinstall script. For example mine looked like this:

echo Running postinstall
qgis-dev.bat...

%COMSPEC% /c
etc\postinstall\qgis-dev.bat>>postinstall.log 2>&1
ren etc\postinstall\qgis-dev.bat
qgis-dev.bat.done>>postinstall.log 2>&1

But I am standardising on using apps/qgis for my qgis directory, so I need to change it to look like this:

echo Running postinstall
qgis-dev.bat...

%COMSPEC% /c
etc\postinstall\qgis.bat>>postinstall.log 2>&1
ren etc\postinstall\qgis.bat
qgis.bat.done>>postinstall.log 2>&1

Similarly you need to check in etc/postinstall to make sure that the qgis.bat scrip referred to here exists:

[unpacked] ls etc/postinstall/

grass.bat  msvcrt.bat msys.bat  openssl.bat  pyqt4.bat  qgis-dev.bat
qt4-libs.bat  sip.bat

You can see once again the name doesn’t gel, so I will rename mine to ’qgis.bat’:

mv etc/postinstall/qgis-dev.bat
etc/postinstall/qgis.bat

Lastly, are the actual contents of that batch file consistent?

textreplace -std -t
bin\qgis-dev.bat
mkdir "%OSGEO4W_STARTMENU%"

xxmklink "%OSGEO4W_STARTMENU%\Quantum GIS (1.7.0).lnk"  ...
"%OSGEO4W_ROOT%\bin\qgis-dev.bat"
" " \ "Quantum GIS - Desktop GIS (1.7.0)"  ...
 1 "%OSGEO4W_ROOT%\apps\qgis-dev\icons\QGIS.ico"

xxmklink "%ALLUSERSPROFILE%\Desktop\Quantum GIS (1.7.0).lnk" ...
"%OSGEO4W_ROOT%\bin\qgis-dev.bat"
" " \ "Quantum GIS - Desktop GIS (1.7.0)" ...
1 "%OSGEO4W_ROOT%\apps\qgis-dev\icons\QGIS.ico"

set O4W_ROOT=%OSGEO4W_ROOT%
set OSGEO4W_ROOT=%OSGEO4W_ROOT:\=\\%

textreplace -std -t
"%O4W_ROOT%\apps\qgis-dev\bin\qgis.reg"

"%WINDIR%\regedit" /s
"%O4W_ROOT%\apps\qgis-dev\bin\qgis.reg"

They are not so I will update them to correct that:

textreplace -std -t bin\qgis.bat
 
mkdir "%OSGEO4W_STARTMENU%"

xxmklink

"%OSGEO4W_STARTMENU%\Quantum GIS (1.7.0).lnk"   ...
"%OSGEO4W_ROOT%\bin\qgis.bat" " " ...
"Quantum GIS - Desktop GIS (1.7.0)" 1  ...
"%OSGEO4W_ROOT%\apps\qgis\icons\QGIS.ico"

xxmklink "%ALLUSERSPROFILE%\Desktop\Quantum GIS (1.7.0).lnk" \
"%OSGEO4W_ROOT%\bin\qgis.bat" " " ...
 "Quantum GIS - Desktop GIS (1.7.0)" ...
1 "%OSGEO4W_ROOT%\apps\qgis\icons\QGIS.ico" 

 
set O4W_ROOT=%OSGEO4W_ROOT%
set OSGEO4W_ROOT=%OSGEO4W_ROOT:\=\\%

textreplace -std -t
"%O4W_ROOT%\apps\qgis\bin\qgis.reg"
"%WINDIR%\regedit" /s
"%O4W_ROOT%\apps\qgis\bin\qgis.reg"

Whether or not you actually need to tweak these scripts is largely dependent on your starting point. I can only say I had lots of issues because I had mismatches between references to qgis and qgis-dev to it is useful to understand the sequence of events. The last script we edited will ultimately create the shortcut icon on the users desktop and menu system.

You also need to check that the preremove scripts are similarly consistent, so check these scripts are there and internally correct:

preremove.bat
etc/preremove/qgis-dev.bat

Lastly you should remove bin/qgis.bat if it is present and check that bin/qgis.bat.templ is there. If it is named as qgis-dev.bat.templ rather than qgis then rename accordingly to qgis.bat.templ. Now check inside this script to make sure everything is consistent there too. The following diagram describes these processes visually.

How batch files are used in QGIS windows edition

Customising what gets shipped.

Here is the opportunity to add some customisations to the installer contents. For example I wanted to include ecw and sid support in my  installer, as well as a bunch of the nicer python plugins that are out there. Simply copy the files you need into the correct place in the unpacked directory tree. For example, I took a range of python plugins from my ~/.qgis/python/plugins/ directory and placed them in:

unpacked/apps/qgis/python/plugins/

e.g.

cd unpacked/apps/qgis/python/plugins/
cp -r ~/.qgis/python/plugins/bcccoltbl1/ .
cp -r ~/.qgis/python/plugins/cadtools/ .
cp -r ~/.qgis/python/plugins/metatools/ .
cp -r ~/.qgis/python/plugins/numericalDigitize/ .
cp -r ~/.qgis/python/plugins/numericalVertexEdit .
cp -r ~/.qgis/python/plugins/openlayers .
cp -r ~/.qgis/python/plugins/openlayersov .
cp -r ~/.qgis/python/plugins/postgis_manager .
cp -r ~/.qgis/python/plugins/QGISFileBrowser .
cp -r ~/.qgis/python/plugins/qgsAffine .
cp -r ~/.qgis/python/plugins/rawrasterfileimport .
cp -r ~/.qgis/python/plugins/shadedrelief .
cp -r ~/.qgis/python/plugins/tablemanager .
cp -r ~/.qgis/python/plugins/topocolour .
cp -r ~/.qgis/python/plugins/valuetool .

Building an installer

Now we get to the part you have been waiting for – building an installer. There is a convenience script here that will do most of the work for you:

<qgis source tree root>/ms-windows/quickpackage.sh

Before you run the script, you should check that the shortname inside this file matches the one we used in the steps above (in this case ’qgis’ rather than ‘qgis-dev’).

makensis \
-DVERSION_NUMBER='1.7.0' \
-DVERSION_NAME='Wroclaw' \
-DSVN_REVISION='0' \
-DQGIS_BASE='Quantum GIS' \
-DINSTALLER_NAME='QGIS-1-7-0-Sansa-Edition-Setup.exe' \
-DDISPLAYED_NAME='Quantum GIS 1.7.0' \
-DBINARY_REVISION=1 \
-DINSTALLER_TYPE=OSGeo4W \
-DPACKAGE_FOLDER=osgeo4w/unpacked \
-DSHORTNAME=qgis \
QGIS-Installer.nsi

You can also use the opportunity to customise the name of the installer and the name of the version and release number of QGIS if you like  (though I don’t recommend that latter two much as they provide a good point of reference to users). When you are finished, simply run the quickpackage script and wait a while:

 ./quickpackage.sh

After a few minutes the script will come to an end and you should have a nice shiny new QGIS-1-7-0-Sansa-Edition-Setup.exe (or whatever you called it) created in the ms-windows directory. Now its simple a matter of installing it in a clean windows machine to test it, running it through a virus checker and then putting it online for people to enjoy.

Our custom QGIS showing the modified version name in the title bar

Ethics

With all the flexibility that QGIS offers, being open source and ‘out there’ for anyone to hack on, its easy to take what the project provides and rebrand it substantially:

• you can create a custom icon theme
• you can remove unwanted user interface elements
• you can replace help text, about text etc
• you can rename the application itself to something else
• you can change the application icon

and so on and so on. One thing you may not do is claim the fundamental work to be your own. Any time you make a publically available custom version of QGIS, be sure to remember that you need to redistribute the full source code (including all your customisations) with it in order to comply with the letter and spirit of the GPL.

One final thing to mention is that at the Lisbon hackfest, Martin Dobias and Radim Blazek demonstrated a really slick and easy to use framework for creating stripped down versions of the QGIS user interface. It introduces the idea of profiles that you can nominate at application start up. So a school teacher may for example provide her students with just the few icons and tools they need to complete a particular exercise. Look out for that in a future version of QGIS I guess…..

If anyone is interested in having custom branded versions of QGIS for use within their organisation, we (linfiniti) will be happy to help - just pop us an email.

In November 2010, I posted some charts showing some  QGIS download stats. Today I made some updates to these charts (click on them for full image) – enjoy!

Cumulative download counts

Download decay chart

In November 2010, I posted some charts showing some  QGIS download stats. Today I made some updates to these charts (click on them for full image) - enjoy!

The GeoNames dataset provides a list of global placenames, their location and some additional information such as population and so on. It is published under the Creative Commons Attribution 3.0 License, which allows you to use the data for your own purposes. You can download the data by country, or the entire global dataset. In this article, I will walk you though how I downloaded the entire dataset, loaded it into PostgreSQL and added a geometry column so that I could view it in QGIS. Note that you can substitute these instructions for a specific country’s data easily.

First up, lets get the data from the geonames download page!

wget -c http://download.geonames.org/export/dump/allCountries.zip

Note the download is around 196mb so if you live in an internet backwater like I do, expect it to take a little while. If the download gets disconnected, just rerun the same command again – the ‘-c’ option tells wget to continue where it left off last time.

Once the data is downloaded, unzip it:

unzip allCountries.zip

You should now have a text file called allCountries.txt weighing in at just under 900mb. Next we can load it into PostgreSQL using a variation of this article. I highly recommend the use of schemas to partition your database into logical units. In the code listings that follow, it is assumed you have a schema called ‘world’. If you need to create it, simply do:

create schema world;

From the psql prompt. Since I am only interested in the geoname table at the moment I simply do this in my database.

create table world.geoname (
         geonameid       int,
         name            varchar(200),
         asciiname        varchar(200),
         alternatenames  varchar(8000),
         latitude        float,
         longitude       float,
         fclass  char(1),
         fcode   varchar(10),
         country varchar(2),
         cc2 varchar(60),
         admin1  varchar(20),
         admin2  varchar(80),
         admin3  varchar(20),
         admin4  varchar(20),
         population      bigint,
         elevation       int,
         gtopo30         int,
         timezone varchar(40),
         moddate         date
 );

You will notice that I extended the alternatenames field size from the original tutorial’s 4000 characters to 8000 characters in order to accommodate some longer entries that were causing my imports to fail with 4000 chars.

Next we can import the data (also from the psql prompt):

copy world.geoname (geonameid,name,asciiname,alternatenames,
latitude,longitude,fclass,fcode,country,cc2,
admin1,admin2,admin3,admin4,population,elevation,gtopo30,
timezone,moddate) from '/home/web/allCountries.txt' null as '';

Once again this is similar to the import line used by the original article I used, except I have used a full path to my allCountries.txt file. The import may take a little while depending on the speed of your computer.

When it is completed, you should have a bunch of data (~7.5 million records) in your table:

gis=# select count(*) from world.geoname ;
  count
---------
 7664712
(1 row)

It is going to be useful to have a unique id for every record – QGIS for one would like to have it, so lets add one (in addition to the geonameid field):

alter table world.geoname add column id serial not null;
CREATE UNIQUE INDEX idx_geoname_id ON world.geoname (id);

Because I know I will be using some other fields as basis for subset queries etc., I added some indexes on those too:

CREATE INDEX idx_geoname_population ON world.geoname (population);
CREATE INDEX idx_geoname_fcode ON world.geoname(fcode);

Ok thats all great, but there is no geometry column yet so we can’t view this in QGIS easily. So lets GIS enable the table! First we add a new geometry column:

alter table world.geoname add column the_geom geometry;

Now we populate the geometry column:

update world.geoname set the_geom = st_makepoint(longitude,latitude);

Next we add a constraint to ensure that the column always contains a point record or is null

alter table world.geoname add constraint enforce_geotype_the_geom
CHECK (geometrytype(the_geom) = 'POINT'::text OR the_geom IS NULL);

Finally lets index the table on the geometry field:

CREATE INDEX idx_geoname_the_geom ON world.geoname USING gist(the_geom);

Ok next we can connect to the database using QGIS and view our data! Note that you may want to filter the data or set up scale dependent visibility so that QGIS doesn’t try to render all 7.5 million points when you zoom out.

I added a query filter like this to the layer properties -> general tab -> subset:

"population" >= 10000 AND fcode != 'PPLA' and fcode != 'PCLI' AND fcode != 'ADM1'

You should experiment and read the geonames documentation in order to define a filter that is useful for your purposes.

Geonames layer loaded in QGIS

The Geonames dataset is a wonderful asset to the GIS community, particularly to those of us who value Free Software and Free Data.

Update 6 Aprl 2011:

I encountered one issue with the above procedure: the SRID for the imported points is -1 when loaded instead of 4326. This cause problems e.g. in mapserver you get an error like this:

ERROR:  Operation on two geometries with different SRIDs

To fix the issue I ran the following update query to assign all points a proper SRID:

update world.geoname set the_geom = ST_SETSRID(the_geom,4326);

Note that it takes quite a while to run.  When it completes, you can verify that the points are all in the correct CRS by running this little query:

gis=# select distinct ST_SRID(the_geom) from world.geoname;

Which should produce something like this:

 st_srid
---------
 4326
(1 row)

For good measure, I also added the geoname table to my geometry columns table:

insert into geometry_columns (f_table_catalog,f_table_schema,f_table_name,f_geometry_column,
coord_dimension,srid,type) values ('','world','geoname','the_geom',2,4326,'POINT');

Lastly I also gave select permissions to my readonly user (which I use when publishing in a web environment):

grant usage on schema world to readonly;
grant select on world to readonly;

I have also fixed the formatting of some of the SQL snippets above for those who struggled to read it within the width of this page.

One of the nice new features coming in QGIS 1.7 is the ability to use data driven labelling interactively. I made a little video to illustrate below. This work was carried out by Marco Hugentobler of Sourcepole.

Oh, and just for Andreas, here is the same video as a webm file

One of the nice new features coming in QGIS 1.7 is the ability to do a routing analysis across a road network. This new plugin was written by Sergey Yakushev. The little screencast that follows gives a quick demonstration of its usage.

I haven’t had much time for updating my blog lately but I thought I would just post a quick note to mention the awesome work that is taking place in QGIS. We are a few scant days from entering feature freeze in the run up to the 1.7 release, and the project has been a hive of activity in the last few weeks. There are lots of new features coming in 1.7, but also a lot of spit and polish. We have a roadmap that will take us to a 2.0 release later this year and it will be another huge milestone in our project. I’m not going to mention a bunch of new features today – we’ll save that for the release announcement. But I did want to mention that on the fly raster reprojection support was added to QGIS ‘trunk’ today by Radim Blazek.

I’ve been testing his work for the last week or two and it is fast and seamless. It’s a bit hard to show it in a static image, but here are two screenies that try to do just that.

In the first image here we see my raster layer is in Lambert Conic Conformal projection.

Raster layer: Native CRS - Lambert Conic Conformal

Next, here is a picture of my project properties. You can see on the fly reprojection is enabled, and that it is set to Google Mercator.

Project : CRS Google Mercator

The roads layer you see overlaid is Open Street Map data also in Google Mercator. Before the addition of Radims raster improvements, you would have needed to set the project CRS to Lambert Conic Conformal (to match the raster) if you wanted to overlay the two. Now QGIS is reprojecting the raster from Lambert Conic Conformal to Google Mercator.

QGIS 1.7 should be out within the next month, and I hope you enjoy using it as much as we have been!

I haven’t used ccache for some time, but recently I bought  a little netbook. On a netbook environment, any optimisation you can make to the compilation process will have real benifit since QGIS takes a looooong time to compile without it. Back at my desktop computer I decided to set up ccache too and do a simple benchmark of how much faster compilation is when ccache is enabled.

But perhaps I am getting ahead of myself. What is ccache? Its a piece of software that caches compiled C++ object code. When a compilation task occurs for a piece of unchanged code that is in the cache, ccache will simply return the precompiled object file instead of compiling the sources from scratch.

Installing ccache is simple:

sudo apt-get install ccache

After it is installed you need to enable it by doing something like:

sudo su -
ln -s /usr/bin/ccache /usr/local/bin/gcc
ln -s /usr/bin/ccache /usr/local/bin/g++
ln -s /usr/bin/ccache /usr/local/bin/cc
exit

In your QGIS build directory, you should now edit CMakeCache.txt and change the CMAKE_CXX_COMPILER:FILEPATH entry to /usr/local/bin/g++ :

CMAKE_CXX_COMPILER:FILEPATH=/usr/local/bin/g++

Then run

cmake ..

Note: After running cmake .., check your build prefix and build type, grass path etc. are still as expected – for me cmake reverted them to default values.

Now you can compile. To do a simple benchmark, open a separate terminal and type:

watch -n 20 ccache -s

The latter allows you to watch how objects are being added to your cache, and how often the cache is being used (you want it to be used a lot!). The output looks something like this:

cache directory                     /home/timlinux/.ccache
cache hit                            911
cache miss                           916
called for link                       94
compile failed                         3
not a C/C++ file                       7
files in cache                      1832
cache size                         101.5 Mbytes
max cache size                     976.6 Mbytes

Now build your project the first time from a clean state, and watch how long it takes to build:

make clean
time make install -j4

When it was built it reported a build time looking like this:

real	6m15.478s
user	18m46.230s
sys	2m33.810s

After it is build repeat the above and you should see the new build time is markedly shorter:

make clean
time make install -j4

Here is the build time with ccache doing its thing:

real	1m15.627s
user	2m28.500s
sys	0m35.380s

Conclusion: Spending a few minutes setting up ccache can save you a lot of aimless waiting around while QGIS compiles – well worth the effort!

I have previously posted here about kCube Consulting’s donation of 6 months developer time to the QGIS project. Today I took the opportunity to interview Kumaran Narayanaswamy and SunilRaj Kiran from kCube to find out more about their plans and motivations.

Kumaran Narayanaswamy

SunilRaj Kiran

TS: Welcome Kumaran and SunilRaj to this interview. Could we start by asking each of you to give us a brief introduction to yourselves?

KN: Let me give a brief introduction about myself and kCube. kCube is a Startup company founded in year 2006 by myself and my college buddy (Senthil Kumar) and we specialize in providing Geospatial Services. I did my Engineering in Geoinformatics and graduated in 1999 and have been working in the industry mainly Logistics domain. I was working with Cognizant after graduation and have been executing projects for Logistics clients based out of US. I moved back to India in 2006 and started kCube along with my college buddy Senthil Kumar.

We did a market research before starting and we realized that Open Source want not being leveraged to a larger extent in India as it was being in western countires and we felt that Open Source Services is a huge market  in India kCube is not a exclusive Open Source Company but we push Open Source as our first priority.
We try to sell the benefits of Open Source to the customers and if they buy in our ideas we go ahead with the open source based solutions if the client already has a technology preference and if he wants to stick with that we go ahead and implement based on his technology preference. We started in 2006 as a 2 member team and now we have a team of 15 members. We provide Application Development,Data Management and Capacity Building Services.

SK:  I did my undergraduation B.E in the department of Geoinformatics right now im doing my masters in Geomatics. I have been doing various C++ applications and projects right from my college days. I joined Kcube as an employee last year and i have been dealing with Qgis since then. Initially started with building the source from trunk on various platforms and was analyzing all Qgis tools. resently I’m looking into the Qgis API for understanding. Also have developed simple plugins on Qgis to make sure I have understood the API properly.

TS: Kumaran, are you able to say what the split is in your business between FOSS GIS and Proprietary GIS solutions? In otherwords, how much demand is there for FOSS GIS solutions?

KN: All our Application Development Projects have been completely Open Source projects only for Data Projects some of the clients prefer to do their work using their infrastructre which is proproetary. We do lot of Classification work for which we use GRASS. Our target customers are NGO’s, Smaller Enterprises and Governement Agencies. We have been able to sell FOSS GIS easily in this segment. We also do a lot of Capacity Building Programs on various FOSS GIS products. In India we are the first to start on this segment.

TS:  Sunil, did you learn C++ and programming as part of your undergrad course or are you self taught?

SK: Although it is  a part of my curriculum, I always had a passion towards C and C++ programming.

TS: Recently kcube caused some excitement in the QGIS community when you announced that you would be donating 6 months of developer time to the QGIS project. What prompted kcube to do this?

KN: We have generated revenue through QGIS and we felt it is our time to give back to the community and make QGIS product even better. We do Capacity building programs on QGIS and we are promoting QGIS in India. We started working with QGIS from 2006 -  I think it was 0.98 version – and from then we have been closely following QGIS more as an end user.

I have been doing training [with QGIS] and the feedback that we have recieved for QGIS is overwhelming, especially [since]these are the users who are currently with proprietary GIS packages. Open Source is always a community based development and we felt that also should be part of the development community and hence this initiative.

TS: SunilRaj, you have been appointed as the developer to work on QGIS for 6 months. Can you tell us something about yourself from a QGIS perspective? Are you new to the project, have you done any coding on QGIS before? What kinds of development work do you usually do?

SK: As I have mentioned earlier , I have just started developing small QGIS plugins for understanding purposes. During my college days I have used Arc Objects to customize ArcMap. Other than this I was doing only non Gis C++ development. Now I am getting familiar with QGIS API and it is very interesting.

TS: Kumaran, what did you think of the process we followed to decide what the SunilRaj should work on? Are you happy that his time will be spent on bug fixing?

KN: As I said earlier Open Source is always a Community driven approach and we are perfectly okay with what Community has decided. At the end of the day they are going to use the product and if we deliver what they want it will be a huge success for the product. Also during the training [sessions we hold] the bugs crop in and sometimes I also feel that we need to trash these bugs first and that was the same answer which we got from the community.

TS: SunilRaj – are you happy to work on bug fixing for six months? Often people arriving on the project are more interested in working on cool new features rather then maintaining existing code. Can you see personal benifits to working on bug fixing?

SK: From my view, bug fixing is a good point to start with. Since I am a learner of QGIS API, it is good that I start with bug fixing so that I will come to know as many API functionality as possible. Once I successfully fix some of the bugs, I will get the confidence to look into new features development.

TS: As a general question: How well is FOSS GIS being adopted in India? Are there any good success stories of it being taken up in government and industry?

KN: There is a huge adoption for Open Source in general in India but for FOSS GIS there is not much of awareness. One of our first plan is to create awareness on various Open source GIS products. We are running road shows along with NRCFOSS and OSGeo India chapter NRCFOSS is the [government agency] set up by Indian government to promote FOSS at Government and Academia. I would say the movement at GIs has just got started and will take a while to get the momentum but people who are exposed to FOSS GIS get fascinated from day one and give us the feedback that these packages are [more] user friendly than proprietary.

Coming to Case studies, we have implemented couple of webGIS and desktop GIS  for NGO’s and government agencies. We have used mainly GeoServer, MapServer, OpenLayers, GeoExt, PostgreSQL/PostGIS stack for Web GIS  and for Desktop we go with QGIS Customization.

TS: Is there anything else you need to make a success of this 6 month bug fixing period – anything you would like to ask for help on from the QGIS community?

KN: As Sunil was mentioning he is overall comfortable with QGIS and its development process. To start with we have advised him to pick up bugs which are not more complicated so that he can understand the API first and gradually as he progresses he can pick up more complex bugs and resolve the same. Currently he is doing his development under ubuntu environment. If you have any other development environment suggestions you can let us know. Also currently we were trying to set up a integrated Debugging environment to debug issues for QGIS Code base. We would like to know from you on how the core team does the debugging? Do you use any IDE based debugging or is it through message displays?

TS:  Qt4 Creator is probably a good option for you – personally I am using command line tools – vim, gdb etc. which allow also interactive debugging, variable inspection, backtracing etcwith qt-creator you can do the same but in a point and click way. Ubuntu linux makes a very good development platform – in fact almost all QGIS developers are working on linux as far as I know as the development tools are great.

KN:  We would certainly need the help of developer community to make this a success as and when we hit road blocks we to get the required help
TS:  Many thanks to both of you for agreeing to this interview – on behalf of the QGIS community, we would like to thank you both for the awesome contribution you are making to the project. I would like to do a follow up interview at the end of this six month period so that we can reflect on how things went.

KN:  Its our pleasure Tim to be part of this great QGIS team who have been doing an outstanding job. We will send you monthly reports on how bugs are progressing which you can also share with the community.