Related Plugins and Tags

QGIS Planet

A simple QGIS python plugin tutorial

Introduction to Python Plugins and QGIS

Note:This tutorial requires QGIS 1.6 (not yet released at time of writing) or QGIS Trunk r14052 (available from osgeo4w as a nightly build or build it yourself on linux from source).

Generating your plugin

There is an online tool for [creating a plugin: http://pyqgis.org/builder/plugin_builder.py]. Simply fill in the blanks and it will generate a simple plugin framework for you. We are going to make a simple plugin to perform a histogram stretch based on the min max values in the current view extents. Lets start by filling in some details in the plugin builder:

Class name (use CamelCase)     : RasterScale
Short descriptive title        : Raster Local Histogram Stretch
Description                    : Scale the min max of the raster to
                                 the min max within the view.
Version number                 : 0.1
Minimum required QGIS version  : 1.5
Text for the menu item         : Raster
Author/Company name            : Linfiniti Consulting CC
Email address                  : tim@linfiniti.com

After clicking the 'build it' link you will see a screen like this:

Generation complete for RasterScale. You can download it here.
What Next?
Unzip the plugin into your QGIS plugin directory and test it.
Modify it by editing the implementation file RasterScale.py
Create your own custom icon, replacing default icon.png
Modify your user interface by opening RasterScale.ui in
Qt Designer (don't forget to compile it with pyuic4
after changing it)
Use the Makefile to compile your Ui and resource files
if you make changes to them (requires gmake)

Click the download link and save it to your local disk.

Testing your plugin

Use your operating system to extract the plugin to your home directory .qgis folder:

/home/[yourname]/.qgis/python/plugins/

If this directory does not already exist, you should create it.

Now open QGIS.

Next do:

Plugins -> Manage Plugins

In the filter box enter

Stretch

Now tick the box next to the plugin to enable it then click 'OK'.  An icon will appear in the plugin toolbar and if you click it, your plugin will run!

Install the plugin reloader plugin in QGIS

Normally when you change a plugin you have to close and reopen QGIS to see the result of your changes. This can become a little tedious. To work around this, you can install the plugin reloader plugin like this:

Start QGIS
Plugins -> Fetch Python Plugins
Repositories tab
Click the 'Add 3rd party repositories button' and click Ok for the message that appears.
Wait a few moments while the repository list is updated.
On the Options tab, check the 'Show all plugins, even those marked experimental' radio button
In the Plugins tab, type reload into the filter box
Select the Plugin Reloader plugin from the list
Click Install, then Ok

Now we want to configure the plugin reloader to reload our raster scale plugin so do this:

Press Shift+F5
Choose rasterscale from the plugin list
Press Ok

Now whenever you press the F5 key, your plugin will be reloaded along with any changes you might have made to it.

First tweaks to our plugin

Lets make our first little tweaks to our plugin - just to test out the development process. Look at the name of your plugin:

click Plugins -> Raster

In the raster submenu you will see our plugin is named ``Raster``. Lets rename it to ``Raster Scale``. To do this, in Eric, open the RasterScale.py file from the list of files in your project on the left. Now look at the initiGui method and change the line that creates the menu action - it looks like this:

self.action = QAction(QIcon(":/plugins/rasterscale/icon.png"), \
"Raster", self.iface.mainWindow())

Now change it so that it looks like this:

self.action = QAction(QIcon(":/plugins/rasterscale/icon.png"), \
"Raster Scale", self.iface.mainWindow())

You will see above that I have simply added the characters 'Scale' to the QAction's name.

Now save that file and go back to QGIS. Hit the ``F5`` key to reload your plugin. Now once again do

click Plugins -> Raster

You should see the submenu is now named 'Raster Scale' instead of just 'Raster'.

This is the general process you should follow when writing a plugin -

Edit code

Save source file

Reload plugin in QGIS (F5)

Test

In the units that follow we will assume that you do this each time we ask you to modify your plugin sources.

First steps into the QGIS api

Writing python plugins in QGIS requires knowledge of three things:

1 Python
2 PyQt - the Python language bindings for the Qt4 framework
3 The QGIS api

A really good resource for learning PyQt is the command prompt reference

Each of these three things is documented and generally searchable with google. We will look at the documentation in a little while. but first lets get our hands dirty and start modifying our plugin.

The first thing I would like to do is outline the functionality that we are going to give our plugin. Here is the logic flow:

image0

Getting and checking the active layer

As a first step, we are going to replace the run() method in RasterScale.py with our own logic - the comments in the code explain step for step what is going on:

# run method that performs all the real work
def run(self):
  # get the currently active layer (if any)
  layer = self.iface.mapCanvas().currentLayer()
  # test if a valid layer was returned
  if layer:
    # test if the layer is a raster from a local file (not a wms)
    if layer.type() == layer.RasterLayer and ( not layer.usesProvider() ):
      # Test if the raster is single band greyscale
      if layer.rasterType()==QgsRasterLayer.GrayOrUndefined:
        #Everything looks fine so show a little message and exit
        QMessageBox.information(None,"Raster Scale","Layer is ok")
        return
  # One of our tests above failed - show and error message and exit
  QMessageBox.information(None,"Raster Scale", \
      "A single band greyscale raster layer must be selected")
  return

Now press F5 in QGIS to reload the plugin (since we have changed its code) and then click the plugin icon with no layers loaded in your project. You should get a message telling you that there needs to be a layer available.

Next add a greyscale raster (e.g. a dem) and then click on the plugin icon again. This time you should get a 'Layer is Ok' message.

Computing the min max values within the view extent for the layer

Now we want to answer the question: 'what is the minimum and maximum value in my current view extent?' Fortunately its pretty easy to do - using the raster layer computeMinimumMaximumFromLastExtent method e.g.:

# compute the min and max for the current extent
extentMin, extentMax = layer.computeMinimumMaximumFromLastExtent( band )

If for example our current view extent contains pixels with values from 20 to 140, extentMin will now be assigned a value of 20 and extentMax will now be assigned a value of 140. The idea is to then scale the colour assigments made to each pixel to this range, such that a pixel of value 20 will be painted black and a pixel of value 140 will be painted white. The image below tries to explain this better:

image1

Setting the band min and max values

Now what we need to do is tell the raster layer to consider the min max values from the current extent to be the min max values for the whole layer, and then to stretch the colour pallette accross the new min/max value range for the layer.

Ok so to do this we can use the following api calls:

# For greyscale layers there is only ever one band
band = layer.bandNumber( layer.grayBandName() ) # base 1 counting in gdal
# We don't want to create a lookup table
generateLookupTableFlag = False
# set the layer min value for this band
layer.setMinimumValue( band, extentMin, generateLookupTableFlag );
# set the layer max value for this band
layer.setMaximumValue( band, extentMax, generateLookupTableFlag );

This should be self explanatory except maybe the part about a lookup table. Lookup tables are used for creating custom colour pallettes. Since in our case we are not interested in creating a custom colour pallette, we can leave it out of the equation for now by setting it to false.

Extra house keeping

Just a little extra house keeping is needed. First we have to ensure that standard deviations are disabled as it will affect the values given to each pixel. Next we let the raster layer know that we are using a user defined min and max range rather than the true range of the data in the raster. Next we clear any cached image for the raster (used to speed up drawing in some situations). Lastly we tell the layer to redraw itself!

# ensure that stddev is set to zero
layer.setStandardDeviations( 0.0 );
# let the layer know that the min max are user defined
layer.setUserDefinedGrayMinimumMaximum( True );
# ensure any cached render data for this layer is cleared
layer.setCacheImage( None );
# make sure the layer is redrawn
layer.triggerRepaint();

Putting it all together

Lets look at our complete run method now:

# run method that performs all the real work
def run(self):
  # get the currently active layer (if any)
  layer = self.iface.mapCanvas().currentLayer()
  # test if a valid layer was returned
  if layer:
    # test if the layer is a raster from a local file (not a wms)
    if layer.type() == layer.RasterLayer and ( not layer.usesProvider() ):
      # Test if the raster is single band greyscale
      if layer.rasterType()==QgsRasterLayer.GrayOrUndefined:
        #Everything looks fine so set stretch and exit
        #For greyscale layers there is only ever one band
        band = layer.bandNumber( layer.grayBandName() )
        extentMin = 0.0
        extentMax = 0.0
        generateLookupTableFlag = False
        # compute the min and max for the current extent
        extentMin, extentMax = \
                         layer.computeMinimumMaximumFromLastExtent( band )
        # set the layer min value for this band
        layer.setMinimumValue( band, extentMin, generateLookupTableFlag )
        # set the layer max value for this band
        layer.setMaximumValue( band, extentMax, generateLookupTableFlag )
        # ensure that stddev is set to zero
        layer.setStandardDeviations( 0.0 );
        # let the layer know that the min max are user defined
        layer.setUserDefinedGrayMinimumMaximum( True )
        # ensure any cached render data for this layer is cleared
        layer.setCacheImage( None )
        # make sure the layer is redrawn
        layer.triggerRepaint()
        #QMessageBox.information(None, 'Raster Scale', \
          "Min %s : Max %s" % ( extentMin , extentMax ))
        return
  # One of our tests above failed - show and error message and exit
  QMessageBox.information(None,"Raster Scale", \
        "A single band raster layer must be selected")
  return

Testing

Refresh the plugin using the plugin reloader F5 keyboard shortcut. Now zoom to an area on your greyscale raster and then click the RasterScale plugin icon. You should see something happen like that shown below:

image2
Before scaling
image3
After scaling

Exercise

See if you can update the plugin so that it works for RGB and palletted images too!

Exercise Solution

/***************************************************************************
RasterScale
A QGIS plugin
Scale the min max of the raster to the min max within the view.
               -------------------
begin        : 2010-08-05
copyright      : (C) 2010 by Linfiniti Consulting CC.
email        : tim@linfiniti.com
 ***************************************************************************/

/***************************************************************************
 *                                     *
 *  This program is free software; you can redistribute it and/or modify *
 *  it under the terms of the GNU General Public License as published by *
 *  the Free Software Foundation; either version 2 of the License, or   *
 *  (at your option) any later version.                  *
 *                                     *
 ***************************************************************************/
"""
# Import the PyQt and QGIS libraries
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from qgis.core import *
from qgis.gui import *

# Initialize Qt resources from file resources.py
import resources
# Import the code for the dialog
from RasterScaleDialog import RasterScaleDialog

class RasterScale:

 def __init__(self, iface):
  # Save reference to the QGIS interface
  self.iface = iface

 def initGui(self):
  # Create action that will start plugin configuration
  self.action = QAction(QIcon(":/plugins/rasterscale/icon.png"), \
    "Raster Scale", self.iface.mainWindow())
  # connect the action to the run method
  QObject.connect(self.action, SIGNAL("triggered()"), self.run)

  # Add toolbar button and menu item
  self.iface.addToolBarIcon(self.action)
  self.iface.addPluginToMenu("&Raster", self.action)

 def unload(self):
  # Remove the plugin menu item and icon
  self.iface.removePluginMenu("&Raster",self.action)
  self.iface.removeToolBarIcon(self.action)

 # run method that performs all the real work
 def run(self):
  # Allowed drawing styles that can have a local histogram stretch:

  allowedGreyStyles = [ QgsRasterLayer.SingleBandGray,
             QgsRasterLayer.MultiBandSingleBandPseudoColor,
             QgsRasterLayer.MultiBandSingleBandGray,
             QgsRasterLayer.SingleBandPseudoColor ]
  allowedRgbStyles = [ QgsRasterLayer.MultiBandColor ]
  # get the currently active layer (if any)
  layer = self.iface.mapCanvas().currentLayer()
  # test if a valid layer was returned
  if layer:
    # test if the layer is a raster from a local file (not a wms)
    if layer.type() == layer.RasterLayer and ( not layer.usesProvider() ):
      # Test if the raster is single band greyscale
      if layer.drawingStyle() in allowedGreyStyles:
        #Everything looks fine so set stretch and exit
        #For greyscale layers there is only ever one band
        band = layer.bandNumber( layer.grayBandName() ) # base 1 counting in gdal
        extentMin = 0.0
        extentMax = 0.0
        generateLookupTableFlag = False
        # compute the min and max for the current extent
        extentMin, extentMax = \
                          layer.computeMinimumMaximumFromLastExtent( band )
        # set the layer min value for this band
        layer.setMinimumValue( band, extentMin, generateLookupTableFlag )
        # set the layer max value for this band
        layer.setMaximumValue( band, extentMax, generateLookupTableFlag )
        # ensure that stddev is set to zero
        layer.setStandardDeviations( 0.0 )
        # let the layer know that the min max are user defined
        layer.setUserDefinedGrayMinimumMaximum( True )
        # ensure any cached render data for this layer is cleared
        layer.setCacheImage( None )
        # make sure the layer is redrawn
        layer.triggerRepaint()
        return
      if layer.drawingStyle() in allowedRgbStyles:
        #Everything looks fine so set stretch and exit
        redBand = layer.bandNumber( layer.redBandName() )
        greenBand = layer.bandNumber( layer.greenBandName() )
        blueBand = layer.bandNumber( layer.blueBandName() )
        extentRedMin = 0.0
        extentRedMax = 0.0
        extentGreenMin = 0.0
        extentGreenMax = 0.0
        extentBlueMin = 0.0
        extentBlueMax = 0.0
        generateLookupTableFlag = False
        # compute the min and max for the current extent
        extentRedMin, extentRedMax = layer.computeMinimumMaximumFromLastExtent( redBand )
        extentGreenMin, extentGreenMax = layer.computeMinimumMaximumFromLastExtent( greenBand )
        extentBlueMin, extentBlueMax = layer.computeMinimumMaximumFromLastExtent( blueBand )
        # set the layer min max value for the red band
        layer.setMinimumValue( redBand, extentRedMin, generateLookupTableFlag )
        layer.setMaximumValue( redBand, extentRedMax, generateLookupTableFlag )
        # set the layer min max value for the red band
        layer.setMinimumValue( greenBand, extentGreenMin, generateLookupTableFlag )
        layer.setMaximumValue( greenBand, extentGreenMax, generateLookupTableFlag )
        # set the layer min max value for the red band
        layer.setMinimumValue( blueBand, extentBlueMin, generateLookupTableFlag )
        layer.setMaximumValue( blueBand, extentBlueMax, generateLookupTableFlag )
        # ensure that stddev is set to zero
        layer.setStandardDeviations( 0.0 )
        # let the layer know that the min max are user defined
        layer.setUserDefinedRGBMinimumMaximum( True )
        # ensure any cached render data for this layer is cleared
        layer.setCacheImage( None )
        # make sure the layer is redrawn
        layer.triggerRepaint()
        return
  # One of our tests above failed - show and error message and exit
  QMessageBox.information(None,"Raster Scale", \
        "A single band raster layer must be selected")
  return

Complete Plugin

The complete plugin is available here. I will add it to a plugin repository once QGIS 1.6 comes out.

Updated to remove all those semi-colons from the end of lines.....

3 New Case Studies for QGIS

Otto Dassau has posted 3 new case studies of QGIS in use in a production environment. It's always gratifying to see people using QGIS to run their businesses, do their studies and get on with their daily work.

I'm looking forward to seeing more case studies arrive on the QGIS home page - if you are doing something interesting with QGIS, do let Otto know about it!

A week of FOSSGIS training at Linfiniti

Its been another great week here at Linfiniti. Last weekend we (the QGIS project) released QGIS 1.5 Tethys - its great to have another release out the door.

Also making my week a good one was the fact that we ran a week long FOSSGIS course here at Linfiniti HQ in Swellendam, Western Cape, South Africa.

The Presenter

image0

Horst Duester - Linfiniti's 'FOSSGIS Guru in Residence' for the months of July and August ran the course. Horst has very kindly donated his time and considerable expertise over his two month sabbatical to the development of FOSSGIS GIS skills in South Africa. The course consisted of 2 days' QGIS, 3 days PostGIS. With these two projects you can address a huge range of GIS functionality, and having a whole week to dedicate to the topic let Horst really go in depth into the subject.

The Particpants

image1

We had four participants for the course - all of whom have some experience in GIS in general and who wanted to get a deeper understanding of QGIS and PostGIS.

image2

Andiswa Silinga works out of Johannesburg where she runs a small business - GGES (Gemini GIS and Environmental Services) - which provides GIS solutions to government and enterprise customers. GGES also has special skills in the area of Environmental GIS.

image3

Originally from Zimbabwe, Petronella Tizora was a former intern with Linfiniti consulting. Petronella currently works at GGES with Andiswa and is applying her FOSSGIS skills to her daily work.

image4

Sindile Bidla works out of East London in the Eastern Cape, South Africa. Sindile's small GIS consulting business Intengu Technologies provides solutions - he is currently involved in a project to map informal settlements. Sindile formerly worked for the Dept. Land Affairs where he was the principle instigator of the Gentle Introduction to GIS project which saw the introduction of FOSSGIS into the school curriculum in the Eastern Cape.

image5

Brendon runs a small consulting company - Integrated Geodata - from his offices in Pretoria, South Africa. Brendon's company provides general GIS consulting services, with a special interest in providing services to the water industry.

The course was really well received by the attendees, and Horst thoroughly enjoyed the work of presenting it. It is really gratifying to see how - with a little training - FOSSGIS can provide a basis for building sophisticated GIS solutions for the enterprise. Better still it provides these entrepreneurs a fantastic edge - the ability to offer their clients a solution that has no vendor lock in with maximum value for money.

In the coming months I am planning to formalise the course content developed by Horst and try to increase my focus on training - the course certainly reaffirmed the notion for me that providing good quality, affordable training is the best way to maximise the penetration of FOSSGIS into the market place.

A bash one-liner for resizing images

I have always ejoyed ImageMagick - in particular (much like gdal) I get a wierd kind of pleasure from being able to edit images from something so innately non-graphical as the command line. One common activity I do is resizing images. In this case, I wanted every image in a directory to be 200px wide whilst it's height should be proportionately resized to match the width.

I also wanted to remove pesky spaces from the file names, and make sure they were all writable. Here is a little one liner I came up with:

sudo chmod a+w *.jpg; for FILE in *; do mv "$FILE" $(echo $FILE | sed 's/ /_/g'); \
echo $FILE; done; for FILE in *.jpg; do echo $FILE; convert -resize 200x $FILE $FILE; done

I know it could have been done more efficiently in a single loop but since I was knocking this together in a jiff, it wouldn't have been worth the time...

The real take-home command is

convert -resize 200x $FILE $FILE

which resizes the file to 200px wide and then saves it over itself.

A week of FOSSGISS-ing in Johannesburg, SA

Horst Duester and I spent the week in Johannesburg (South Africa) doing FOSSGIS training. As I mentioned in a previous article, Horst is visiting South Africa on a sabbatical. He has kindly donated his time to provide training services in order to promote and facilitate the use of FOSSGIS. We badly need more people evangelising FOSSGIS here - there are many people looking for low cost and good GIS solutions and few people with the time and know-how to show them how to proceed.

We drove up to Jhb from Swellendam in one day - 1400km - a first for Horst who calculated that with an equally long trip from his home town he would have landed up in Russia!

Our first port of call was at Wits University (the University of Witwatersrand, Jhb). Devlyn Hardwick convened the course at Wits and kindly provided us with accommodation for the week. We spent the monday afternoon installing the Ubuntu VMWare image we had prepared onto all of the computers in the lab. At the end of the day it was a real thing of beauty to see so many Ubuntu workstations.

From Tuesday to Thursday we (ok mainly Horst) presented an enlightening overview of QGIS and GRASS to the participants. He covered the process you need to follow to produce a professional quality map - from importing or digitising your date, to symbolising that data to conducting an analysis and finally culminating in the use of the QGIS map composer to create a great map. Much of the development of the map composer was funded by the SO!GIS group at Solothurn and it was really great to revisit just how far QGIS has come along in its ability to produce a high quality map.

Training at Wits University

Training at Wits University

Devlyn and Horst enjoying sumptuous fare at the Ethiopian restaurant

Devlyn and Horst enjoying sumptuous fare at the Ethiopian restaurant

Horst putting QGIS through its paces

Horst putting QGIS through its paces

On friday our activities moved to the CSIR Meraka institute in Pretoria (which is very close to Johannesburg). At Meraka we held a one day PyWPS workshop which was convened by Graeme McFerren. During the workshop we walked the participants through the process that you need to follow to get PyWPS set up on your server, and then followed that with some examples of how to write a PyWPS process. The workshop was quite technical in nature. Those particpants with less technical knowledge were given an understanding of what the OGC WPS standard sets out to facilitate, while those with deeper technical knowledge left with the building blocks they needed to go off and deploy their own PyWPS processes.

PyWPS training on the go

PyWPS training on the go

One of Horst's motivations for spending his sabbatical here in South Africa was to improve his English. So it was quite a feat on his behalf to conduct his training sessions entirely in English for four days straight.

All in all it was a great and successful week and we are now en-route back to the Cape - though we are breaking up the trip with an extremely pleasant stay over at the Karoo National Park. In true geek fashion we are sitting in our chalet using e2fsck to repair Horst's dodgy hard disk which seems to have thrown a wobbly after being exposed to our South African electrical system.

Horst in the Great Karoo

Horst taking a break from riding through the desert on a horse with no name

As an aside for those interested, the way to check your disk under Ubuntu / Linux and mark bad sectors so they don't get used is this:

::
e2fsck -c /dev/sda5

In the weeks to come Horst will be presenting more free and commercial FOSSGIS training workshops here in South Africa. Hopefully in the future others who feel like a break from the daily grind will come and pay a visit to South Africa and follow suite - the effort is greatly appreciated here!

jqGrid: Hands down the nicest grid control for JQuery

I've been surveying various grid controls for a web project. I thought I would save others from doing the same trawling and point you to jqGrid. Its beautiful, flexible and powerful - try some of their demos to see what I mean!

Cleaning geometries inside PostGIS

Its been an enlightening few days. Horst Duester (of SO!GIS fame) is paying me a visit for a two month sabbatical. Horst is here to provide valuable training and knowledge exchange to South Africans interested in FOSS GIS (more on that to come in future blog posts no doubt!). Its been really great having an opportunity to sit down with a fellow FOSSGIS geek and exchange notes - and today Horst showed me something really cool. Horst has written a PostGIS function to clean geometries within the database. Normally I use GRASS if I want to clean the topology of my vector dataset. With Horst's approach this is no longer needed which is a great timesaver. Here's what to do in a nutshell:

Assuming you already have a spatial database created (with lwpostgis, spatial_ref_sys etc. sql scripts run on it), download and install Horst's clean geometry function. Here is on way to do that:

wget http://www.sogis1.so.ch/sogis/dl/postgis/cleanGeometry.sql
psql mydatabase < cleanGeometry.sql

Now lets create a test table and show you how to use it. I used Martin Dobias' excellent PostGIS Manager plugin (available in the python plugin repositories) to create a table as shown in the screenshot below (click for a larger version). As an aside, Martin's tool is a great way for new users to get started with building their own spatially enabled database.

Next I created a self-intersecting polygon (in this case a figure of eight-ish shape) as shown below:

borken-poly

The little green cross in the middle of the polygon shows the self-intersection. To clean the geometry, I then ran a little sql command that uses Horst's cleangeometry to fix all the geometries in the table like this:

update testpolys set the_geom=cleangeometry(the_geom);

The result is a multipart polygon with no self intersections but preserving the original shape and attributes. In the image below I've pulled the connecting vertices in the center apart so you can see that two sub-polygons were created:

mulitpolygons-after-clean

Eye-Spaghetti: Visualising QGIS dependencies

While looking at the options for apt-cache today I noticed 'dotty' as an option. I presumed this could be used to make grahpviz diagrams of package dependencies. After a bit on googling on how to use that, I came across debtree:

sudo apt-get install debtree graphviz

Of course the first command I ran with it was this :-)

debtree -I --rdeps-depth=3 qgis | dot -Tpng -o qgisdeps.png ; display qgisdeps.png

Take a look at the beautiful (ok and just a teeny bit complex) diagram it produced. Its quite amazing really to see how many different components there are out there in the FOSS world that can be combined to produce cool software like QGIS. It really lets you see that QGIS is just as much an integration effort as it is a bespoke software development effort.

Great page of Open Layers examples

The more I use OpenLayers, the more I enjoy using it. I often refer to the OpenLayers examples page to look at the source of examples as inspiration for my work. Today I happened across another great website providing some more brilliant examples of how to get the most out of OpenLayers, so I thought I would share the resource here in case others also find it useful.

Enjoy!

Translating the Gentle Introduction to QGIS videos (take #2)

A little while ago I posted here that the videos we created for the Gentle Introduction to GIS series were online at dotsub.com. I received feedback from some readers that the videos were still locked and so subtitles could not be added. I believe they have been unlocked now and if folks would like to subtitle in their own language (which would be great!) you can go ahead and do this. Currently all the videos should be available with both English and Spanish subtitles and it would be great to see more languages added to the list. Here are the links to all the vids:

Slimming down OpenLayers

The recent release of the 2.9.x OpenLayers javascript library has seen it grow still fatter 857kb for the OpenLayers.js if you use it out of the box from the OpenLayers download.

If you live in a bandwidth impoverished country like South Africa, those 857kb equate to a lengthy wait time while a page with a map on it first loads.

I've never really delved much into the inner workings of OpenLayers before - I'm just a contented user - but with this new release I thought I would take a little poke around to see if it was possible to trim some of the fat off the library.

The fine folks at OpenLayers already minify the javascript (which removes redundant white space, compacts var names etc), so my best option was to look at removing parts of the code base I don't really use. OL supports many data formats that I'm not using e.g. WFS, bing maps, etc.

It turns out its pretty easy to create your own custom build. What I did was unpack the OL 2.9.1 tarball and then do:

cd build

To enter the build directory. Then I created a file called tim.cfg that looks like this:

# This file includes the OpenLayers code to create a build for everything that
# does not require vector support. build.py uses this profile if no other one
# is specified.

[first]
OpenLayers/SingleFile.js
OpenLayers.js
OpenLayers/BaseTypes.js
OpenLayers/BaseTypes/Class.js
OpenLayers/Console.js
OpenLayers/Util.js
OpenLayers/Events.js
OpenLayers/Handler.js
Rico/Corner.js

[last]

[include]
OpenLayers/Format/WKT.js
OpenLayers/Format/KML.js
OpenLayers/Format/WFS.js
OpenLayers/Format.js
OpenLayers/Handler/Point.js
OpenLayers/Geometry/Point.js
OpenLayers/Geometry/Surface.js
OpenLayers/Geometry.js
OpenLayers/Layer/Vector.js
OpenLayers/Control/Attribution.js
OpenLayers/Control/Button.js
OpenLayers/Handler/Box.js
OpenLayers/Handler/Click.js
OpenLayers/Handler/Drag.js
OpenLayers/Handler/Feature.js
OpenLayers/Handler/Hover.js
OpenLayers/Handler/Keyboard.js
OpenLayers/Handler/MouseWheel.js
OpenLayers/Handler/Path.js
OpenLayers/Handler/Point.js
OpenLayers/Handler/Polygon.js
OpenLayers/Handler/RegularPolygon.js
OpenLayers/Control/ZoomBox.js
OpenLayers/Control/ZoomToMaxExtent.js
OpenLayers/Control/DragPan.js
OpenLayers/Control/Measure.js
OpenLayers/Control/MouseDefaults.js
OpenLayers/Control/MousePosition.js
OpenLayers/Control/KeyboardDefaults.js
OpenLayers/Control/PanZoom.js
OpenLayers/Control/ArgParser.js
OpenLayers/Control/Permalink.js
OpenLayers/Control/Scale.js
OpenLayers/Feature/Vector.js
OpenLayers/Renderer
OpenLayers/Renderer/Elements.js
OpenLayers/Renderer/SVG.js
OpenLayers/Renderer/VML.js
OpenLayers/Renderer.js
OpenLayers/Map.js
OpenLayers/Layer/WMS.js
[exclude]

Once that is created, you can build the library then copy it into your project to test:

python build.py tim.cfg
cp OpenLayers.js [your project dir]

Then reload your web page with a map on it. You can incrementally remove features while testing your page still works properly until you reach the happy medium of size vs functionality. In my case my custom OpenLayers.js came out to a much smaller size - 342kb - and given time I could probably strip it down further.

Thanks to the OpenLayers devs for a great library!

Update After posting this I realised that I hadnt removed WFS and KML - after removing those two lines in the above file, the library now weighs in at 290kb!

Python testing with Nose - even easier!

Recently, a friend pointed me to Python nose, a package providing a very easy way to build tests, based on unittest. I still have to write some code on my own to prove this, but I am too impatient to talk about it! Stay tuned for some code snippets in the next future.. Have a look at nose's page: nose

Happy Python coding,

--anne

Automated backup of multiple PostgreSQL/PostGIS databases

I have a cron job that backs up my PG databases and emails them to me every night. Today I wanted to upgrade my PostgreSQL 8.3 databases to PG 8.4 so I made a few modifications to my script so that I could dump out my PG data, and then restore it under PG 8.4. In case you are wondering I am doing this because Ubuntu Lucid now ships with PG 8.4 (yay!). I also made the script generate another script to restore the databases. So basically the procedure is to run the script below on your 8.3 cluster, shut that down and bring up your 8.4 cluster, and then restore your databases into that. Here follows my little script:

MYDATE=`date +%d-%B-%Y`
MYBACKUPDIR=/home/timlinux/sql_backups
MYRECIPIENT=tim@linfiniti.com

DBLIST=`psql -l \
  | awk '{print $1}' | grep -v "+" | grep -v "Name" | \
  grep -v "List" | grep -v "(" | grep -v "template" | \
  grep -v "postgres"`
for DB in ${DBLIST}
do
  echo "Backing up $DB"
  FILENAME=${MYBACKUPDIR}/PG_${DB}.${MYDATE}.sql.tar.gz
  pg_dump -f ${FILENAME} -x -O -F tar ${DB}
  #If you want to email the database uncomment
  #below (will cause issues if backups are large)
  #mpack -s "Daily $DB PG backup for ${MYDATE}" $FILENAME $MYRECIPIENT
done

echo "Procedure to restore one of your backups:"
echo "createdb "
echo "pg_restore -F t .sql.tar.gz |psql "
echo "Or to restore in a batch make a script like this:"
echo "for FILE in /home/timlinux/sql_backups/*; do DB=\$(echo $FILE | \"
echo "  sed 's\/home\/timlinux\/sql_backups\/PG_//g' | sed 's/.${MYDATE}.sql.tar.gz//g'); "
echo " 'Restoring: \$DB'; createdb \$DB; pg_restore -F t \$FILE |psql \$DB; done"

Update 02 May 2010 Uncommented pg_dump line which was inadvertantly commented in my original post.

Your chance to translate the Gentle Introduction to GIS videos

From the 'this is cool dept': dotsub is now hosting the GIS gentle intro videos - you can go there and add subtitles. I still need to add markers to them to indicate where each sentance should start - at the moment I don't have time or bandwidth...maybe someone else will pick that up...

Biodiversity at FOSS4G2010

If you are interested in biodiversity and attending FOSS4G2010 in Spain this year, take a look at my friend Javier de la Torre's initiative to form a FOSSGIS special interest group in Biodiversity at the conference. Unfortunately I can't attend the conference but Javi's work really resonates with me and FOSSGIS has a lot to offer those who are passionate about keeping out planet alive for another year or two!

Final report on my experience @ Linfiniti

Time for a final report has come...

First of all, I thank Tim for all trust he had in my programming and managing capabilities, and for giving me this great job opportunity. I'm deeply thankful to the interns I mentored: Robert, Sam and Petty, the nicest people I could meet, it has been so natural to work together. They made me feel comfortable in our office with flickering internet and power outages, teaching me the African friendliness and patience :)

What I learned:

  • Django- the main project I worked on taught me how to find my way in a huge directory tree and three programming languages. After a tough beginning, I've been able to quickly indentify the point(s) of the code to edit and do less side-effect damages possible. Django is really a simple and powerful tool to build websites and manage backend databases.
  • OpenLayers - JavaScript turned to be a quick and flexible enough way to display and manage maps. Still have lot to learn but at least I can read and understand almost all JS code I encounter.
  • LTSP- our office ran Linux Terminal Server Project, on a common PC as server and three FitPC as thin clients. This solution has many advantages: less power consumption, data and programs in only one machine (tribute to SPOT rule), less cost than having N full-featured PCs , very good scalability - it can be easlily used in classrooms and labs.
  • Mentoring - there is a special ability required to transmit knowledge. While explaining concepts, I realised I was using most of them without understanding them fully, and I learned how to build more clear and structured lessons, and to say "I don't know the answer" instead of inventing something... Thanks to my interns for all precious feedback!

What I planned to achieve:

  • Test-driven development - unfortunately none of us knows enough of it to quickly build tests. Moreover, the code for our webGIS builds an interactive GUI with JavaScript and AJAX. Therefore we have been reduced to test by hand, and quite often the commits broke functionality and I didn't notice until I had time to extensively test - when the Internet allowed me to refresh the page in less than 10 minutes.
  • Extreme Programming - We succeeded in applying some of its principles: communication (even if English is mother tongue only for Tim), extensive use of the whiteboard for mind dumps and lessons, short release cycles, feedback. We would like to improve our office management integrating other valuable tips - Robert is reading "Extreme Programming Explained" and he's really enthusiastic!

This experience has been really precious and I will never forget it.

I enjoyed a lot working in a pure open source environment, with strong belief in its values by my employer. It is too common to work for a company and relegate open source to spare time or to side activities.

Living here for three months made me realise how lucky is Europe, with all first-world facilities, and how open source is a valid alternative to the common closed-source solutions. In GIS, especially, there is little place for ESRI to claim the huge (for South African standards) licence fees, while there is a great need of GIS for land management at any level.

I strongly suggest to open source geeks to come and stay in South Africa and work at Linfiniti! It will be a great experience abroad, in a country full of beauty, humanity and hope.

Look forward to come back!!

--anne

Learning GIT

For source code management in the QGIS project we have been happily using subversion for some time now, having made a not-too-painful transition over from CVS several years ago. Lately we have bandied around the idea of moving over to GIT. This is mostly out of fear of being called bad things by Linus Torvalds for using SVN (ok I'm kidding). Actually there are a lot of technical reasons why a move over to GIT would work well for us - not least of which is the idea of easier merging of branches and the redunancy offered by multiple repos being 'out there'.

Adding a little impetus to my interest in learning GIT is the fact that more people are using it for managing their code so I need to update my skills to be able to collaborate with them effectively.

All this brings me to my real point: today I found a great resource which provides a guide for those familiar with SVN and wanting to migrate to GIT.

2 Great tools for web developers

I just thought I would make a note of two little gems:

favicon.cc is an online tool for creating that perfect favicon for your web site. Yes the irony hasnt escaped me that my own site needs one!

ajaxload.info is a cool little site that helps you to create an 'waiting' gif that you would typically use in an ajax application while the user waits for something to happen.

Debugging django syncdb

Sometimes I make changes to my django models that causes syncdb to not run and I can't figure out why. Here is an example of an error I might get:

python manage.py syncdb
.
. ommitted for brevity
.
    return self.get_query_set().get_or_create(**kwargs)
  File "/usr/lib/pymodules/python2.6/django/db/models/query.py",
    line 343, in get_or_create
    raise e
psycopg2.IntegrityError: duplicate key value violates unique
    constraint "auth_permission_pkey"

These messages are a bit opaque so one useful technique I found is to temporarily enable postgresql logging to a file:

sudo vim /etc/postgresql/8.3/main/postgresql.conf

Now add or uncomment the following:

log_destination = 'stderr'
logging_collector = on
log_directory = '/tmp'
log_statement = all

Next you need to restart PostgreSQL (obviously you dont want to do this on a production system without taking the needed precautions!).

sudo /etc/init.d/postgresql-8.3 restart

Next I ran my syncdb command again and at the same time watched my PostgreSQL logs being generated in /tmp (I used the unix tail command to do that):

2010-03-29 14:29:22 SAST LOG:  statement: INSERT INTO "auth_permission"
("name", "content_type_id", "codename") VALUES (E'Can add Clip', 64, E'add_clip')
2010-03-29 14:29:23 SAST ERROR:  duplicate key value violates unique
constraint "auth_permission_pkey"
2010-03-29 14:29:23 SAST STATEMENT:  INSERT INTO "auth_permission"
("name", "content_type_id", "codename") VALUES (E'Can add Clip', 64, E'add_clip')
2010-03-29 14:29:23 SAST LOG:  statement: ROLLBACK TO SAVEPOINT s1217227072_x1

Ok from that I could see that django was having trouble inserting entries into the auth_permissions table for my Clip model. The postgresql logging functions can prove indispensible when you just want to know what low level requests and responses are coming and going from your database.

Batch importing shapefiles into PostGIS

Batch importing data into PostGIS is a topic I have covered previously in my article on importing CDSM (Chief Directorate: Surveys and Mapping) data into postgis. However its a popular question I get asked so today I am going to take another look at the problem. In particular the client has a directory heirachy of zipped shapefiles divided by regions in Africa. The shapefiles use a common naming convention which makes things easier. The convention is basically something like:

<number><region_name>/<region_abbreviation>_master/<region_abbreviation>_<feature_type>_<feature name>.zip

For example:

02 Mozambique Malawi/moz_master/moz_a_parks.zip

When unzipped the shapefiles are consistently named across regions e.g.:

moz_a_parks.shp
moz_a_parks.shx
moz_a_parks.dbf
moz_a_parks.sbn
moz_a_parks.sbx

From that we can define a few rules:

  • Recurse the top level directory looking for all zip files
  • Get the base name of the zip file without its extension (e.g. moz_a_parks)
  • Strip the region off the base name - this will give us the standard feature class name / table name used across all regions for that feature type.
  • Extract the zip file to /tmp
  • If the feature class / table does not exist in PostGIS, import it with the -c (create) option
  • Otherwise import it with the -a (append) option

So here is the little script I wrote that achieves this.

#!/bin/bash
DATABASE=foo
IFS="$(echo -e '\n\r')";
FILES=$(find -name "*.zip");
for FILE in $FILES
do
  BASE=$(basename $FILE .zip)
  TABLE=$( echo $BASE | sed 's/^[a-z]*_//g')
  # double check the extracted files have been cleaned up or zip will moan
  rm /tmp/${BASE}.*
  echo "$BASE to $TABLE"
  unzip -d /tmp $FILE
  MATCH=$(echo "\d" | psql ${DATABASE} | grep -o "${TABLE}")
  if [ "$MATCH" ]
  then
    # Append to the exisitng table
    echo "Appending to ${TABLE}"
    shp2pgsql -s 4326 -I -S -a -W UTF-8 "/tmp/${BASE}.shp" $TABLE | psql $DATABASE
  else
    # Create the table
    echo "Creating $TABLE"
    shp2pgsql -s 4326 -I -S -c -W UTF-8 "/tmp/${BASE}.shp" $TABLE | psql $DATABASE
  fi
  # clean up
  rm /tmp/${BASE}.*
done

If your shapefiles are already unzipped into a directory structure, you could easily modify the script above to find '.shp' files rather than zip files and leave out hte unzipping , cleaning up steps.

Once you have run the scripts, open your favourite PostGIS client (I use psql) and take a look at the tables created there. If all looks good you should be able to start browsing the datasets with QGIS or publishing them with Mapserver!

Back to Top

Sustaining Members