After almost a year and a lot of hard work QGIS 1.8 is finally out.  This is the best QGIS version so far, packed full of fancy new features.

The official release notice can be found here: http://qgis.org/index.php?option=com_content&view=article&id=149 and downloads can be found at http://download.qgis.org

Here is the change log of all the new stuff in 1.8:

- QGIS Browser - a stand alone app and a new panel in QGIS. The
browser lets you easily navigate your file system and connection based
(PostGIS, WFS etc.) datasets, preview them and drag and drop items
into the canvas.
- DB Manager - the DB manager is now officially part of QGIS core. You
can drag layers from the QGIS Browser into DB Manager and it will
import your layer into your spatial database. Drag and drop tables
between spatial databases and they will get imported. You can use the
DB Manager to execute SQL queries against your spatial database and
then view the spatial output for queries by adding the results to QGIS
as a query layer.
- Action Tool - now there is a tool on the map tools toolbar that will
allow you to click on a vector feature and execute an action.
- MSSQL Spatial Support - you can now connect to your Microsoft SQL
Server spatial databases using QGIS.
- Customization - allows setting up simplified QGIS interface by
hiding various components of main window and widgets in dialogs.
- New symbol layer types - Line Pattern Fill, Point Pattern fill
- Composers - have multiple lines on legend items using a specified character
- Expression based labelling
- Heatmap tool - a new core plugin has been added for generating
raster heatmaps from point data. You may need to activate this plugin
using the plugin manager.
- GPS Tracking - The GPS live tracking user interface was overhauled
and many fixes and improvements were added to it.
- Menu Re-organisation - The menus were re-organised a little – we now
have separate menus for Vector and Raster and many plugins were
updated to place their menus in the new Vector and Raster top level
menus.
- Offset Curves - a new digitising tool for creating offset curves was added.
- Terrain Analysis Plugin - a new core plugin was added for doing
terrain analysis – and it can make really good looking coloured relief
maps.
- Ellipse renderer - symbollayer to render ellipse shapes (and also
rectangles, triangles, crosses by specifying width and height).
Moreover, the symbol layer allows to set all parameters (width,
height, colors, rotation, outline with) from data fields, in mm or map
units
- New scale selector with predefined scales
- Option to add layers to selected or active group
- Pan To Selected tool
- New tools in Vector menu - densify geoemtries, Build spatial index
- Export/add geometry column tool can export info using layer CRS,
project CRS or ellipsoidal measurements
- Model/view based tree for rules in rule-based renderer
- Updated CRS selector dialog
- Improvements in Spatial Bookmarks
- Plugin metadata in metadata.txt
- New plugin repository
- Refactored postgres data provider: support for arbitrary key
(including non-numeric and multi column), support for requesting a
certain geometry type and/or srid in QgsDataSourceURI
added gdal_fillnodata to GDALTools plugin
- Support for PostGIS TopoGeometry datatype
- Python bindings for vector field symbollayer and general updates to
the python bindings.
- New message log window
- Benchmark program
- Row cache for attribute table
- Legend independent drawing order
- UUID generation widget for attribute table
- Added support of editable views in SpatiaLite databases
- Expression based widget in field calculator
- Creation of event layers in analysis lib using linear referencing
- Group selected layers option added to the TOC context menu
- load/save layer style (new symbology) from/to SLD document
- WFS support in QGIS Server
- Option to skip WKT geometry when copying from attribute table
- upport for zipped and gzipped layers
- Test suite now passes all tests on major platforms and nightly tests
- Copy and paste styles between layers
- Set tile size for WMS layers
- Support for nesting projects within other projects

Thanks to all the sponsors and everyone who put a lot of work into this release!

Something that I always find interesting is how people are using different open source tools to get their work done.  This post attempts to outline how I/we are using QGIS at work for different projects.

Kerb mapping, condition, and defect pickup

This project is currently being done by a 67 year old foreman who has worked for the council for a very long time and has great knowledge of the town.   QGIS, with the main working layers stored in PostGIS, was setup so that he can:

• Digitize kerb lines from aerial photos.
• Split the existing kerb lines into segments depending on different asset rules.
• Give each segment an overall condition rating.
• Add defect points along the each kerb segment e.g. broken, lifted, etc,

Each defect point is snapped to the underlying  kerb line and chainages (distance along line) is generated using a update statement at the end of the project (could be done using a insert trigger if needed) using ST_line_locate_point(line, point).

Defect points coloured by risk captured against the kerb line

Overall QGIS has been great for this project.  The built in data entry forms have been a great help to allow fast and correct data entry. Each form has four drop downs all with present values and descriptions to aid in data entry.

Flood damaged claim maps

We recently suffered, like the rest of Queensland, some really major flooding which caused large amounts of damage to our road infrastructure. We got off pretty light compared to some places, nevertheless we still had a lot of damaged assets.  And so began the process of collecting data that could be used for state government funding claims.

Anyway, onto the QGIS bit.  QGIS was installed on one of the main engineers computers in order for him to make maps for each claim.  Having the ability for him to have one map window but multiple frames in the composer helped him to create multiple  views of the same data with ease.

In total there are 42 QGIS project files with a main project file which served the base layers to the other projects, using the cool Embed Layers and Groups feature.  This means any change in main base project was reflected up(down?) to the other projects next time they are opened.  The main project file has things like, property layer; normal road layers, with labels; road layer with roads for claims.   The other 42 projects have a filtered, and styled, road layer to only show roads in that batch, and its composers (print layouts).

Normally we would use MapInfo for this kind of thing but consider this: There are at least 3 print layouts per claim, each layout could have more then one map frame.  Now with MapInfo only being able to have a 1:1 ratio between the map window and the map frame in the layout you would need at least 3 map windows per claim.  Quick calc:

42 * 3 =  126+n map windows + 126 print layouts (n = extra map frames in layouts)

Each map window has its own copy of every layer, making change once apply every where changes hard.  This of course doesn’t apply to styles as they are stored in the .map (tab) file, but does for labels, style overrides, etc.   I’ll pass.

QGIS is no means perfect for printing or print layouts but the 1:N map window to map frame ratio worked really well for this project.  The styling options in QGIS also helped to change the display of the map depending on what was needed to be shown quickly, one even used the rule-based rendering.

You get the point.
Moving on.

Processing GPS photos with road chainages

This one I am quite proud of.  It’s nothing fancy but still saves a lot of time.  While not really QGIS only but a combination of QGIS+Spatialite it process GPS photos and assigns them a road name and chainage.

The issue: A large influx of GPS photos for the different flood damage projects and the need to process them quickly so that they got assigned to the correct road and chainage.  Now you can map GPS photos easy enough but then you still have to go to each one and assign a road name, chainage, and move it into the correct folder.  To hell with doing that by hand, this is why we invented GIS.

The result is a little (140 line) python script that:

1. extracts the coordinates from each photos,
2. finds the closest (within tolerance) road distance node (distance nodes are generated at 5m intervals along the road, around 800,000 in total for the whole shire),
3. gets the road name, locality, and chainage for that node,
4. creates a folder with that road name,
5. renames the photo with {name} @ {chainage},
6. moves it into the road name folder it is assign to.
7. inserts a record for that photo into the spatialite database that can be viewed in QGIS.

Porting our planning scheme maps

I have been involved in creating, and maintaining, our planning scheme maps for the council.  It’s been a pretty fun project, apart from the constant moving target that is the state planning specifications, but I digress.

Planning scheme in QGIS

This project was done, and still is, in MapInfo. While there is nothing technically wrong with that, it has become a bit more of a pain to maintain then one would hope.  The planning scheme is not just one map but rather a series of different maps all with different scales and requirements.  I’m sure by now you can start to see the issues that can arise:

1. No dynamic scale bar for layouts (not even a scale bar object rather just text and boxes made to look like a scale bar. With no group items feature moving these around is a pain).
2. 1:1 map window to map frame means excessive map windows when the data is all the same with just different views.
3. Legends don’t support ordering, adding items, or groups.
4. With no embedding base maps feature like in QGIS it’s hard to change one thing and apply it to all the map windows/workspaces.

The specifications also ask for lines with symbols along them to show things like bikeways, footpaths etc, something that can’t be done in MapInfo, well it can by using the line style editor but I would rather stab myself in the eye.

The one thing I haven’t fully worked out how to do in QGIS yet is fully automate the printing process. Currently I open MapInfo using a batch file and pass it a workspace and MBX which prints the layouts and exits. I do this for each map type.    In QGIS I have a few options:

1. Create a plugin that runs though each project and prints off its composers.
2. Create a python script that runs from a batch file using qgis.core and qgis.gui QGIS python bindings.
3. add a –code option to the command line of QGIS so that you could run: qgis.exe –noplugins –code “print.py”, which would open QGIS and run the python code and exit.

I’m yet to explore what option is the best for this project but I’ll get back to you on that.  Once I have the above issue sorted I plan on creating the maps in QGIS to see how it would turn out (time permitting)

One thing I didn’t explain very well  in my other post was how to correctly set up value binding between your custom form and QGIS.  I didn’t explain it because at the time I didn’t know how.

The other day I was building a custom form QGIS for a project I am working on. I had named all the fields right, set the ui as the edit form for the layer, but only the line edits were getting bound to the correct values.

 So having a dig around in the code I noticed that QGIS uses the same methods to bind the built-in edit forms as it does for the custom forms, meaning that you must set what kind of control you want to use in Layer Properties -> Fields 

Correctly binding values

First create the form with the controls you need, remember to name them the same as your fields.

Custom form with controls using the same name as the fields

Note that here I have a QComboBox with the FeatureCla name, this will bind the combo box to the FeatureCla field in my dataset in QGIS.

Now set the custom form as the Edit UI for the layer

Set the Edit UI to your form

Tip: You can use relative paths if you store the form along side your project file

 Flick to the Fields tab and set up the Edit Widget type for each field that you have used on the custom feature form.

Set the Edit Widget that matches your control

I have set the FeatueCla field to use Unique values widget, this tells QGIS to collect all the unique values from that column and add them to the QComboBox.  There are a range of different edit widgets you can set

Each will map to a different set of control types (Widgets) e.g. If you want to have a checkbox on your form you must select Checkbox in the Edit Widget list to get it to bind correctly.

Save the properties and head back to you map.  Use the Identify Tool to select a feature.

Values bound to form

And that is it. Pretty cool hey!

Final thoughts

This is one feature I really like in QGIS.  The ability to create custom forms for people to do data entry without the need to build a plugin is very cool.  Couple this the built-in GPS module for QGIS and you have yourself a nice simple field data collection program.

I have some ideas to make this feature even more powerful, but more on that later once I get some time to add it in.

Ok the title is a bit of a lie. QGIS did support MS SQL Server 2008 before by using OGR but this is a native provider so it’s a lot more integrated..

Good news everyone!

QGIS now has a native MS SQL 2008 provider. The provider can found using the new toolbar button (purple icon) or in the MS SQL node in the QBrowser tree. The provider also supports drag and drop import.

The work was sponsored by Digital Mapping Solutions (Australia) and completed by Tamas Szekeres

Any bugs can be assigned to “tamas” on hub.qgis.org.

A big thanks to both Digital Mapping Solutions and Tamas.

This addition will open QGIS up to a whole new set of users who have to use MS SQL but love QGIS.

Currently this is only in master but I will be in the 1.8 release when it comes out.

Note: At the moment you have to have a geometry_columns table in the database in order to connect, this table is the same format used by PostGIS and can be made by importing a layer using the ogr2ogr method. There will be a fix coming for this at some stage.

Gary Sherman has just published a new Python plugin for QGIS that I think people will find very handy, I know I will.  The plugin allows you to run Python scripts inside QGIS for tasks that don’t really require, or warrant, a whole plugin.

Go check out Gray’s post about the new plugin at http://spatialgalaxy.net/2012/01/29/script-runner-a-plugin-to-run-python-scripts-in-qgis/

The new plugin can be installed via the Plugin Installer using the “runner” or “script”.  The Plugin Installer is another one of my favorite plugins for QGIS, being able to push out a new plugin and know that everyone can get it is a good feeling :)

The rule based rendering in QGIS has just got a make over to improve in some of the old usability issues it used to have.  Most of the improvements are UI related. If you would like to try them out you will need to grab a copy of the latest dev build (qgis-dev in OSGeo4W)

Main improvements include:

• Nested rules.  If the parent rule evaluates to false none of the child rules are applied. This replaces the priority system in the old dialog.
• Disable symbol for rules. Rules with no symbol only act as a check for the child rules e.g nothing is rendered for the rule but child rules still are (unless also disabled).
• Drag and Drop rules (multi-selection is supported).  Rules can be dragged onto other rules in order to nest them and set up a rendering hierarchy.
• Inline editing of rule labels, expressions, scales
• Overall tweaks to the dialog

The new rule dialog

As you can see in the screenshot, the rules are now organized in a tree which clearly expresses which rules should be applied and when.

In the example above, all the rules under the Sealed rule will only be applied if that rule is true. The old system would have you managing all rules in one big list and dealing with priorities in order to get the rules to apply right, the new dialog is a major improvement.

And the results! As you can see below, QGIS will only render the colored squares if the Sealed rule is true otherwise it just shows a green line.

The rules applied

The work was sponsored by Ville de Morges, Switzerland and developed by Martin Dobias.  Thanks to both of them for these improvements.

More info:

• See Martin’s notes for a more detailed breakdown, and maybe some future stuff to come, of the improvements  http://lists.osgeo.org/pipermail/qgis-developer/2012-January/017909.html

Note: As this is a brand new feature there might be some bugs, or things that don’t quite work as expected. If you do find something don’t hesitate to file a bug report at hub.qgis.org so it can be fixed, or at least known about.

QGIS finally has expression based labels! (Although you must be running latest dev build)

What does that mean? Well QGIS used to be only able to label with a field from the layer, very limiting if you need to make a nice looking label string. Now you can use expressions (eg ‘Up ‘ || US_Invert || ‘some more text’ ) to label the feature, just like this.

Example of expression based labels

This is something that I missed a lot when moving from MapInfo to QGIS.  After opening a ticket on the QGIS bug list and nothing happening with it for a couple of months (not that I expected anything to, everyone is busy enough as is.  I don’t expect the devs to just jump at all my requests) so I decided I should at least attempt adding it myself.  The joys of open source!

Turns out adding the expression labeling was the easy part, however there was no good generic expression string builder that I could use to build the expression string.  QGIS already had the expression builder for the query window and the field calculator, however the code was very tied down to only work for those implementations, plus they didn’t scale with the increasing function list.  I’m not going to go and make yet another dialog just for the labeling.  Uniformity is the key to good user experience.

After searching around to see what other GIS systems did to get some inspiration it seemed that every example that I came across was, in my opinion, poor.    Although there was one ArcGIS idea ticket that gave me a few ideas http://ideas.arcgis.com/ideaView?id=087300000008IbHAAU

So with that I started working on a generic expression builder that could be used to build an expression string anywhere, replacing the query window and field calculator in due time. One ring widget to rule them all, one ring widget to bind the, etc.

The result

Generic expression builder

Key features of the new expression builder:

• Live validation of expression
• Real time searching
• Live output preview
• Help on selected item.
• Easier to add new functions without changing the UI.  Function list is read right from the expression parser.  No more hidden functions.
• Reusable widget

If the expression hits an error while you type you will be shown an “Expression is invalid” warning (yes I know it’s wrong in the screen shot). Clicking (more info) or hovering over the expression area will show you the error.

Expression has error

Searching can done by using the search box at the top. The function list will reduce to show only functions or fields containing that string (Note: it is case sensitive at the current time) .

Searching for a function or field name.

Still to do

As with all programming it is never bug free so I expect, now that it is open to wider testing, that there might be a few that need to be addressed .   There is also very limited function help written for the functions, although if anyone is willing to have a go at it I’m more than happy.

If you do find any bugs will the widget/dialog you can open a ticket at http://hub.qgis.org/projects/quantum-gis/issues , assign it to me and I’ll see what I can do.

In the works

• Simple Syntax highlighting
• Recent expression used list
• Saved expressions
• Auto bracketing (maybe)
• UI tweaking

I would like to thank Martin from the QGIS team for reviewing my code all though the project and helping me improve the code and idea.  Also thanks to all the other people who gave ideas for the widget/dialog.

Enjoy!

I tell you. It always amazes me how much cool stuff you can do with great open source GIS software these days. GDAL is one of those great open source projects that I have just found a great use for (apart from just opening every raster type under the sun in QGIS).

GDAL has the ability to generate contours from a DEM, something that I have always wanted to try for my town but have never been able due to lack of a good DEMs.

Recently we purchased a set of DEMs that cover a large area as part of a study. Each DEM uses a grid size of 1mx1m. Prefect for generating contours.

Using the GdalTools QGIS plugin.

First make sure that you have the latest version of the GdalTools plugin installed (GdalTools should be installed by default with QGIS. If it’s not, search “Gdal” in the plugin installer). Enable the plugin once it’s installed.

Load the DEM into QGIS using the Load Raster icon.

DEM loaded in QGIS

Now head up to the menu Raster->Extraction->Contour

Raster menu in QGIS

Select the settings that you need. For this DEM I am going to generate 250mm contours.

Contour dialog.

Take note of the text area at the bottom of the dialog as that is the exact command sent to GDAL in order to generate the contours. If you take a copy of that you can run it on the command line for batch processing later.

Hit ok.

250mm contours from the DEM

BAM! :)

Using the command line/shell.

Using QGIS for a one off DEM is fine and dandy but what if you have 3000 DEMs that you need to process. To hell with doing that by hand!

Remember the contour tool in QGIS told us the exact command line args to use, so creating a shell script for automating the process is pretty easy.

for f in *.asc
do
  echo "Processing $f"
 gdal_contour -a ELEV -i 0.25 $f $f-250mm.shp
done

The above code will loop though the current directory and process all the DEMs generating 250mm contours for each one. It saves each new contour file as {filename}-250mm.shp. You will need to change *.asc to whatever format your DEM is in.

Copy the above code into a file somewhere and call it generate_contours.sh. This can then be called from the command line using

sh generate_contours.sh

Running sh on Windows

If you’re a windows user you will need to run OsGeo4W Shell in order to use sh.

Loading OsGeo4w shell.

Once the shell is loaded you can just call:

sh generate_contours.sh

Output from running generate_contours.sh

Final remarks

I ran the above sh script on a folder with about 2500 DEMs and it took around 4 hours to complete the whole folder. Of course performance will vary but it seems pretty quick considering.

Once again the possibly to use open source GIS tools to get my work done is amazing.  No expensive software here.

So far I’m not aware of any line smoothing/generalizing abilities using GDAL/OGR.  Although you can import the contours into GRASS and use that: http://grass.osgeo.org/wiki/V.generalize_tutorial

You can also generate the contours using GDAL via Python but that is a topic for another day.

Another cool open source project that I have become a part of (as a QGIS packager and tester) is the OSGeo-Live project.  The OSGeo-Live project is a live DVD/USB/Virtual Machine built on xUbuntu(striped down Ubuntu linux) that has a lot of cool open source geo spatial programs all set up and ready to use.

The OSGeo-Live project contains:

• Browser clients
• A small sample of crisis management software
• All the popular database engines (PostGIS, SpaitalLite etc)
• Pretty much all the open source desktop GIS apps (QGIS, uDig etc)
• Open Source GPS navigation apps and globes.
• A collection of handy spatial tools
• A ready to go web services ready to try in your browser or desktop GIS.
• Some sample data to get started with for each project
• And quick starts for each program.

The full list of software contained on the OSGeo-Live project can be found at http://live.osgeo.org/en/overview/overview.html

This is a good project if you want to get into the OSGeo tools and experiment but don’t want to install them on main machine until you know what you need.

As it is a live DVD/USB/Virtual Machine some apps will run slower than what they do on a native install but overall the speed is usable and good enough for testing.

Even better is that it was born in Australia :)

The project is also commercial supported by a Australian company http://lisasoft.com

So give it a try if you are interested in the OSGeo movement, which you should be if you are reading my blog :)