QGIS Planet

Scottish QGIS User Group 19th March 2014

scottish thistleThe Scottish QGIS user group meeting is happening on 19th March 2014 in Stirling at the Stirling Management Centre.  Doors open at 9:30 with a 10:00 start and a planned finish of 16:00.  Registration is through Eventbrite and there are 50 places available working on a first come, first served basis.

Details on how to get to the Stirling Management Centre are available here.

The agenda will be published a bit closer to the time once speakers have been finalised.  If you would like to present let me (Ross McDonald) know as it would be good to have a mix of input to the day.  There are both 20 minute and “lightning talk” 5-10 minute slots available.

A big thank-you to thinkWhere for hosting this first QGIS user group event in Scotland.

If you want two days of geo discussion then think about attending the AGI Scotland – Future Cities event in Glasgow the day before.  Check the AGI Scotland website for more details.


1st UK QGIS user group meeting -Wales

This is just an interim post, the full blog is being written up by Kevin Williams and You Tube videos are being put together by Shaun Lewis. To wet your appetite, you can see the introduction presentation and agenda on SlideShare.

Update 27th January 2014 :-

A long overdue update.

Cymru am byth!

The event was a big success!  We had around 50 people attend from various organisations throughout Wales.  The event was kindly sponsored by Exegesis (http://www.esdm.co.uk/) and Astun Technology (www.http://astuntechnology.com/)  both of which gave excellent presentations.

Huge thanks to Shaun Lewis and his manager, Paul Funnell for hosting the event in such a professional manner.  The venue and technology were second to none.

There’s been lots of feedback via email and phonecalls, together with a survey carried out by Shaun Lewis of Brecon Beacons National Park.

From all of the comments, the future is bright for QGIS in Wales.  There is an ever-increasing interest and numbers of users, not just in QGIS but in the full SDI provided by QGIS, postgis, geoserver etc.  I can see the group incorporating elements of the open source stack to give a more detailed advice, collaboration and guidance, but let’s walk before we can run!

-Kevin

Watch this space for more news from around Wales in the QGIS arena!


Scottish QGIS User Group

scottish flagThe inaugural Scottish QGIS user group meeting is being planned and organised for mid-March next year.  If you would like to participate, I am looking for user presentations, case studies, map displays and practical demos and tutorials.

The event will be held in Stirling and will be a full day of networking and open-source geo-goodness.  Full details will go out in the new year and will be available through Eventbrite, this blog, the Google+ group, Twitter and probably a heap of other channels.  After the success of the English and the Welsh events we are hoping the Scottish event will raise the bar even higher.  Please use the contact form to get in touch with me, Ross McDonald.


User Poll

In the last blog I said that I’d pull together the results of a survey, to get a feel for UK qgis users. Unfortunately the number of responses was a bit disappointing with only 28 filling out the survey. However here is what was said….

ExpertiseDonation

workshopsLocations

I’m no statistician and the low return rate makes it harder to make concrete statements but a few things can be generalised upon.

Two-Thirds said that they would be willing to donate to the QGIS project. This is great news; so hypothetically speaking if all those that said yes (21 of you) gave a tenner, that would be £210 we could donate to the QGIS project. I can’t remember the static on the number of pizza’s consumed at the QGIS developer meeting held in Brighton earlier this year was, but i should imagine that £200 worth of pizza could fuel a lot of developers to write a lot of cool qgis code!

On the workshop side of things, there seemed to be a great interest in Plugins, Python and Databases. This has certainly given me some ideas on the kind of workshop that the QGIS User group could hold next year.

Thanks to everyone that took part in the survey.


How the West was Won (or QGIS rocks!)

gazetteer search list!<tl><dr>

Local Authorities work with people and assets. Most people have an address and assets are mostly located somewhere. As my old geography teacher used to say, “everything has geography, geography is everything”. For people who work in Local Authorities being able to find an accurate and up-to-date address easily is key to being able to deliver a service quickly and efficiently. If GIS officers had their way even more of the Council back office systems across the country would hook up to the corporate GI database and do cool stuff with spatial information.

At the recent AGI GeoCom and FOSS4G conferences in Nottingham during Maptember QGIS 2.0 was launched and garnered a lot of attention from people interested in finding solutions to save money and time and still deliver great results. I came away with a resolution to push for the use of QGIS at work in an attempt to get a desktop GIS onto more desktops while not breaking the bank. I had also met Simon Miles (@geosmiles) and eagerly followed his talk of creating a hybrid GI infrastructure using a mix of open-source and proprietary software.

QGIS fitted the bill with its extensive range of functionality, growing list of available plugins, super easy connectivity to a range of data sources including Oracle, SQL Server and PostGIS, all of which we use. The one thing it was lacking was a search plugin to let the users search the corporate address gazetteer for postal address and street names. Our existing GIS has such a plugin and it works well but not with the imminent upgrades. There had to be something that could connect to the gazetteer table in our PostGIS database, search and display an address.

qgis gazetteer pluginEnter the QGIS Gazetteer Plugin developed by Astun Technology, Nathan Woodrow and Matt Walker. Add some keen peeps from the UK QGIS User Group and some social networking and stuff begins to happen.

After downloading and installing the plugin I soon hit a brick wall – or rather a firewall – the security at work wouldn’t let the Yahoo and Geonames gazetteers work and we don’t have Astun’s iShareGIS software stack. I delved into the code to see if I could work out what was happening and hit another wall – how does this Python stuff work? Nought for two after one over.

After some chat on the UK QGIS Google+ group I found some people in Local Government across the UK who have managed to get the plugin to work and work well.  Simon Miles (Royal Borough of Windsor and Maidenhead), Kevin Williams (Neath-Port Talbot) and Matt Travis (Dartmoor National Park) rallied to my call and soon I had some code in my grubby mitts.  An hour after that I had a working plugin searching our address and street gazetteers, postcodes and planning applications.  Listed buildings, building warrants and others soon followed.  I shared the plugin with some of our power users and they were very impressed and have even started asking for enhancements.

This whole process is a great example of how a User Group can work when coupled with FOSS4G fans, social media, open-source software and a common problem to solve.  Being able to present a working solution to users and managers in just a few days certainly strengthens the case for using QGIS alongside the traditional GIS tools and maybe, in time, replacing them.

What follows is a rather more technical description of the installation and configuration of just about the single most useful application a Council officer could ask for. If you want the short version and can do the tech stuff then go to https://github.com/mixedbredie/qgis-gazetteer-search and get it, otherwise hang in there to the end…

What you need:

1. QGIS 2.0 installed

2. the QGIS Gazetteer plugin installed

3. a PostGIS database with some gazetteer tables

4. an Apache2 web server

how-it-works

1. QGIS

Install QGIS 2.0 – I used the standalone installer from http://www.qgis.org/en/site/forusers/download.html

2. QGIS Gazetteer Plugin

Install the QGIS Gazetteer plugin from https://github.com/AstunTechnology/QGIS-Gazetteer-Plugin by downloading the zip file and extracting into your .qgis2 folder. This is located at

C:\Documents and Settings\<Username>\.qgis2\python\plugins\ (Windows XP) or 
C:\Users\<Username>\.qgis2\python\plugins\ (Windows 7+).

Make sure that the extracted directory is called “gazetteersearch”. Matt Walker (@_walkermatt) has upgraded the plugin to work with QGIS 2.0 and added a few enhancements like an OpenStreetMap search.

In the “gazetteers” sub directory there a “config.ini” file which needs to be edited to list the datasets we want to search. I’ll show how to connect to the address and street gazetteers and a table of postcode areas.

I deleted the GeoNames, Yahoo and AstunTechnology lines and inserted the following:

[LLPG]
gazetteer:llpg
[LSG]
gazetteer:lsg
[Postcode]
gazetteer:postcodes

Create three files in the gazetteers folder – llpg.py, lsg.py and postcodes.py

In the llpg.py file put:

import json
import collections
url = "http://10.135.1.69/cgi-bin/llpg_pg.py"
params = {
    'address': '##searchstring##'
}
def parseRequestResults (data):
json_result = json.loads (data)
    for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['address']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 1250
        result.epsg = 27700
        yield result

In the lsg.py file put:

import json
import collections
url = "http://10.12.345.678/cgi-bin/lsg_pg.py"
params = {
    'address': '##searchstring##'
}
def parseRequestResults (data):
json_result = json.loads (data)
for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['address']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 2500
        result.epsg = 27700
        yield result

In the postcodes.py file put:

import json
import collections
url = "http://10.12.345.678/cgi-bin/postcodes_pg.py"
params = {
    'postcode': '##searchstring##'
}
def parseRequestResults (data):
    json_result = json.loads (data)
    for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['postcode']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 2500
        result.epsg = 27700
        yield result

You need to set the correct server IP address in the URL and link to the correct file in the cgi-bin. You can also set the zoom scale for the results and I use ESPG:27700 as everything we have is in British National Grid.

If you restart QGIS and enable the plugin you’ll see there are now four search options in the drop-down list. Right, now to make sure PostGIS has the correct information.

3. PostGIS Database

The installation and configuration of PostgreSQL and PostGIS is more than this post is going to address but you’ll need to have one. With some tables in it with some data that can be searched. We load our address and street gazetteers into PostGIS on a daily basis and these tables are used by the plugin for searching and displaying records. Our tables have a UPRN (Unique Property Reference Number) field and an ADDRESS field with a full address in it (name number street town locality postcode). The address gazetteer has a point geometry. The street gazetteer has a USRN (Unique Street Reference Number) and a NAME field with the full descriptive name of the street in it. It has a line geometry. Almost any table can be searched as long as it has a unique id, a text field for searching, and some geometry.

Our address gazetteer already had the correct fields in so nothing had to be done but the street gazetteer needed some changes. I created a view of the streets to change the name of the fields to fit with what the plugin was expecting:

CREATE OR REPLACE VIEW angusdata.search_lsg_streets AS
SELECT a.usrn AS uprn,
btrim(pg_catalog.concat('usrn:', btrim(a.usrn::text), ' ', btrim(a.street::text), ' ', btrim(a.locality::text), ' ', btrim(a.town::text))) AS address,a.geometry
FROM lsg_streets a;

This changes the USRN field to UPRN and concatenates USRN, STREET, LOCALITY and TOWN to create an ADDRESS field and then adds the geometry.

The Postcodes table has polygon geometry and a postcode field and the python script that creates the web service has been tweaked to work with this. The next section will explain how to create the web services on Apache2.

4. Apache

I have a local Apache2 webserver running on my PC but it’s pretty easy to get one set up on server if you chat to the right people :-) so get one installed (http://www.apachehaus.com/cgi-bin/download.plx). The plugin uses the web services running in the Apache2 cgi-bin to connect to the PostGIS database to query the gazetteers. You’ll also need to have Python (http://www.python.org/download/releases/2.7.5/) installed on the same machine. And the Python psycopg2 module (http://www.stickpeople.com/projects/python/win-psycopg/).

First up, the Apache2 server needs to be configured to allow use of the cgi-bin. Edit the httpd.conf file and make the following changes.

Uncomment the following lines to enable the modules:

LoadModule access_compat_module modules/mod_access_compat.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule rewrite_module modules/mod_rewrite.so

Set the server name to its IP address:

ServerName 10.12.345.678:80

Allow access to the server:

<Directory />
  Options Indexes FollowSymLinks ExecCGI
  AllowOverride All
  Order allow,deny
  Allow from all
  Require all granted
</Directory>

Set the file that Apache will serve if a directory is requested:

<IfModule dir_module>
  DirectoryIndex index.html index.htm index.php index.php3
</IfModule>

Allow access to the CGI directory and enable it to handle python scripts:

<Directory "${SRVROOT}/cgi-bin">
  Options Indexes FollowSymLinks ExecCGI
  AllowOverride All
  Order allow,deny
  Allow from all
  Require all granted
  AddHandler cgi-script .cgi .py
</Directory>

This is a pretty basic configuration and it could probably be improved but it works on my system.

Then in the CGI-BIN directory create some files, one for each web service and to match the gazetteers listed in the config file. I created three files called llpg_pg.py, lsg_pg.py and postcodes_pg.py.

In the llpg_pg.py file the SQL query selects the records that match the search string and uses the point geometry of the records to return to the plugin.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.uprn, a.address, ST_X(a.geometry), ST_Y(a.geometry)
from angusdata.address_gazetteer a
where a.address ilike '%%' || (%(p_address)s) || '%%'
order by address"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_address": form["address"].value})
list = []
for record in cursor:
    data = dict (zip (["uprn", "address", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

In the lsg_pg.py file the SQL query selects the records from the view and converts the line geometry to a centroid point.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.uprn, a.address, ST_X(ST_Centroid(a.geometry)), ST_Y(ST_Centroid(a.geometry))
from angusdata.search_lsg_streets a
where a.address ilike '%%' || (%(p_address)s) || '%%'
order by address"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_address": form["address"].value})
list = []
for record in cursor:
    data = dict (zip (["uprn", "address", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

In the postcodes_pg.py file the SQL query turns the polygon geometry into a point geometry using the PostGIS ST_PointOnSurface function. The psycopg2 parameters have been tweaked to use the postcode tables fields.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.postcode, ST_X(ST_Pointonsurface(a.geometry)), ST_Y(ST_Pointonsurface(a.geometry))
from thirdparty.os_codepointpoly a
where a.postcode ilike '%%' || (%(p_postcode)s) || '%%'
order by postcode"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_postcode": form["postcode"].value})
list = []
for record in cursor:
    data = dict (zip (["postcode", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

Make sure the #!shebang at the top of the python script has the correct path to your Python installation.

Check the Apache2 log files if there are any errors.

Right, we have QGIS installed with the gazetteer search plugin configured to use the new gazetteers.  The PostGIS tables and views are presenting the data in format required by the web services. The Apache2 server is configure to allow access and the python scripts are in the cgi-bin. Fire up QGIS, refresh the plugin and enter a search term and hit GO!  Awesome!  Mine works, at least.

None of this would have been possible without the UK QGIS group and the input from Matt, Kevin and Simon.  Big thanks to Matt Walker and Jo Cook for recent updates to the plugin.  Use the comments to tell us how to do it better.


OS OpenData Workshop – QGIS in the Classroom

Last week I attended an Ordnance Survey OpenData MasterClass in Exeter. This was one of a series of seven masterclasses that the Ordnance Survey were running across the country. They were aimed at letting people know how to get the best of the  OpenData products that they provide.

The workshop was delivered in a format of combining theory and practical sessions. They were aimed at people of various experience; from those new to working with location data to the more advanced users wanting to brush up on their skills.

What was encouraging from a QGIS point of view was that a lot of the course was taught using QGIS 2.0.  After an interesting introduction to the history and current standing of OpenData by Ian Holt we were given our first taste of QGIS. Ably led by Steve Kingston we were shown:

  • How to navigate around QGIS;

  • Import a range of vector and raster datasets (thankfully pre-downloaded);

  • Build virtual rasters (very useful if you’re dealing with lots of OS tiles);

  • Merge shapefiles into one layer

  • Create thematic maps from LSOA boundaries and ONS data (like the one below)  - here I learnt  you can now load non-geographic data from a csv straight into QGIS without using a csvt file, yay!

imd_compressed

After Lunch and a quick insight by Chris Parker  into the next Geovation Challenge , Chris Wesson from the Carto Design team at the OS gave a cartographic design workshop. This started off with a talk about the basic design principles behind cartography and how you should bear these in mind when making a map.

He then showed us how to import vector map district vector data and then style these in a various ways –  I particularly found his tips on creating road casing useful. Some of the maps being were produced were of a really good standard, especially when you this was many people’s first time using a GIS of any sort.

I thought that this was a really worthwhile experience and would recommend going to one if you get the chance in the future (according to their website there are still masterclasses to come in York and Nottingham). I learnt a fair bit myself as I always think it’s useful to watch someone else use a product that you’ve used in isolation. They might do something that you’ve never seen before that’s loads quicker or show you a hidden feature you’ve not come across.

It was encouraging to see people of various backgrounds present, parish councils in particular. As someone who works for a local authority that has many Parishes i’m very keen to promote the use of open data by parishes and local groups and get them using QGIS, etc

In the New Year I’m going to look into having a QGIS South West event and hope to reach out to all  types of QGIS users across the region. I’ll keep you posted on the exact details via the Google+ group but if you read this and are in or around the South West and want to share your experiences of using QGIS (no matter how big or small) then please get in touch.

That’s it for now. Just leaves me to thank all the people that made the OS Masterclasses possible I’m sure they’ve been a success so hopefully they’ll be even more next year to look forward to!
Matt


Programme: South-East QGIS User Group

The agenda for the South-East user group meeting, being held at Imperial College, on the 2nd April is here! We’ve got a mixture of speakers, two workshops and a ‘its your floor’ feature where by you can stand up and let people know what your up to with QGIS! If you haven’t got a ticket, there are still some available.

Time

9.30 – 10

Arrive / teas & coffee / networking

10 – 10.15

Simon Miles: Introductions etc

10.15 – 10.45

David McDermott: QGIS and Atlas: automatic map generation

10.45 – 11.15

Mike Saunt: QGIS and PostGIS: The perfect marriage!

11.15 – 11.30

Break

11.30 – 12.00

Jerry Clough: QGIS and OSM: All you needed to know!

12.00 – 12.30

Andrew Bell: QGIS and PostGIS: TBA

12.30 – 1.00

Lunch: Provided by the Ordnance Survey

Stream 1

Stream 2

1.00 – 2.45

Introduction to QGIS:

Based on the Ordnance Survey 2013 Masterclass tour. Ordnance Survey will give an introduction to QGIS and getting started with OS Opendata.

Python & QGIS:

Lutra Consultancy Pete Wells will demonstrate how you can introduce python into QGIS.

2.45 – 3.00

Break

3.00 – 3.30

Its your floor!

Your chance to share what you’ve been up to with QGIS

3.30 – 4.00

Round up / Discussion / User Poll / Geo-beers?


Scottish QGIS User Group Overview

Scottish QGIS User Group
Stirling, 19th March 2014

It was a long time coming but the wait was worth it. Forty two excited QGIS users and open-source GIS enthusiasts arrived at the Stirling Management Centre on a brilliantly sunny March day.  People had traveled from all over the UK to make the day happen: Charley Glynn from OS in Southampton, Pete Wells, Martin Dobias and Saber Razmjooei from Brighton as well as others from Aberdeen, Inverness, Dundee, Edinburgh, Glasgow, Cumbria and most places inbetween. The event was supported by thinkWhere, based in Stirling, and Neil Benny and Heikki Vesanto provided suitably geeky geo entertainment.

Neil Benny, QGIS EvangelistFirst up was Neil Benny (thinkWhere) who provided us with an overview of QGIS through the years to the current top features available in version 2.2 “Valmiera”. The questions on everyone’s minds were answered when he presented a series of slides outlining the benefits of using open source software, highlighting the savings and investments and the importance of investing in training. His top 10 feature comparison of proprietary v open source desktop GIS provoked much discussion.

After a coffee break I presented a short talk on how Angus Council is moving to a mixed hybrid GIS environment to take advantage of the flexibility of the open source licence and the variety of tools available to deliver results. Available here http://vimeo.com/89959143

Martin Dobias of Lutra Consulting and core QGIS developer revealed some of the performance enhancements available in the development version of QGIS. The multi-threaded multi-core rendering impressed everyone and will prove a huge draw card to seasoned GIS’ers used to single threaded applications.Martin Dobias, Core Developer

Saber Razmjooei (Lutra) filled in an open slot talking about the autoTrace plugin they developed for a group of Local Authorities across the UK. Modeled on the MapInfo trace tool it forms a key part of a lot of Council workflows and is a good example of how future plugin development work can deliver savings.

Pete Wells, plugin developerPete Wells (Lutra) delivered a very comprehensive overview of Python and QGIS and how they interact at different levels through the python bindings. There was a lot of interest in this and this was reflected in the feedback forms we collected where Python, plugins, hands-on workshops and tutorials feature high on the list of wants.

Charley Glynn (OS) unveiled some fantastic cartography using the OS vector products of MasterMap, VectorMap Local and District. He also revealed the work OS has been doing to make corporate styles available to the public and the Ordnance Survey’s bias towards open source software. Again the feedback forms revealed a desire to get hands on with QGIS to create good looking custom cartography. The next Scottish user group meeting will definitely be having some hands-on workshops.Charley Glynn, OS Cartographer IMG_20140319_152620

Heikki Vesanto (thinkWhere) bravely ventured into live demos of how to connect to just about any spatial data format available. Local files, local databases, WMS feeds, WFS feeds, text files, CSV and URLs with images and custom map templates using the Atlas generator. An excellent overview of just how flexible QGIS is when it comes to consuming data and converting data to almost every format supported by OGR and GDAL.

Thanks must go to the generosity of thinkWhere in supporting a feature filled programme of presentations and keeping us topped up with coffee. As a result the first Scottish QGIS user group meeting was a success and there is definitely a desire for more events like this.

Slides and videos of the presentations will be available here shortly.


Programme: Scottish QGIS User Group

The programme of events for the Scottish QGIS user group meeting in Stirling is now available:

Time

Event

Speaker

9:30

Registration

10:00

Welcome Ross McDonald / Alan Moore

10:30

QGIS Evangelism Neil Benny

11:00

Break

11:30

Data auditing in PostGIS Mike Saunt

12:00

QGIS performance enhancements Peter Wells

12:30

Lunch

13:30

Land Terrier Management Mike Saunt
Developing QGIS Martin Dobias
User Group, eh? Ross McDonald

14:30

Break

15:00

Cartography in QGIS Charley Glynn
Data Connections in QGIS Heikki Vesanto

16:00

Close
Geobeers

You can download the ScottishQGISUserGroupAgenda here.


OSGeo Code Sprint, Vienna

This is how OSGeo happens.  These are the folk who bring us a lot of that open-source geo-spatial goodness. You can follow the code sprint on Twitter using the hashtags #csprint and #viennacodesprint14

 


Back to Top

Sustaining Members