QGIS Planet

3rd Portugal QGIS user meeting (Covilhã) and QGIS‬ Conference 2015 (University of Copenhagen)

In the next weeks/months are due a couple of QGIS events where NaturalGIS will be present with presentations and workshops.

May 18th and 19th we will be at the QGIS Conference 2015 organized by the University of Copenhagen, Forestry College, that will take place in Nødebo, Denmark. This event is directly linked with the 13th QGIS Developer Meeting, that will take place from May 20 to 22 in the same location.

At the QGIS Conference we will make a presentation about how to get help/support for QGIS and a workshop about QGIS Server and how to publish OGC services (WMS/WFS/WCS) and webgis applications.

QGIS Conference 2015

June 5th we will be in Covilhã, Serra da Estrela, Portugal, to join the 3rd Portugal QGIS user meeting where we also will make a presentation (the same as in Nødebo) and a workshop about using the QGIS Processing toolbox.

3rd Portugal QGIS user meeting

3rd Portugal QGIS user meeting (Covilhã) and QGIS‬ Conference 2015 (University of Copenhagen)

In the next weeks/months are due a couple of QGIS events where NaturalGIS will be present with presentations and workshops.

May 18th and 19th we will be at the QGIS Conference 2015 organized by the University of Copenhagen, Forestry College, that will take place in Nødebo, Denmark. This event is directly linked with the 13th QGIS Developer Meeting, that will take place from May 20 to 22 in the same location.

At the QGIS Conference we will make a presentation about how to get help/support for QGIS and a workshop about QGIS Server and how to publish OGC services (WMS/WFS/WCS) and webgis applications.

QGIS Conference 2015

June 5th we will be in Covilhã, Serra da Estrela, Portugal, to join the 3rd Portugal QGIS user meeting where we also will make a presentation (the same as in Nødebo) and a workshop about using the QGIS Processing toolbox.

3rd Portugal QGIS user meeting

Datum transformations with NTv2 grids in QGIS: the easy way

NTv2 is a format/method to allow transform coordinates (and vector/raster layers) from one coordinate reference system to a different one (involving a change of datum) with a small error: the precision is usually around a few centimeters (or less) and is much better than the one that can be obtained by reprojecting layers using the standard coordinate reference systems definitions, as they are available in common GIS software/libraries/toolboxes.

Ntv2 Datum transformations precision
An example of the precision given by NTv2 tranformations
compared to the one obtained by reprojecting layers using standard CRS definitions.

Note: This is a rather complex matter so you may want to read some tutorial about it. One of the best we have ever read is this one [Portuguese or English].

The NTv2 method make use of binary files (.gsb) called ”grids” and is fully supported by libraries like Proj and software like GDAL/OGR. Until not long ago the availability of these grids was scarce but in recent times they have become quite easy to find and are freely available for many countries.

Unfortunately using this grid is not easy for the common GIS user: both command line utilities and Desktop GIS software need custom CRS definitions, downloading and copying files to specific locations and eventually do more configurations. See for example the following tutorials for QGIS or ArcGIS.

For the cases when grid files are freely available and redistributable we implemented an easier solution in the form of a QGIS plugin for its Processing toolbox. With this plugin the operation is as simple as just selecting the coordinate reference system of the input layer and the transformation (direct or inverse).

Ntv2 Datum transformations for the QGIS Processing toolbox
Ntv2 Datum transformations GUI

The plugin already comes with support for grids/transformations for a few countries and more will be added given that ntv2 grids are freely available and redistributable. If you want to have a certain grid/transformation/country added please open a request here

https://github.com/NaturalGIS/ntv2_transformations/issues

including all the necessary details.

Ntv2 Datum transformations for the QGIS Processing toolbox
Ntv2 Datum transformations for the QGIS Processing toolbox

Note2: The plugin was directly derived from a similar one that we helped develop for the Portuguese QGIS community/user group for the specific context of datum transformations for Portugal. At some point we felt that this plugin could have been useful also for users from other countries so we released an “international” version of it.

Datum transformations with NTv2 grids in QGIS: the easy way

NTv2 is a format/method to allow transform coordinates (and vector/raster layers) from one coordinate reference system to a different one (involving a change of datum) with a small error: the precision is usually around a few centimeters (or less) and is much better than the one that can be obtained by reprojecting layers using the standard coordinate reference systems definitions, as they are available in common GIS software/libraries/toolboxes.

Ntv2 Datum transformations precision
An example of the precision given by NTv2 tranformations
compared to the one obtained by reprojecting layers using standard CRS definitions.

Note: This is a rather complex matter so you may want to read some tutorial about it. One of the best we have ever read is this one [Portuguese or English].

The NTv2 method make use of binary files (.gsb) called ”grids” and is fully supported by libraries like Proj and software like GDAL/OGR. Until not long ago the availability of these grids was scarce but in recent times they have become quite easy to find and are freely available for many countries.

Unfortunately using this grid is not easy for the common GIS user: both command line utilities and Desktop GIS software need custom CRS definitions, downloading and copying files to specific locations and eventually do more configurations. See for example the following tutorials for QGIS or ArcGIS.

For the cases when grid files are freely available and redistributable we implemented an easier solution in the form of a QGIS plugin for its Processing toolbox. With this plugin the operation is as simple as just selecting the coordinate reference system of the input layer and the transformation (direct or inverse).

Ntv2 Datum transformations for the QGIS Processing toolbox
Ntv2 Datum transformations GUI

The plugin already comes with support for grids/transformations for a few countries and more will be added given that ntv2 grids are freely available and redistributable. If you want to have a certain grid/transformation/country added please open a request here

https://github.com/NaturalGIS/ntv2_transformations/issues

including all the necessary details.

Ntv2 Datum transformations for the QGIS Processing toolbox
Ntv2 Datum transformations for the QGIS Processing toolbox

Note2: The plugin was directly derived from a similar one that we helped develop for the Portuguese QGIS community/user group for the specific context of datum transformations for Portugal. At some point we felt that this plugin could have been useful also for users from other countries so we released an “international” version of it.

Formação SIG: Produção cartográfica com QGIS

A NaturalGIS acrescentou à sua oferta de cursos QGIS uma formação de produção cartográfica, que permitirá aprender a produzir eficientemente mapas de elevada qualidade recorrendo às funcionalidades mais recentes de QGIS.

O curso cobrirá as opções avançadas de simbologia para camadas vectoriais e raster e etiquetas, o compositor de mapas e automatização de produção cartográfica, entre outros. Este curso é ideal para aprender tirar o maior partido do QGIS para criar mapas temáticos e cartografia para os instrumentos de gestão territorial, como por exemplo, PDM, PP, PU, etc.

A primeira edição terá lugar em Lisboa de 4 a 6 de Maio 2015.

Mais informações em: https://www.naturalgis.pt/formacoes.html#tr3

Inscrições em: https://www.naturalgis.pt/subscricao.html

Mapas espantosos com QGIS!
Aprenda a criar mapas tematicos com QGIS

Conteúdo Programático:

  1. Controlar ordem das camadas
  2. Simbologia vectores
  3. Simbologia rasters
  4. Simbologia por categorias
  5. Simbologia por quantidades
  6. Simbologia baseada em regras
  7. Simbologia avançada - níveis
  8. Modos de rendering e transparência
  9. Etiquetas
  10. Criação de símbolos compostos
  11. Gestor de símbolos
  12. Gravar, Importar e exportar símbolos
  13. Compositor de impressão
  14. Elementos do mapa (Escala gráfica, Norte, Legenda,…)
  15. Grelhas de coordenadas
  16. Mapas de enquadramento
  17. Optimização da produção de cartografia
  18. Filtrar legenda consoante visibilidade
  19. A ferramenta “atlas”
  20. Criar séries de mapas com a ferramenta “atlas”
  21. Presets e estilos múltiplos
  22. Controlo dos elementos e formato do mapa através dos atributos
  23. Criação de cartogramas
  24. Criação automática de mapas de ocorrência

Formação SIG: Produção cartográfica com QGIS

A NaturalGIS acrescentou à sua oferta de cursos QGIS uma formação de produção cartográfica, que permitirá aprender a produzir eficientemente mapas de elevada qualidade recorrendo às funcionalidades mais recentes de QGIS.

O curso cobrirá as opções avançadas de simbologia para camadas vectoriais e raster e etiquetas, o compositor de mapas e automatização de produção cartográfica, entre outros. Este curso é ideal para aprender tirar o maior partido do QGIS para criar mapas temáticos e cartografia para os instrumentos de gestão territorial, como por exemplo, PDM, PP, PU, etc.

A primeira edição terá lugar em Lisboa de 4 a 6 de Maio 2015.

Mais informações em: http://www.naturalgis.pt/formacoes.html#tr3

Inscrições em: http://www.naturalgis.pt/subscricao.html

Mapas espantosos com QGIS!
Aprenda a criar mapas tematicos com QGIS

Conteúdo Programático:

  1. Controlar ordem das camadas
  2. Simbologia vectores
  3. Simbologia rasters
  4. Simbologia por categorias
  5. Simbologia por quantidades
  6. Simbologia baseada em regras
  7. Simbologia avançada - níveis
  8. Modos de rendering e transparência
  9. Etiquetas
  10. Criação de símbolos compostos
  11. Gestor de símbolos
  12. Gravar, Importar e exportar símbolos
  13. Compositor de impressão
  14. Elementos do mapa (Escala gráfica, Norte, Legenda,…)
  15. Grelhas de coordenadas
  16. Mapas de enquadramento
  17. Optimização da produção de cartografia
  18. Filtrar legenda consoante visibilidade
  19. A ferramenta “atlas”
  20. Criar séries de mapas com a ferramenta “atlas”
  21. Presets e estilos múltiplos
  22. Controlo dos elementos e formato do mapa através dos atributos
  23. Criação de cartogramas
  24. Criação automática de mapas de ocorrência

A QGIS plugin to run (common) spatial queries with PostGIS

QGIS Processing toolbox is extremely flexible and it allows easily to implement QGIS tools (with a GUI) that are based on command line utilities (see this and this).

When the underlying command line program is ogr2ogr (or any other GDAL/OGR utility) possibilities become endeless. For example ogr2ogr can be used to convert formats or run arbitrary (Spatial) SQL queries.

One possibility is to run Spatial SQL queries using input PostGIS layers and getting as output a PostGIS layer. The intersting bit is that the Spatial SQL query will be run by the server/machine (where the input layers are stored), so possibly not by the destkop machine where you are running ogr2ogr.

Usually to run a (Spatial) SQL query in PostGIS a user would open one of the many SQL shells available (PgAdmin, QGIS DB Manager, phppgadmin, psql, etc.) and run something like the following:


CREATE TABLE dissolved AS
SELECT attribute1,(ST_Dump(ST_Union(a.geom))).geom::geometry(Polygon,4326) as geom
FROM input_table a
GROUP BY attribute1;


and that is ok for us geo-geeks, but what about average Jane/Joe?

Isn’t this much easier?

Dissolve PostGIS polygons
Dissolve PostGIS polygons

With this plugin for the QGIS Processing toolbox (available in the official QGIS plugin repository) we aim to bring into QGIS a set of tools to allow basic PostGIS users to run common queries (Dissolve, Clip, Difference, etc.), using as input PostGIS layer and getting as output PostGIS layer(s).

PostGIS QGIS Processing tools
PostGIS QGIS Processing tools

Please note that the plugin is still in the early stages of development and that it will be greatly improved once the Processing toolbox will support features like multi-attribute selection, adding to (QGIS) map output layers other than shapefiles, and others.

A QGIS plugin to run (common) spatial queries with PostGIS

QGIS Processing toolbox is extremely flexible and it allows easily to implement QGIS tools (with a GUI) that are based on command line utilities (see this and this).

When the underlying command line program is ogr2ogr (or any other GDAL/OGR utility) possibilities become endeless. For example ogr2ogr can be used to convert formats or run arbitrary (Spatial) SQL queries.

One possibility is to run Spatial SQL queries using input PostGIS layers and getting as output a PostGIS layer. The intersting bit is that the Spatial SQL query will be run by the server/machine (where the input layers are stored), so possibly not by the destkop machine where you are running ogr2ogr.

Usually to run a (Spatial) SQL query in PostGIS a user would open one of the many SQL shells available (PgAdmin, QGIS DB Manager, phppgadmin, psql, etc.) and run something like the following:


CREATE TABLE dissolved AS
SELECT attribute1,(ST_Dump(ST_Union(a.geom))).geom::geometry(Polygon,4326) as geom
FROM input_table a
GROUP BY attribute1;


and that is ok for us geo-geeks, but what about average Jane/Joe?

Isn’t this much easier?

Dissolve PostGIS polygons
Dissolve PostGIS polygons

With this plugin for the QGIS Processing toolbox (available in the official QGIS plugin repository) we aim to bring into QGIS a set of tools to allow basic PostGIS users to run common queries (Dissolve, Clip, Difference, etc.), using as input PostGIS layer and getting as output PostGIS layer(s).

PostGIS QGIS Processing tools
PostGIS QGIS Processing tools

Please note that the plugin is still in the early stages of development and that it will be greatly improved once the Processing toolbox will support features like multi-attribute selection, adding to (QGIS) map output layers other than shapefiles, and others.

New geoprocessing tools in the QGIS Processing toolbox

QGIS 2.8 was not officially announced yet and, as always happens in the QGIS project, new features are already being added to QGIS master, aka next future release, in this case 2.10.

At NaturalGIS we do our share of effort, especially regarding improvements for the QGIS “Processing” toolbox, and recently started to add new geoprocessing tools for vectors. At the time we are writing the tools we added are:

  • Buffer
  • Single sided buffer (or offset lines)
  • Clip by extent
  • Clip by polygon
  • Create points along lines
  • Dissolve
ogr2ogr geoprocessing tools for QGIS Processing toolbox
New QGIS geoprocessing tools

Some are completely new, like the Single sided buffer (or offset lines), Clip (vector) by extent (a similar tool is already available in QGIS but only to clip rasters layers) and Create points along lines, others are not (Buffer, Clip by polygon and Dissolve) as there are already plenty of alternatives in QGIS. The point here is that this “new” tools are quite faster than the already existing QGIS counterparts, or they offer new options.

For example the “Dissolve” tool is on average several times faster (up to 7 times, in our simple tests) than the QGIS counterpart, moreover the tool has the ability to compute some statistics on numerical attributes. See the image below:

ogr2ogr geoprocessing Dissolve tool
Dissolve with stats

Under the hood the work is done by that great software that is ogr2ogr. In this case is used to run spatial SQL queries, using as engine SQLite/Spatialite

More of this ogr2ogr/sql based tools will be added in the next future, especially when a few missing features will be added to the QGIS Processing toolbox, meanwhile you can help us testing the above ones.

Under MS Windows you can install the development version of QGIS using the OSGeo4W installer. Under Ubuntu GNU/Linux you can use the nightly builds repository.

New geoprocessing tools in the QGIS Processing toolbox

QGIS 2.8 was not officially announced yet and, as always happens in the QGIS project, new features are already being added to QGIS master, aka next future release, in this case 2.10.

At NaturalGIS we do our share of effort, especially regarding improvements for the QGIS “Processing” toolbox, and recently started to add new geoprocessing tools for vectors. At the time we are writing the tools we added are:

  • Buffer
  • Single sided buffer (or offset lines)
  • Clip by extent
  • Clip by polygon
  • Create points along lines
  • Dissolve
ogr2ogr geoprocessing tools for QGIS Processing toolbox
New QGIS geoprocessing tools

Some are completely new, like the Single sided buffer (or offset lines), Clip (vector) by extent (a similar tool is already available in QGIS but only to clip rasters layers) and Create points along lines, others are not (Buffer, Clip by polygon and Dissolve) as there are already plenty of alternatives in QGIS. The point here is that this “new” tools are quite faster than the already existing QGIS counterparts, or they offer new options.

For example the “Dissolve” tool is on average several times faster (up to 7 times, in our simple tests) than the QGIS counterpart, moreover the tool has the ability to compute some statistics on numerical attributes. See the image below:

ogr2ogr geoprocessing Dissolve tool
Dissolve with stats

Under the hood the work is done by that great software that is ogr2ogr. In this case is used to run spatial SQL queries, using as engine SQLite/Spatialite

More of this ogr2ogr/sql based tools will be added in the next future, especially when a few missing features will be added to the QGIS Processing toolbox, meanwhile you can help us testing the above ones.

Under MS Windows you can install the development version of QGIS using the OSGeo4W installer. Under Ubuntu GNU/Linux you can use the nightly builds repository.

A new QGIS tool (based on ogr2ogr) to import vectors in PostGIS

In QGIS there are many tools that can be used to import vectors inside a PostGIS database, each one has pros and cons:

  • SPIT core plugin: available since early QGIS releases but now unmaintained tool and therefore candidate to be removed from future QGIS releases. It has the advantage to allow import several vectors in one run, but on the other hand it does not have an option to launder table/columns names and is overall quite slow especially for fair large vectors.
  • DB Manager: it has several pros, like supporting drag & drop import and a few important other options, but misses to allow import several vectors and is overall slow especially for fair large vectors.
  • QGIS browser: it allows import vectors using drag & drop but as the above tools missed to allow multiple vectors import. Overall slow especially for fair large vectors.
  • Processing toolbox ”Import into PostGIS”: it can import several vectors at once because, as any other tool in the QGIS Processing toolbox, it can run in batch mode. Overall slow especially for fair large vectors.

There are of course also command line alternatives, in particular shp2pgsql (together with psql) and ogr2ogr. Each one is rich of options/switches and they can be scripted to import several vectors in one go. While shp2pgsql is installed were PostGIS is installed, usually it is not on common users desktop machines. On the other hand ogr2ogr is installed and available on any machine where QGIS is installed because is part of the GDAL/OGR libary, that is basic dependency of any QGIS installation.

We compared how importing vectors in PostGIS performed using ogr2ogr compared to the tools available in QGIS, and then also compared to shp2pgsql. In short, the results are the following:

  • even without recurring to any particular switch/trick, ogr2ogr is on average much more faster than any available tools available in QGIS.
  • ogr2ogr and shp2pgsql performed in a similar way.

To compare ogr2ogr and shp2pgsql we used as input dataset a 4 million features (polygons) shapefile (1.3GB of space occupied) and also a small subset of it (4000 features, 10MB) using PostGIS installed on the local machine (Ubuntu GNU/Linux 14.04).

Without using any particular switch to make imports faster (like “-D” for shp2pgsql or “-config PG_USE_COPY YES” for ogr2ogr) ogr2ogr is much faster than shp2pgsql/psql with the small dataset (2.5 seconds against 35 seconds).

With the large dataset things gets the other way, with shp2pgsql/psql ending the task in 17 minutes against 19.5 minutes with ogr2ogr.

Adding the options “-D” to shp2pgsql and “-config PG_USE_COPY YES” to ogr2ogr is possible to get a dramatic improvement of the performace of both tools: ogr2ogr takes 0.8 seconds to process the small dataset and 2.21 minutes the process the big dataset, while shp2pgsql/psql takes respectively 24 seconds and 1.56 minutes.

ogr2ogr seemed a good choice to create a new tool for QGIS to allow import vectors in a fast way. We implemented such tool as part of the QGIS Processing toolbox and therefore is available the brand new QGIS 2.8 release.

QGIS Processing tools to import vector layers in PostGIS
The new tool(s) in the QGIS Processing toolbox

The tool also exposes options that are not usually available in any other QGIS tool. Aming the others: Vector dimensions, Append, Append and add new fields, Skip failures, Simplification, Densification, Import selected features by extent, Import clipped features by extent and a few others.

Tool GUI
Tool GUI

A new QGIS tool (based on ogr2ogr) to import vectors in PostGIS

In QGIS there are many tools that can be used to import vectors inside a PostGIS database, each one has pros and cons:

  • SPIT core plugin: available since early QGIS releases but now unmaintained tool and therefore candidate to be removed from future QGIS releases. It has the advantage to allow import several vectors in one run, but on the other hand it does not have an option to launder table/columns names and is overall quite slow especially for fair large vectors.
  • DB Manager: it has several pros, like supporting drag & drop import and a few important other options, but misses to allow import several vectors and is overall slow especially for fair large vectors.
  • QGIS browser: it allows import vectors using drag & drop but as the above tools missed to allow multiple vectors import. Overall slow especially for fair large vectors.
  • Processing toolbox ”Import into PostGIS”: it can import several vectors at once because, as any other tool in the QGIS Processing toolbox, it can run in batch mode. Overall slow especially for fair large vectors.

There are of course also command line alternatives, in particular shp2pgsql (together with psql) and ogr2ogr. Each one is rich of options/switches and they can be scripted to import several vectors in one go. While shp2pgsql is installed were PostGIS is installed, usually it is not on common users desktop machines. On the other hand ogr2ogr is installed and available on any machine where QGIS is installed because is part of the GDAL/OGR libary, that is basic dependency of any QGIS installation.

We compared how importing vectors in PostGIS performed using ogr2ogr compared to the tools available in QGIS, and then also compared to shp2pgsql. In short, the results are the following:

  • even without recurring to any particular switch/trick, ogr2ogr is on average much more faster than any available tools available in QGIS.
  • ogr2ogr and shp2pgsql performed in a similar way.

To compare ogr2ogr and shp2pgsql we used as input dataset a 4 million features (polygons) shapefile (1.3GB of space occupied) and also a small subset of it (4000 features, 10MB) using PostGIS installed on the local machine (Ubuntu GNU/Linux 14.04).

Without using any particular switch to make imports faster (like “-D” for shp2pgsql or “-config PG_USE_COPY YES” for ogr2ogr) ogr2ogr is much faster than shp2pgsql/psql with the small dataset (2.5 seconds against 35 seconds).

With the large dataset things gets the other way, with shp2pgsql/psql ending the task in 17 minutes against 19.5 minutes with ogr2ogr.

Adding the options “-D” to shp2pgsql and “-config PG_USE_COPY YES” to ogr2ogr is possible to get a dramatic improvement of the performace of both tools: ogr2ogr takes 0.8 seconds to process the small dataset and 2.21 minutes the process the big dataset, while shp2pgsql/psql takes respectively 24 seconds and 1.56 minutes.

ogr2ogr seemed a good choice to create a new tool for QGIS to allow import vectors in a fast way. We implemented such tool as part of the QGIS Processing toolbox and therefore is available the brand new QGIS 2.8 release.

QGIS Processing tools to import vector layers in PostGIS
The new tool(s) in the QGIS Processing toolbox

The tool also exposes options that are not usually available in any other QGIS tool. Aming the others: Vector dimensions, Append, Append and add new fields, Skip failures, Simplification, Densification, Import selected features by extent, Import clipped features by extent and a few others.

Tool GUI
Tool GUI

Back to Top

Sustaining Members