In the next weeks/months are due a couple of QGIS events where NaturalGIS will be present with presentations and workshops.
May 18th and 19th we will be at the
QGIS Conference 2015
organized by the University of Copenhagen, Forestry College, that will take place in Nødebo, Denmark.
This event is directly linked with the
13th QGIS Developer Meeting, that will take place from May 20 to 22 in the same location.
At the QGIS Conference we will make a presentation about how to get help/support for QGIS and a workshop about
QGIS Server and how to publish OGC services
(WMS/WFS/WCS) and webgis applications.
June 5th we will be in Covilhã, Serra da Estrela, Portugal, to join the
3rd Portugal QGIS user meeting
where we also will make a presentation (the same as in Nødebo) and a workshop about using the QGIS Processing toolbox.
In the next weeks/months are due a couple of QGIS events where NaturalGIS will be present with presentations and workshops.
May 18th and 19th we will be at the
QGIS Conference 2015
organized by the University of Copenhagen, Forestry College, that will take place in Nødebo, Denmark.
This event is directly linked with the
13th QGIS Developer Meeting, that will take place from May 20 to 22 in the same location.
At the QGIS Conference we will make a presentation about how to get help/support for QGIS and a workshop about
QGIS Server and how to publish OGC services
(WMS/WFS/WCS) and webgis applications.
June 5th we will be in Covilhã, Serra da Estrela, Portugal, to join the
3rd Portugal QGIS user meeting
where we also will make a presentation (the same as in Nødebo) and a workshop about using the QGIS Processing toolbox.
NTv2
is a format/method to allow transform coordinates (and vector/raster layers)
from one coordinate reference system to a different one (involving a change of
datum) with a small error:
the precision is usually around a few centimeters (or less)
and is much better than the one that can be obtained by reprojecting layers using the standard
coordinate reference systems definitions, as they are available in common GIS software/libraries/toolboxes.
An example of the precision given by NTv2 tranformations compared
to the one obtained by reprojecting layers using standard CRS definitions.
Note: This is a rather complex matter so you may want to read some tutorial about it.
One of the best we have ever read is this one [Portuguese or
English].
The NTv2 method make use of binary files (.gsb) called ”grids” and is fully supported by libraries like
Proj
and software like
GDAL/OGR.
Until not long ago the availability of these grids was scarce but in recent
times they have become quite easy to find and are freely available for many countries.
Unfortunately using this grid is not easy for the common GIS user: both command line utilities and Desktop GIS software need
custom CRS definitions, downloading and copying files to specific
locations and eventually do more configurations. See for example the following tutorials for
QGIS or
ArcGIS.
For the cases when grid files are freely available and redistributable we implemented an easier solution in the form of a
QGIS plugin for its Processing toolbox. With this plugin the operation is as simple
as just selecting the coordinate reference system of the input layer and
the transformation (direct or inverse).
Ntv2 Datum transformations GUI
The plugin already comes with support for grids/transformations for a few countries
and more will be added given that ntv2 grids are freely available and redistributable.
If you want to have a certain grid/transformation/country added please open a request here
Ntv2 Datum transformations for the QGIS Processing toolbox
Note2: The plugin was directly derived from a
similar one that we helped develop for the
Portuguese QGIS community/user group for the specific context of datum transformations for Portugal.
At some point we felt that this plugin could have been useful also for users from other countries so we released an “international” version of it.
NTv2
is a format/method to allow transform coordinates (and vector/raster layers)
from one coordinate reference system to a different one (involving a change of
datum) with a small error:
the precision is usually around a few centimeters (or less)
and is much better than the one that can be obtained by reprojecting layers using the standard
coordinate reference systems definitions, as they are available in common GIS software/libraries/toolboxes.
An example of the precision given by NTv2 tranformations compared
to the one obtained by reprojecting layers using standard CRS definitions.
Note: This is a rather complex matter so you may want to read some tutorial about it.
One of the best we have ever read is this one [Portuguese or
English].
The NTv2 method make use of binary files (.gsb) called ”grids” and is fully supported by libraries like
Proj
and software like
GDAL/OGR.
Until not long ago the availability of these grids was scarce but in recent
times they have become quite easy to find and are freely available for many countries.
Unfortunately using this grid is not easy for the common GIS user: both command line utilities and Desktop GIS software need
custom CRS definitions, downloading and copying files to specific
locations and eventually do more configurations. See for example the following tutorials for
QGIS or
ArcGIS.
For the cases when grid files are freely available and redistributable we implemented an easier solution in the form of a
QGIS plugin for its Processing toolbox. With this plugin the operation is as simple
as just selecting the coordinate reference system of the input layer and
the transformation (direct or inverse).
Ntv2 Datum transformations GUI
The plugin already comes with support for grids/transformations for a few countries
and more will be added given that ntv2 grids are freely available and redistributable.
If you want to have a certain grid/transformation/country added please open a request here
Ntv2 Datum transformations for the QGIS Processing toolbox
Note2: The plugin was directly derived from a
similar one that we helped develop for the
Portuguese QGIS community/user group for the specific context of datum transformations for Portugal.
At some point we felt that this plugin could have been useful also for users from other countries so we released an “international” version of it.
A NaturalGIS acrescentou à sua
oferta de cursos QGIS uma formação de
produção cartográfica, que permitirá aprender a produzir
eficientemente mapas de elevada qualidade recorrendo às
funcionalidades mais recentes de QGIS.
O curso cobrirá as opções avançadas de simbologia para camadas
vectoriais e raster e etiquetas, o compositor de mapas e automatização
de produção cartográfica, entre outros. Este curso é ideal para
aprender tirar o maior partido do QGIS para criar mapas temáticos e
cartografia para os instrumentos de gestão territorial, como por
exemplo, PDM, PP, PU, etc.
A primeira edição terá lugar em Lisboa de 4 a 6 de Maio 2015.
A NaturalGIS acrescentou à sua
oferta de cursos QGIS uma formação de
produção cartográfica, que permitirá aprender a produzir
eficientemente mapas de elevada qualidade recorrendo às
funcionalidades mais recentes de QGIS.
O curso cobrirá as opções avançadas de simbologia para camadas
vectoriais e raster e etiquetas, o compositor de mapas e automatização
de produção cartográfica, entre outros. Este curso é ideal para
aprender tirar o maior partido do QGIS para criar mapas temáticos e
cartografia para os instrumentos de gestão territorial, como por
exemplo, PDM, PP, PU, etc.
A primeira edição terá lugar em Lisboa de 4 a 6 de Maio 2015.
QGIS
Processing toolbox is extremely flexible and it allows easily to implement QGIS tools (with a GUI) that
are based on command line utilities (see
this and
this).
When the underlying command line program is
ogr2ogr
(or any other GDAL/OGR utility)
possibilities become endeless. For example ogr2ogr can be used to convert formats or run arbitrary (Spatial) SQL queries.
One possibility is to run Spatial SQL queries using input PostGIS layers and getting as output a PostGIS layer. The intersting
bit is that the Spatial SQL query will be run by the server/machine (where the input layers are stored),
so possibly not by the destkop machine where you are running ogr2ogr.
Usually to run a (Spatial) SQL query in PostGIS a user would open one of the many SQL shells available (PgAdmin, QGIS DB Manager,
phppgadmin, psql, etc.) and run something like the following:
CREATE TABLE dissolved AS
SELECT attribute1,(ST_Dump(ST_Union(a.geom))).geom::geometry(Polygon,4326) as geom
FROM input_table a
GROUP BY attribute1;
and that is ok for us geo-geeks, but what about average Jane/Joe?
Isn’t this much easier?
Dissolve PostGIS polygons
With this plugin for the QGIS Processing
toolbox (available in the official QGIS plugin repository) we aim to bring into QGIS a set of tools to allow basic PostGIS users
to run common queries (Dissolve, Clip, Difference, etc.), using as input PostGIS layer and getting as output PostGIS layer(s).
PostGIS QGIS Processing tools
Please note that the plugin is still in the early stages of development and that it will be greatly improved once
the Processing toolbox will support features like multi-attribute selection, adding to (QGIS) map output layers other than shapefiles, and
others.
QGIS
Processing toolbox is extremely flexible and it allows easily to implement QGIS tools (with a GUI) that
are based on command line utilities (see
this and
this).
When the underlying command line program is
ogr2ogr
(or any other GDAL/OGR utility)
possibilities become endeless. For example ogr2ogr can be used to convert formats or run arbitrary (Spatial) SQL queries.
One possibility is to run Spatial SQL queries using input PostGIS layers and getting as output a PostGIS layer. The intersting
bit is that the Spatial SQL query will be run by the server/machine (where the input layers are stored),
so possibly not by the destkop machine where you are running ogr2ogr.
Usually to run a (Spatial) SQL query in PostGIS a user would open one of the many SQL shells available (PgAdmin, QGIS DB Manager,
phppgadmin, psql, etc.) and run something like the following:
CREATE TABLE dissolved AS
SELECT attribute1,(ST_Dump(ST_Union(a.geom))).geom::geometry(Polygon,4326) as geom
FROM input_table a
GROUP BY attribute1;
and that is ok for us geo-geeks, but what about average Jane/Joe?
Isn’t this much easier?
Dissolve PostGIS polygons
With this plugin for the QGIS Processing
toolbox (available in the official QGIS plugin repository) we aim to bring into QGIS a set of tools to allow basic PostGIS users
to run common queries (Dissolve, Clip, Difference, etc.), using as input PostGIS layer and getting as output PostGIS layer(s).
PostGIS QGIS Processing tools
Please note that the plugin is still in the early stages of development and that it will be greatly improved once
the Processing toolbox will support features like multi-attribute selection, adding to (QGIS) map output layers other than shapefiles, and
others.
QGIS 2.8 was not officially announced yet and,
as always happens in the QGIS project, new features are already being added to QGIS master, aka next future release, in this case 2.10.
At NaturalGIS we do our share of effort,
especially regarding improvements for the QGIS “Processing” toolbox, and recently started
to add
new geoprocessing tools for vectors. At the time we are writing the tools we added are:
Buffer
Single sided buffer (or offset lines)
Clip by extent
Clip by polygon
Create points along lines
Dissolve
New QGIS geoprocessing tools
Some are completely new, like the Single sided buffer (or offset lines), Clip (vector) by extent
(a similar tool is already available in QGIS but only to clip rasters layers) and Create points along lines,
others are not (Buffer, Clip by polygon and Dissolve) as there are already
plenty of alternatives in QGIS. The point here is that this “new” tools are quite faster than the already existing QGIS counterparts,
or they offer new options.
For example the “Dissolve” tool is on average several times faster (up to 7 times, in our simple tests) than the QGIS counterpart,
moreover the tool has the ability to compute some statistics on numerical attributes. See the image below:
Dissolve with stats
Under the hood the work is done by that great software that is ogr2ogr. In this case is used to run spatial SQL queries,
using as engine SQLite/Spatialite
More of this ogr2ogr/sql based tools will be added in the next future, especially when a few missing features will be added
to the QGIS Processing toolbox, meanwhile you can help us testing the above ones.
Under MS Windows you can install the development version of QGIS using the
OSGeo4W installer. Under
Ubuntu GNU/Linux you can use the
nightly builds repository.
QGIS 2.8 was not officially announced yet and,
as always happens in the QGIS project, new features are already being added to QGIS master, aka next future release, in this case 2.10.
At NaturalGIS we do our share of effort,
especially regarding improvements for the QGIS “Processing” toolbox, and recently started
to add
new geoprocessing tools for vectors. At the time we are writing the tools we added are:
Buffer
Single sided buffer (or offset lines)
Clip by extent
Clip by polygon
Create points along lines
Dissolve
New QGIS geoprocessing tools
Some are completely new, like the Single sided buffer (or offset lines), Clip (vector) by extent
(a similar tool is already available in QGIS but only to clip rasters layers) and Create points along lines,
others are not (Buffer, Clip by polygon and Dissolve) as there are already
plenty of alternatives in QGIS. The point here is that this “new” tools are quite faster than the already existing QGIS counterparts,
or they offer new options.
For example the “Dissolve” tool is on average several times faster (up to 7 times, in our simple tests) than the QGIS counterpart,
moreover the tool has the ability to compute some statistics on numerical attributes. See the image below:
Dissolve with stats
Under the hood the work is done by that great software that is ogr2ogr. In this case is used to run spatial SQL queries,
using as engine SQLite/Spatialite
More of this ogr2ogr/sql based tools will be added in the next future, especially when a few missing features will be added
to the QGIS Processing toolbox, meanwhile you can help us testing the above ones.
Under MS Windows you can install the development version of QGIS using the
OSGeo4W installer. Under
Ubuntu GNU/Linux you can use the
nightly builds repository.
In QGIS there are many tools that can be used to import vectors inside a PostGIS database, each one has pros and cons:
SPIT core plugin: available since early QGIS releases but now unmaintained tool and therefore candidate to be removed from future QGIS releases.
It has the advantage to allow import several vectors in one run, but on the other hand it does not have an option to launder table/columns names and is overall
quite slow especially for fair large vectors.
DB Manager: it has several pros, like supporting drag & drop import and a few important other options, but misses to allow import
several vectors and is overall slow especially for fair large vectors.
QGIS browser: it allows import vectors using drag & drop but as the above tools missed to allow multiple vectors import.
Overall slow especially for fair large vectors.
Processing toolbox ”Import into PostGIS”: it can import several vectors at once because, as any other tool in the QGIS Processing
toolbox, it can run in batch mode. Overall slow especially for fair large vectors.
There are of course also command line alternatives, in particular
shp2pgsql
(together with psql) and
ogr2ogr. Each one is rich of options/switches and
they can be scripted to import several vectors in one go. While shp2pgsql is installed
were PostGIS is installed, usually it is not on common users desktop machines.
On the other hand ogr2ogr is installed and available on any machine where QGIS is installed because is part of the
GDAL/OGR libary, that is basic dependency of any QGIS installation.
We compared how importing vectors in PostGIS performed using ogr2ogr compared to the tools available in QGIS, and then also compared to shp2pgsql. In short, the
results are the following:
even without recurring to any particular switch/trick, ogr2ogr is on average much more faster than any available tools available in QGIS.
ogr2ogr and shp2pgsql performed in a similar way.
To compare ogr2ogr and shp2pgsql we used as input dataset a 4 million features (polygons) shapefile (1.3GB of space occupied) and also a small
subset of it (4000 features, 10MB) using PostGIS installed on the local machine (Ubuntu GNU/Linux 14.04).
Without using any particular switch to make imports faster (like “-D” for shp2pgsql or “-config PG_USE_COPY YES” for ogr2ogr) ogr2ogr is much faster than shp2pgsql/psql
with the small dataset (2.5 seconds against 35 seconds).
With the large dataset things gets the other way, with shp2pgsql/psql ending the task in 17 minutes against 19.5 minutes with ogr2ogr.
Adding the options “-D” to shp2pgsql and “-config PG_USE_COPY YES” to ogr2ogr is possible to get a dramatic improvement of the performace of both tools:
ogr2ogr takes 0.8 seconds to process the small dataset and 2.21 minutes the process the big dataset, while shp2pgsql/psql
takes respectively 24 seconds and 1.56 minutes.
ogr2ogr seemed a good choice to create a new tool for QGIS to allow import vectors in a fast way.
We implemented such tool
as part of the QGIS Processing toolbox and therefore is available the brand new QGIS 2.8 release.
The new tool(s) in the QGIS Processing toolbox
The tool also exposes options that are not usually available in any other QGIS tool. Aming the others: Vector dimensions, Append, Append and add new fields, Skip failures, Simplification,
Densification, Import selected features by extent, Import clipped features by extent and a few others.
In QGIS there are many tools that can be used to import vectors inside a PostGIS database, each one has pros and cons:
SPIT core plugin: available since early QGIS releases but now unmaintained tool and therefore candidate to be removed from future QGIS releases.
It has the advantage to allow import several vectors in one run, but on the other hand it does not have an option to launder table/columns names and is overall
quite slow especially for fair large vectors.
DB Manager: it has several pros, like supporting drag & drop import and a few important other options, but misses to allow import
several vectors and is overall slow especially for fair large vectors.
QGIS browser: it allows import vectors using drag & drop but as the above tools missed to allow multiple vectors import.
Overall slow especially for fair large vectors.
Processing toolbox ”Import into PostGIS”: it can import several vectors at once because, as any other tool in the QGIS Processing
toolbox, it can run in batch mode. Overall slow especially for fair large vectors.
There are of course also command line alternatives, in particular
shp2pgsql
(together with psql) and
ogr2ogr. Each one is rich of options/switches and
they can be scripted to import several vectors in one go. While shp2pgsql is installed
were PostGIS is installed, usually it is not on common users desktop machines.
On the other hand ogr2ogr is installed and available on any machine where QGIS is installed because is part of the
GDAL/OGR libary, that is basic dependency of any QGIS installation.
We compared how importing vectors in PostGIS performed using ogr2ogr compared to the tools available in QGIS, and then also compared to shp2pgsql. In short, the
results are the following:
even without recurring to any particular switch/trick, ogr2ogr is on average much more faster than any available tools available in QGIS.
ogr2ogr and shp2pgsql performed in a similar way.
To compare ogr2ogr and shp2pgsql we used as input dataset a 4 million features (polygons) shapefile (1.3GB of space occupied) and also a small
subset of it (4000 features, 10MB) using PostGIS installed on the local machine (Ubuntu GNU/Linux 14.04).
Without using any particular switch to make imports faster (like “-D” for shp2pgsql or “-config PG_USE_COPY YES” for ogr2ogr) ogr2ogr is much faster than shp2pgsql/psql
with the small dataset (2.5 seconds against 35 seconds).
With the large dataset things gets the other way, with shp2pgsql/psql ending the task in 17 minutes against 19.5 minutes with ogr2ogr.
Adding the options “-D” to shp2pgsql and “-config PG_USE_COPY YES” to ogr2ogr is possible to get a dramatic improvement of the performace of both tools:
ogr2ogr takes 0.8 seconds to process the small dataset and 2.21 minutes the process the big dataset, while shp2pgsql/psql
takes respectively 24 seconds and 1.56 minutes.
ogr2ogr seemed a good choice to create a new tool for QGIS to allow import vectors in a fast way.
We implemented such tool
as part of the QGIS Processing toolbox and therefore is available the brand new QGIS 2.8 release.
The new tool(s) in the QGIS Processing toolbox
The tool also exposes options that are not usually available in any other QGIS tool. Aming the others: Vector dimensions, Append, Append and add new fields, Skip failures, Simplification,
Densification, Import selected features by extent, Import clipped features by extent and a few others.
