Since the last update, our “SLYR” ESRI to QGIS compatibility suite has gained a ton of new functionality, including full support for conversion of ArcMap MXD documents (with page layouts!). In this update, we’ll explore some of the new functionality available in the tool — but instead of focusing solely on SLYR, this time we’ll also explore the enhancements we’ve been making in QGIS itself that have helped improve the quality of ArcMap document conversion.

While many of these enhancements are already available to all users of QGIS 3.12, others are exciting additions to the upcoming QGIS 3.14 release. Let’s dive in!

Improved legend customisation

One shortcoming we realised early on during our work on SLYR was that QGIS map legends just didn’t offer a comparable level of customisation as ArcMap legends. We could convert the basic layout of a legend, but we just couldn’t get the legend appearance in QGIS sufficiently close to its original appearance in the MXD file.

To address this we’ve been extending QGIS’ inbuilt legend support by adding finer control over the legend layout and spacing.

As a result, one exciting addition we’ve recently made for QGIS 3.14 is adding the ability to customise legend patch shapes and sizes on an item-by-item basis! Previously, legends in QGIS were rather boring, with all polygon layers showing as a plain rectangle and line layers as a horizontal line.

Now, users have full control over setting custom shapes for their legend patches! This makes for much more user-friendly legends, as you can now show representative shapes in your legends — e.g. a river symbol can be shown as a wiggly line, instead of an unrealistic straight horizontal line.

You can also override the size of a legend patch on an item-by-item basis too, which allows for further control over the final legend appearance. Checkout the screencast below showing both these features in action (naturally, using a legend from a converted MXD document… ArcMap users will likely recognise the fonts and patch shapes used here!)

We really wanted custom legend patch shapes to be a full first-class citizen in QGIS, so we also added support for managing them in user’s style databases. This makes it easy to setup your own libraries of custom legend shapes and share them with others. As a nice bonus, the SLYR tool even offers support for converting area and line patch shapes while converting ESRI .style databases:

Marker north arrows

Another issue we ran into while converting ArcMap page layouts was converting north arrows. QGIS used a very different approach to north arrows compared with ArcMap — in QGIS, north arrows were always based on existing SVG files, while ArcMap uses a rotated marker symbol for north arrows. Both approaches have their advantages and disadvantages, but we struggled to get good results when trying to convert ArcMap’s marker based approach to QGIS’ SVG based approach.

In the end, we weren’t happy with the result, so we took the step of implementing full support for marker symbol north arrows in QGIS 3.14. Now QGIS users have a choice of both north arrow styles — you can still create north arrows direct from SVG files, but you’ve also now got the flexibility to create them from standard marker symbols instead!

Adding support for marker based north arrows in QGIS allows us to get an perfect match when converting ESRI Page Layouts with north arrows:

Hollow and stepped line scale bars

While adding support for scale bar conversion to SLYR, we identified that some scale bar styles which are widely used in ArcMap just weren’t possible to reproduce in QGIS. Accordingly, from QGIS 3.14 on, we added native support for “Stepped line” and “Hollow” scale bar styles:

Embedded images in QGIS print layouts

ArcMap offers users the ability to directly embed images inside symbol definitions or page layouts. Whilst QGIS has offered embedded image support in symbols for a number of releases, this previously wasn’t possible to do in print layouts.

This was an issue for us while converting ArcMap page layouts, because we didn’t have any way to represent embedded images in QGIS layouts. Accordingly, for QGIS 3.14 we’ve added native support for directly embedding images (either raster images or SVG pictures) inside a page layout:

One handy consequence of this improvement is that it’s now possible to create completely self contained print layout templates for QGIS — you no longer have to separately distribute any required images (such as company logos) along with your QPT templates!

Naturally, our SLYR plugin now automatically converts any embedded images it finds in an ArcMap page layout and correctly creates a converted, embedded version of the image in the QGIS print layout.

Scalebar numeric formats

Another missing customisation in QGIS’ scale bar functionality was allowing users control over the scale bar’s number format. Previously, QGIS offered no customisation for these numbers, so you got only what QGIS decided you wanted. We improved this in QGIS 3.12 by offering users the ability to control exactly how they want their scale bar numbers to appear. There’s options for manually controlling the thousand and decimal separators, rounding, and much more:

This enhancement allowed us to get an exact match when converting ArcMap scale bars — the converted results should appear identical to their original ArcMap appearance!

Random marker fill

Until recently, one of the few remaining layer symbolisation gaps between QGIS and ArcMap was that QGIS had no symbology option for randomised marker placements for fill symbols. This was a big issue for us, because without it there just wasn’t any way that SLYR could convert layers styled with ArcMap’s “dot density renderer”  or using a marker fill’s “random offset” option.

So for QGIS 3.12, we add inbuilt support for a new “Random marker fill” symbol type:

This new symbol layer type allows for randomised (or stable seed-based) placement of markers inside polygon features. You’re given the option of either an absolute number of points to show in the feature, or a density-based count which retains its dot density regardless of the map’s scale!

Aside from being a useful symbology option in it’s own right, adding this functionality allowed us to accurately convert random markers or dot density renders from ArcMap to QGIS.

Other enhancements

The highlights above are only a small subset of the work we’ve done in QGIS to improve its interoperability with ArcMap via the SLYR plugin! Some of the other work we’ve done includes:

• Many improvements to QGIS’ bad layer handling and automatic repair functionality. For instance, QGIS now emulates ArcMap’s helpful behaviour where ALL similar broken layer paths in a document are fixed automatically after fixing the path to one broken layer. Handling broken layer paths was a pain point for our customers, so we’ve sought to make this as painless as possible.
• Support for plugins to handle pasting content into QGIS print layouts. We use this in SLYR to offer the ability to directly copy and paste content from ArcMap page layouts into a QGIS print layout.
• Support for plugins to hook into the standard QGIS “Open Project” dialog, offering support for opening projects of their own custom types. We use this to allow users to directly open MXD, MXT, PMF and SXD files from the QGIS Open Project menu action.
• We’ve worked closely with the upstream proj project, to ensure that coordinate reference systems from ESRI documents are correctly matched to known EPGS/ESRI CRS definitions in certain circumstances.

Other new features in SLYR

Aside from all the goodness we’ve explored above, the latest versions of SLYR offer TONS of new functionality for conversion of ArcMap documents, including:

• Full support for joins and relations when converting MXD documents
• Print layouts, including support for conversion of data driven pages to QGIS print atlases and support for multi-map page layouts using multiple data frames.
• Support for reading MXD document metadata (and converting this to QGIS document metadata)
• Support for dragging and dropping layers direct from ArcMap or ArcCatalog to a QGIS window, respecting all the layer styling.
• Support for AVL style conversion
• A new tool for dumping the full structure of MXD or LYR files to a json document. This is very handy for digging right into the full internals of the documents and for diagnosing corrupted documents.
• Full support for conversion of vector and raster layers
• Support for converting MXD, MXT and PMF documents
• Support for converting ArcScene SXD documents to 2-dimensional QGIS maps

Read more are the SLYR home page, or contact us today to discuss purchasing SLYR and your licensing needs!

You are using QGIS and look for support services to improve your experience and solve problems ? Oslandia is here to help you with our full QGIS editor service range ! Discover our team members below.

You will probably interact first with our pre-sales engineer Bertrand Parpoil. He leads Geographical Information System projects for 15 years for large corporations, public administrations or hi-tech SME. Bertrand will listen to your needs and explore your use cases, to offer you the best set of services.

Régis Haubourg also takes part in the first steps of projects to analyze your usages and improve them. GIS Expert, he knows QGIS by heart and will make the most of its capabilities. As QGIS Community Manager at Oslandia, he is very active in the QGIS community of developers and contributors. He is president of the Francophone OSGeo local chapter ( OSGeo-fr ), QGIS voting member, organizes the French QGIS day conference in Montpellier, and participates to QGIS community meetings. Before joining Oslandia, he led the migration to QGIS and PostGIS at the Adour-Garonne Water Agency, and now guides our clients with their GIS migrations to OpenSource solutions. Régis is also a great asset when working on water, hydrology and other specific thematic subjects.

Loïc Bartoletti develops QGIS, specializing in features corresponding to his fields of interests : network management, topography, urbanism, architecture… We find him contributing to advanced vector editing in QGIS, writing Python plugins, namely for DICT management. Pushing CAD and migrations from CAD tools to GIS and QGIS is one of his major goals. He will develop your custom applications, combining technical expertise and functional competences. When bored, Loïc packages software on FreeBSD.

Vincent Mora is senior developer in Python and C++, as well as PostGIS expert. He has a strong experience in numerical simulation. He likes coupling GIS (PostGIS, QGIS) with 3D numerical computing for risk management or production optimization. Vincent is an official QGIS committer and can directly integrate your needs into the core of the software. He is also GDAL committer and optimizes low-level layers of your applications. Among numerous activities, Vincent serves as lead developer of the development team for Hydra Software, a tool dedicated to unified hydraulics and hydrology modelling and simulation based on QGIS.

Hugo Mercier is an officiel QGIS committer too for several years. He regularly talks in international conferences on PostGIS, QGIS and other OpenSource GIS softwares. He will implement your needs with new QGIS features, develop innovative plugins ( like QGeoloGIS ) and design and build your new custom applications, solving all kind of technological challenges.

Paul Blottière completes our QGIS committers : very active on core development, Paul has refactored the QGIS server component to bring it to an industry-grade quality level. He also designed and implemented the infrastructure allowing to guarantee QGIS server performances. He dedicated himself to QGIS server OGC certification, especially for WMS (1.3). Thanks to this work QGIS is now a reference OGC implementation.

Julien Cabièces recently joined Oslandia, and quickly dived into QGIS : he contributes to the core of this Desktop GIS, on the server component, as well as applications linked to numerical simulation. Coming from a satellite imagery company with industrial applications, he uses his flexibility to answer all your needs. He brings quality and professionalism to your projects, minimizing risks for large production deployments.

You may also meet Vincent Picavet. Oslandia’s founder is a QGIS.org voting member, and is involved in the project’s evolution and the organization of the community.

Aside from these core contributors, all other Oslandia members also master QGIS integrate this tool into their day-to-day projects.

Bertrand, Régis, Loïc, Vincent (x2), Hugo, Paul et Julien are in tune with you and will be happy to work together for your migrations, application development, and all your desires to contribute to the QGIS ecosystem. Do not hesitate to contact us !

Sorry, this entry is only available in French.

QGIS Processing offers a rich and expandable set of algorithms which can operate on spatial data, along with a powerful Model Designer which allows users to string together these algorithms to create custom workflows.

Since its introduction in QGIS 2, the Processing framework has seen an intensive amount of development and optimisation efforts. In recent QGIS releases it offers a very user-friendly way of performing complex spatial data processing tasks, all without requiring ANY expensive third-party tools or software licenses!

At North Road we are passionate about the QGIS Processing framework, and have invested considerable effort in this framework over the past 5 years. We’re proud to announce that our latest crowd-funding campaign is focused on further expanding the capabilities and flexibility of Processing and the Processing Model Designer!

Unlike a typical crowdfunding campaign, where a specific funding target and deadline is set, we’re running this campaign a little differently. Instead, this campaign is taking the form of a “à la carte” menu of Processing enhancements. These range from small “paper-cut” style fixes, through to larger architectural improvements, and are each individually priced accordingly. We are asking backers to pick individual enhancements from this “menu of enhancements” and fund that enhancement’s development in full. In order to make this campaign affordable for a wide range of backers, we’ve included a huge range of enhancements which vary in price from smaller amounts to larger amounts.

You can read the full details of the campaign and browse the list of proposed enhancements at the campaign page.

The goal of the new API is twofold:

1. provide a unified way to store and retrieve data provider connections in the QGIS settings
2. provide an abstract set of methods to perform most common operation on DB data sources (e.g. browse tables, drop/create a table/schema, run arbitrary SQL commands etc.)

The new API is documented in https://qgis.org/api/classQgsAbstractProviderConnection.html and it provides a few specializations for DB connections (https://qgis.org/api/classQgsAbstractDatabaseProviderConnection.html) and an initial PR implementation for web service-based connections (https://github.com/qgis/QGIS/pull/33045).

While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:

• postgres
• geopackage (OGR)
• spatialite

The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).

For developers interested in working with the new API, a set Python tests are available to show how to use the methods:  https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsproviderconnection_ogr_gpkg.py (see also the postgres and spatialite companion tests).

There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:

• port all remaining data providers to the new API
• refactor and eliminate the remaining DB-manager connectors to make use of the abstract API
• eliminate duplicate and untested code inside the Processing framework for working with Postgres databases and port the code to the new, stable, well-tested API
• refactor and eliminate the remaining QGIS browser data items to make use of the abstract API 

For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.

Background/motivation

• DB-Manager is an important part of the QGIS interface, which allows browsing/previews of different DB-based data sources, complex queries, management of layers, import-export etc., DB creation and deletion etc.
• After the QGIS 3.0 release, improvements within the core browser widgets implemented in C++ have resulted in a (constantly growing) degree of overlapping functionality between the browser and db manager.
• After QGIS 3 API improvements concerning layer import and export functionality, there are many duplicated implementations between browser and db manager – some functionality is better in browser, some functionality is better in db manager. Users are forced to choose between two competing semi-complete alternatives, instead of having one, complete, well integrated solution.
• There are no unit tests for DB-Manager and this leads to frequent regressions, which (aside from being frustrating for users) consume a substantial part of our development time and budget during the bugfixing programs. Furthermore the nature of large Python codebases like db manager makes it very easy to accidentally break functionality with no warning or errors during development.

Proposed solution

We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.

The clear advantages are:

• no duplicate functionality, so it’s easier for users to understand and use
• more usage of well-tested and well-maintained core C++ API
• testability and immediate feedback on API breaks (an advantage of C++ is that the application won’t even build if an API is changed or accidentally misused)
• better performance
• the ability to expose database management functionality via stable PyQGIS API, allowing other plugins and scripts to utilise this functionality. In future, Processing algorithms may also be developed which would take advantage of these functions (e.g. “create schema”, “drop table”, “vacuum table” algorithms)
• DB management functionality would be available within the main QGIS window (from the Browser panel), instead of as a separate dialog.

Grant proposal package

The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.

• create an interface for databases that expose the required functions to a coherent API
• add missing tests and documentation for the a.m. API
• porting some basic functions from db manager to the new api:
  • create table (with native field types support)
  • create schema
  • delete table
  • Rename table

The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.

The post QGIS Abstract Connections API first appeared on Open Web Solutions, GIS & Python Development.

A few months ago, we proposed to the QGIS grant program to make improvements to the snap cache in QGIS. The community vote selected our project which was funded by QGIS.org. Developments are now mostly finished.

In short, snapping is crucial for editing geospatial features. It is the only way to ensuring they are topologically related, ie, connected vertices have exactly the same coordinates even if manual digitizing on screen is imprecise by nature.  Snapping correctly supposes QGIS have in memory an indexed cache of the geometries to snap to. And maintainting this cache when data is modified, sometimes by another user or database logic, can be a real challenge. This it exactly what this work adresses.

The proposal was divided into two different tasks:

• Manage circular dependencies
• Relax the snap cache index build

Manage cicular data dependencies

Data dependencies

Data dependency is an existing feature that allows you to configure QGIS to reload layers (and their snapping cache) when a layer is modified.

It is useful when you store your data in a database and you set up triggers to maintain consistency between the different tables of your data model.

For instance, say you have topological informations containing lines and nodes. Nodes are part of lines and lines go through nodes. Then, you move a node in QGIS, and save your modifications to the database. In order to keep the data consistent, a trigger updates the geometry of the line going through the modified node.

Node 2 is modified, Line 1 is updated accordingly

QGIS, as a database client, has no information that the line layer currently displayed in the canvas needs to be refreshed after the trigger. Although the map canvas will be up to date, because QGIS fetches data for display without any caching system, the snapping cache is not and you’ll end up with ghost snapping highlights issues.

Snapping highlights (light red) differ from real line (orange)

Defining a dependency between nodes and lines layers tells QGIS that it has to refresh the line layer when a node is modified.

Dependencies configuration: Lines layer will be refreshed whenever Nodes layer is modified

It also have to work the other way, modifying a line should update the nodes to ensure they still are on the line.

Circular data dependencies

So here we are, lines depend on nodes which depend on lines which depend on nodes which…

That’s what circular dependencies is about. This specific behavior was previously forbidden and needed a special way to deal with it. Thanks to this recent development, it is now possible.

It’s also possible to add the layer itself as one of its own dependencies. It helps dealing with specific cases where one feature modification could lead to a modification of another feature in the same layer (to keep consistency on road networks for instance).

Road 2 is modified, Road 1 is updated accordingly

This feature is available in the next QGIS LTR version 3.10.

Relax the snapping cache index build

If you work in QGIS with huge projects displaying a lot of vector data, and you enable snapping while editing these data, you probably already met this dialog:

Snap indexing dialog

This dialog informs you that data are currently being indexed so you can snap on them while you will edit feature geometry. And for big projects, this dialog can last for a really long time. Let’s work on speeding it up!

What’s a snap index?

Let’s say you want to move a line and snap it onto another one. While you drag your line with the mouse, QGIS will look for an existing geometry beneath the mouse cursor (with a certain pixel tolerance) every time you move your mouse. Without spatial index, QGIS will have to go through every geometry in your layer to check if the given geometry is beneath the cursor position. This would be very ineffective.

In order to prevent this, QGIS keeps an index where vector data are stored in a way that it can quickly find out what geometry is beneath the mouse cursor. The building of this data structure takes time and that is what the progress dialog is about.

Firstly: Parallelize snap index build

If you want to be able to snap on all layers in your project, then QGIS will have to build one snap index for each layer. This operation was made sequentially meaning that if you have for instance 20 layers and the index building last approximatively 3 seconds for each, then the whole index building will last 1 minute. We made modifications to QGIS so that index building could be done in parallel. As a result, the total index building time could theoretically be 3 seconds!

4 layers snap index being built in parallel

However, parallel operations are limited by the number of CPU cores of your machine, meaning that if you have 4 cores (core i7 for instance) then the total time will be up to 4 times faster than when the building is sequential (and last 15 seconds in our example).

Secondly: relax the snap build

For big projects, parallelizing index building is not enough and still takes too much time. Futhermore, to reduce snap index building, an existing optimisation was to build the spatial index for a specific area of interest (determined according to the displayed area and layer size). As a consequence, when you’ve done waiting for an index currently building and you move the map or zoom in/out, you could possibly trigger another snap index building and wait again.

So, the idea was to avoid waiting at all. Snap index is now built whenever it needs to (when you first enable snapping, when you move or zoom) but the user doesn’t have to wait for the build to be over and can continue what it was doing (creating feature, moving…). Snapping highlights will be missing when the index is currently being built and will appear gradually as soon as they finished. That’s what we call the relaxing mode.

No waiting dialog, snapping highlights appears as soon as snap index is ready

This feature has been merged into current QGIS master and will be present in future QGIS 3.12 release. We keep working on this feature in order to make it more stable and efficient.

What’s next

We’ll continue to improve this feature in the coming days, if you have the chance to test it and encounter issues please let us know on the QGIS tracker. If you think about a missing feature or just want to know more about QGIS, feel free to contact us at [email protected]. And please have a look at our support offering for QGIS.

Many thanks to QGIS grant program for funding these new features. Thanks also to all the people involved in reviewing the code and helping to better understand the existing mechanism.

Rececently the ruimtelijkeplannen plugin for using dutch spatial zoning plans in QGIS was renewed. A lot of extra functionality was added, sponsored by LBP|SIGHT, a company which uses this plugin frequently. As it happened, we now have a small list list of organizations who contributed to the development of this plugin, illustrating the power of […]

Sorry, this entry is only available in Dutch.

Sorry, this entry is only available in Dutch.

It’s a been a month full of huge improvements since the last update, and we have some exciting news to share about our SLYR ESRI to QGIS compatibility suite. With the recently published plugin version 3.7, MXD conversion has moved from a “beta” state to being fully supported and available out-of-the-box for all users!

Based on our massive library of reference files (almost 10,000 files covering a huge range of ArcGIS versions and features!), the tool is now able to successfully convert 96% of LYR files and 94.5% of MXD documents. This is a significant milestone, and with it we decided that MXD conversion support is now stable enough to move out of its previous beta state.

Aside from this milestone, the 3.7 release brings many more enhancements and improvements, including:

• SLYR now has full support for PMF published map documents created by ArcGIS Publisher, along with a new Processing algorithm to convert from a PMF document to a QGS projects
  
• We’ve also added support for converting ArcScene SXD scenes to QGS projects. This conversion is 2-dimensional only for now, but we plan on adding 3D conversion when QGIS’ 3D support further matures.
• We now convert all data frames contained within MXD documents, instead of just the first data frame. Currently, these are exposed as their own individual groups within the project layer tree (when we enable support for page layout conversion we’ll be automatically creating corresponding map themes from each data frame).
• We’ve added support for reading many more layer types, including raster catalog layers, topology layers, terrain layers, and LAS dataset layers. While QGIS doesn’t have support for these layer types, we need to fully parse them in order to convert the rest of the MXD document contents. Whenever an unsupported layer type like these are encountered the plugin shows a warning advising users which layers could not be successfully converted.
• We’ve also added support for reading TIN layers. Although previous QGIS versions had no means to read ESRI tin layers, thanks to work done in the MDAL library the upcoming QGIS 3.10.1 release adds full support for reading these data files! Accordingly, we’ll be unlocking support for converting TIN layers contained within MXD documents following the 3.10.1 release.
• Full support for WMTS and tiled internet layers
• Support for reading MXD documents which have repaired by the MXD Doctor utility
• Support for layers with a geopackage source
• Conversion of ImageServer based layers (since QGIS only has basic support for ESRI ImageServers, we convert these layers to their equivalent MapServer versions wherever possible)
• Basic support for representation renderers. Although QGIS has no capability to utilise the symbology linked with a representation renderer, we’ve added support for rendering these layers using any geometry overrides which may be present for the features.
• Conversion support for simple scale dependent renderers (these are a funny beast, which can’t be created directly through the ArcMap interface and which require custom ArcObjects code to create! That said, we’ve encountered a few examples of these inside our test library so have added support for converting them to the equivalent QGIS rule based renderer).
• We added a new “random marker fill” symbol type to the upstream QGIS project, which will be available in QGIS 3.12 along with support in SLYR for conversion of ESRI random marker fills.

So what’s next for SLYR? Over the remainder of 2019 we’ll be working furiously toward 100% conversion rates for LYR and MXD files. We’ll also start rolling out conversion support for page layouts to QGIS print layouts, and support for automatic conversion of ArcMap TIN layers to QGIS mesh layers.

Keep an eye on this blog and our Twitter channel for further updates!

Last week saw the exciting release of version 3 of the QGIS DataPlotly plugin, which incorporates all the work done as a result of our Print Layouts Graphs and Charts crowdfunding campaign crowd funding campaign. Now, beautiful charts and graphs are available directly within QGIS print layouts, and all it takes is the easy installation of the “DataPlotly” plugin from your QGIS install!

In this post we’ll showcase the functionality which has been added during this campaign, and which is available today in the plugin.

UI modernisation and tweaks

First up, during our work on this plugin we’ve invested some time in refreshing the plugin’s UI to ensure it follows all the widget conventions used elsewhere in QGIS. Now, the plugin blends seamlessly into your QGIS window, and all the chart setting widgets behave in just the way you’re used to. We’ve also used this opportunity to fix a number of issues the plugin had when running on hi-dpi displays (such as Apple retina displays)!

Dynamic charting of selected and visible features

If you’ve used earlier versions of the DataPlotly plugin, you’ll likely notice that there’s many new options shown in the screenshot above. Possibly the most exciting of these is the new “Use only selected features” and “Use only visible features” checkboxes. When either of these options are enabled, then your chart will immediately respond to changes in layer selections or changes to the visible map canvas extent respectively. Previously, interactivity in the plugin only went one way (from the chart to the canvas) – but now the charts are truly interactive, and respond dynamically to changes in the canvas too!

Improved handling of “data defined” settings

During the plugin refactoring, we reworked how “data defined” settings are handled within charts. If you’re not familiar with these, “data defined” settings are QGIS’ approach for exposing per-feature control over the map rendering process. In DataPlotly charts, we expose this functionality to allow per-feature control over the chart appearance (e.g., showing different scatter plot dot colors based on feature attributes). The new code uses the same code model as QGIS itself, so data-defined settings in your charts now have full access to the whole suite of QGIS expression functions and variables that you’re used to! Additionally, QGIS data-defined assistants are fully supported in the charts too. Ultimately, this enables some very advanced styling options, such as charts which dynamically change color and appearance on every page of your print atlas…

Charts in print layouts

We’ve previously covered this feature in depth, but the DataPlotly v3 release brings print-layout based charts to the masses! When a chart is inserted into a print layout, some additional options are available for controlling the plot behavior:

These new options allow you to link the chart to a map item within the layout, which lets you filter the content of the chart to only include features visible within the map. If your print layout is setup as an atlas export, you can also filter out included features to only show those which are geographically located inside the current atlas feature.

Our partners from Faunalia demonstrate this in the screencast below:

Saving plot configuration

An often-requested functionality previous missing from the plugin was the ability to save and restore plot configuration. Now, plot configuration is automatically saved within your QGIS project and restored when you reopen the project. You no longer have to re-create all your charts from scratch at every session (ouch!). We also added the ability to export chart configuration to XML files, allowing you to share and reuse chart configuration across projects.

Behind the scenes work

Aside from all the wonderful new features added to the plugin, we’ve extensively refactored most of the plugin backend. Unit tests and CI infrastructure have been added, which will ensure the plugin remains stable and regression-free in future releases. The code cleanup and simplification has drastically lowered the barrier of contribution to the project, and we’ve already seen new contributors adding more new features to the plugin as a result of this! (Kudos to Simon Gröchenig, who added the new “Feature subset” expression option you can see in the above screenshots!).

Project sponsors

All this work is thanks to the backers of our crowdfunding campaign. Without their contributions this work would not have been possible! In no particular order, our thanks go out to:

• Federico Gianoli
• Papercraft Mountains
• Liam McCrae
• Henry Walshaw
• Raúl Sangonzalo
• Ferdinando Urbano
• pitsch-ing.ch
• Carbon-X
• Gabriel Diosan
• Rene Giovanni Borella
• Enrico Bertonati
• Guido Ingwer
• David Addy
• Gerd Jünger
• Andreas Neumann
• Stefano Campus
• Michael Jabot
• Korto
• Enrico Ferreguti
• Carlo A. Nicolini
• Salvatore Fiandaca
• Alberto Grava
• Hans van der Kwast
• Ben Hur Pintor
• Silvio Grosso
• Nobusuke Iwasaki
• Alasdair Rae
• Manori Senanayake
• Canton de Neuchâtel
• Matthias Daues
• Alteri Seculo
• SunGIS Ltd.
• Stu Smith
• Keolis Rennes
• Gabriel Diosan
• Aiden Price
• Giacomo Ponticelli
• Diane Fritz
• Gemio Bissolati
• Claire Birnie
• Nicolas Roelandt
• Rocco Pispico
• Gabriel Bengtsson
• Birds Eye View
• Barend Köbben
• Roberto Marzocchi (GTER)
• Yoichi Kayama
• Alessandro Sarretta
• Luca Angeli
• Luca Bellani
• giswelt
• Stefan Giese
• Ben Harding
• Joao Gaspar
• Romain Lacroix
• Ryan Cooper
• Daniele Bonaposta
• QGIS Swedish User Group
• Nino Formica
• Michael Gieding
• Amedeo Fadini
• Andrew Hannell
• Stefano
• Phil Wyatt
• Brett Edmond Carlock
• Transitec

Keep an eye out on the North Road blog for future crowd-funding initiatives. Coming soon: a QGIS Processing grab-bag of ETL modelling improvements!

Sorry, this entry is only available in Dutch.

Sorry, this entry is only available in French.

Recently, staff at North Road have been hard at work on our SLYR “ESRI to QGIS compatiblity suite“, and we thought it’s time to share some of the latest exciting updates with you.

While SLYR begun life as a simple “LYR to QGIS conversion tool”, it quickly matured into a full ArcGIS compatibility suite for QGIS. Aside from its original task of converting ESRI LYR files, SLYR now extends the QGIS interface and adds seamless support for working with all kinds of ArcGIS projects and data files. It’s rapidly becoming a must-have tool for any organisation which uses a mix of ESRI and open source tools, or for any organisation exploring a transition away from ArcGIS to QGIS.

Accordingly, we thought it’s well past time we posted an update detailing the latest functionality and support we’ve added to SLYR over the past couple of months! Let’s dive in…

• Full support for raster LYR file conversion, including unique value renderers, color map renderers, classified renderers, RGB renderers and stretched color ramp renderers:

  

  

• Support for conversion of fill symbol outlines with complex offsets, decorations and dashed line templates
• Conversion of 3D marker and simple 3D lines to their 2d equivalent, matching ArcMap’s 2D rendering of these symbol types
• Beta support for converting map annotations and drawings, including custom text labels and reference scale support
  
• Label and annotation callout support*
• Support for converting bookmarks stored in MXD documents*
• Support for converting ESRI bookmark “.dat” files via drag and drop to QGIS*
  
• Correct conversion of OpenStreetMap and bing maps basemap layers
• SLYR now presents users with a friendly summary of warnings generated during the LYR or MXD conversion process (e.g. due to settings which can’t be matched in QGIS)
  
• Added support for MXD documents generated in very early ArcMap versions
• We’ve added QGIS Processing algorithms allowing for bulk LYR to QLR and MXD to QGS conversion. Now you can run a batch conversion process of ALL MXD/LYR files held at your organisation in one go!
  
• Greatly improved matching of converted symbols to their original ArcGIS appearance, including more support for undocumented ArcGIS symbol rendering behavior
• Support for conversion of text symbols and label settings stored in .style databases*
  
• Directly drag and drop layers and layer groups from ArcMap to QGIS to add them to the current QGIS project (maintaining their ArcGIS symbology and layer settings!)*
• Directly drag and drop layers from ArcCatalog to QGIS windows to open in QGIS*
• Support for ESRI MapServer layers

(*requires QGIS 3.10 or later)

Over the remainder of 2019, we’ll be hard at work further improving SLYR’s support for MXD document conversion, and adding support for automatic conversion of ArcMap print layouts to QGIS print layouts.

While SLYR is not currently an open-source tool, we believe strongly in the power of open source software, and accordingly we’ve been using a significant portion of the funds generated from SLYR sales to extend the core QGIS application itself. This has directly resulted in many exciting improvements to QGIS, which will become widely available in the upcoming QGIS 3.10 release. Some of the features directly funded by SLYR sales include:

• A “Segment Center” placement mode for marker line symbols
• Reworked bookmark handling in QGIS, with a greatly enhanced workflow and usability, and a stable API for 3rd party plugins and scripts to hook into
• Improved handling of layer symbology for layers with broken paths
• Auto repair of all other broken layers with a matching data source whenever a single layer path is fixed in a project
• Support for managing text formats and label settings in QGIS style libraries, allowing storage and management of label and text format presets
• A new Processing algorithm “Combine Style Databases“, allowing multiple QGIS style databases to be merged to one
• Adding a “Save layer styles into GeoPackage” option for the “Package Layers” algorithm
• New expression functions which return file info, such as file paths and base file names
• Adding new options to autofill the batch Processing dialog, including adding input files using recursive filter based file searches
• Coming in QGIS 3.12: A new option to set the color to use when rendering nodata pixels in raster layers
• Coming in QGIS 3.12: A new “random marker fill” symbol layer type, which fills polygons by placing point markers in random locations

You can read more about our SLYR ESRI to QGIS compatibility tool here, or email [email protected] to discuss licensing arrangements for your organisation! Alternatively, send us an email if you’d like to discuss your organisations approach to open-source GIS and for assistance in making this transition as painless as possible.

PT | EN

As I was preparing a QGIS Project to read a database structured according to the new rules and technical specifications for the Portuguese Cartography, I started to configure the editing forms for several layers, so that:

1. Make some fields read-only, like for example an identifier field.
2. Configure widgets better suited for each field, to help the user and avoid errors. For example, date-time files with a pop-up calendar, and value lists with dropdown selectors.

Basically, I wanted something like this:

Let me say that, in PostGIS layers, QGIS does a great job in figuring out the best widget to use for each field, as well as the constraints to apply. Which is a great help. Nevertheless, some need some extra configuration.

If I had only a few layers and fields, I would have done them all by hand, but after the 5th layer my personal mantra started to chime in:

“If you are using a computer to perform a repetitive manual task, you are doing it wrong!”

So, I began to think how could I configure the layers and fields more systematically. After some research and trial and error, I came up with the following PyQGIS functions.

Make a field Read-only

The identifier field (“identificador”) is automatically generated by the database. Therefore, the user shouldn’t edit it. So I had better make it read only

To make all the identifier fields read-only, I used the following code.

def field_readonly(layer, fieldname, option = True):
    fields = layer.fields()
    field_idx = fields.indexOf(fieldname)
    if field_idx >= 0:
        form_config = layer.editFormConfig()
        form_config.setReadOnly(field_idx, option)
        layer.setEditFormConfig(form_config)

# Example for the field "identificador"

project = QgsProject.instance()
layers = project.mapLayers() 

for layer in layers.values():
    field_readonly(layer,'identificador')

Set fields with DateTime widget

The date fields are configured automatically, but the default widget setting only outputs the date, and not date-time, as the rules required.

I started by setting a field in a layer exactly how I wanted, then I tried to figure out how those setting were saved in PyQGIS using the Python console:

>>>layer = iface.mapCanvas().currentLayer()
>>>layer.fields().indexOf('inicio_objeto')
1
>>>field = layer.fields()[1]
>>>field.editorWidgetSetup().type()
'DateTime'
>>>field.editorWidgetSetup().config()
{'allow_null': True, 'calendar_popup': True, 'display_format': 'yyyy-MM-dd HH:mm:ss', 'field_format': 'yyyy-MM-dd HH:mm:ss', 'field_iso_format': False}

Knowing this, I was able to create a function that allows configuring a field in a layer using the exact same settings, and apply it to all layers.

def field_to_datetime(layer, fieldname):
    config = {'allow_null': True,
              'calendar_popup': True,
              'display_format': 'yyyy-MM-dd HH:mm:ss',
              'field_format': 'yyyy-MM-dd HH:mm:ss',
              'field_iso_format': False}
    type = 'Datetime'
    fields = layer.fields()
    field_idx = fields.indexOf(fieldname)
    if field_idx >= 0:
        widget_setup = QgsEditorWidgetSetup(type,config)
        layer.setEditorWidgetSetup(field_idx, widget_setup)

# Example applied to "inicio_objeto" e "fim_objeto"

for layer in layers.values():
    field_to_datetime(layer,'inicio_objeto')
    field_to_datetime(layer,'fim_objeto')

Setting a field with the Value Relation widget

In the data model, many tables have fields that only allow a limited number of values. Those values are referenced to other tables, the Foreign keys.

In these cases, it’s quite helpful to use a Value Relation widget. To configure fields with it in a programmatic way, it’s quite similar to the earlier example, where we first neet to set an example and see how it’s stored, but in this case, each field has a slightly different settings

Luckily, whoever designed the data model, did a favor to us all by giving the same name to the fields and the related tables, making it possible to automatically adapt the settings for each case.

The function stars by gathering all fields in which the name starts with ‘valor_’ (value). Then, iterating over those fields, adapts the configuration to use the reference layer that as the same name as the field.

def field_to_value_relation(layer):
    fields = layer.fields()
    pattern = re.compile(r'^valor_')
    fields_valor = [field for field in fields if pattern.match(field.name())]
    if len(fields_valor) > 0:
        config = {'AllowMulti': False,
                  'AllowNull': True,
                  'FilterExpression': '',
                  'Key': 'identificador',
                  'Layer': '',
                  'NofColumns': 1,
                  'OrderByValue': False,
                  'UseCompleter': False,
                   'Value': 'descricao'}
        for field in fields_valor:
            field_idx = fields.indexOf(field.name())
            if field_idx >= 0:
                print(field)
                try:
                    target_layer = QgsProject.instance().mapLayersByName(field.name())[0]
                    config['Layer'] = target_layer.id()
                    widget_setup = QgsEditorWidgetSetup('ValueRelation',config)
                    layer.setEditorWidgetSetup(field_idx, widget_setup)
                except:
                    pass
            else:
                return False
    else:
        return False
    return True
    
# Correr função em todas as camadas
for layer in layers.values():
    field_to_value_relation(layer)

Conclusion

In a relatively quick way, I was able to set all the project’s layers with the widgets I needed.

This seems to me like the tip of the iceberg. If one has the need, with some search and patience, other configurations can be changed using PyQGIS. Therefore, think twice before embarking in configuring a big project, layer by layer, field by fields.

QGIS-versioning is a QGIS and PostGIS plugin dedicated to data versioning and history management. It supports :

• Keeping full table history with all modifications
• Transparent access to current data
• Versioning tables with branches
• Work offline
• Work on a data subset
• Conflict management with a GUI

QGIS versioning conflict management

In a previous blog article we detailed how QGIS versioning can manage data history, branches, and work offline with PostGIS-stored data and QGIS. We recently added foreign key support to QGIS versioning so you can now historize any complex database schema.

This QGIS plugin is available in the official QGIS plugin repository, and you can fork it on GitHub too !

Foreign key support

TL;DR

When a user decides to historize its PostgreSQL database with QGIS-versioning, the plugin alters the existing database schema and adds new fields in order to track down the different versions of a single table row. Every access to these versioned tables are subsequently made through updatable views in order to automatically fill in the new versioning fields.

Up to now, it was not possible to deal with primary keys and foreign keys : the original tables had to be constraints-free.  This limitation has been lifted thanks to this contribution.

To make it simple, the solution is to remove all constraints from the original database and transform them into a set of SQL check triggers installed on the working copy databases (SQLite or PostgreSQL). As verifications are made on the client side, it’s impossible to propagate invalid modifications on your base server when you “commit” updates.

Behind the curtains

When you choose to historize an existing database, a few fields are added to the existing table. Among these fields, versioning_ididentifies  one specific version of a row. For one existing row, there are several versions of this row, each with a different versioning_id but with the same original primary key field. As a consequence, that field cannot satisfy the unique constraint, so it cannot be a key, therefore no foreign key neither.

We therefore have to drop the primary key and foreign key constraints when historizing the table. Before removing them, constraints definitions are stored in a dedicated table so that these constraints can be checked later.

When the user checks out a specific table on a specific branch, QGIS-versioning uses that constraint table to build constraint checking triggers in the working copy. The way constraints are built depends on the checkout type (you can checkout in a SQLite file, in the master PostgreSQL database or in another PostgreSQL database).

What do we check ?

That’s where the fun begins ! The first thing we have to check is key uniqueness or foreign key referencing an existing key on insert or update. Remember that there are no primary key and foreign key anymore, we dropped them when activating historization. We keep the term for better understanding.

You also have to deal with deleting or updating a referenced row and the different ways of propagating the modification : cascade, set default, set null, or simply failure, as explained in PostgreSQL Foreign keys documentation .

Nevermind all that, this problem has been solved for you and everything is done automatically in QGIS-versioning. Before you ask, yes foreign keys spanning on multiple fields are also supported.

What’s new in QGIS ?

You will get a new message you probably already know about, when you try to make an invalid modification committing your changes to the master database

Error when foreign key constraint is violated

Partial checkout

One existing Qgis-versioning feature is partial checkout. It allows a user to select a subset of data to checkout in its working copy. It avoids downloading gigabytes of data you do not care about. You can, for instance, checkout features within a given spatial extent.

So far, so good. But if you have only a part of your data, you cannot ensure that modifying a data field as primary key will keep uniqueness. In this particular case, QGIS-versioning will trigger errors on commit, pointing out the invalid rows you have to modify so the unique constraint remains valid.

Error when committing non unique key after a partial checkout

Tests

There is a lot to check when you intend to replace the existing constraint system with your own constraint system based on triggers. In order to ensure QGIS-Versioning stability and reliability, we put some special effort on building a test set that cover all use cases and possible exceptions.

What’s next

There is now no known limitations on using QGIS-versioning on any of your database. If you think about a missing feature or just want to know more about QGIS and QGIS-versioning, feel free to contact us at [email protected]. And please have a look at our support offering for QGIS.

Many thanks to eHealth Africa who helped us develop these new features. eHealth Africa is a non-governmental organization based in Nigeria. Their mission is to build stronger health systems through the design and implementation of data-driven solutions.

Thanks to the success of our recent QGIS Print Layouts Graphs and Charts crowdfunding campaign, staff at North Road and Faunalia have been busy updating and improving the QGIS “DataPlotly” plugin with the goal of allowing beautiful charts inside your print layouts.

We’re super-excited to announce that the beta release of this new functionality is now available! With this beta installed, you’ll see a new icon in your QGIS Print Layout designer window:

Clicking this button will allow you to draw a new plot onto your print layout, just like you would any other standard layout item (like legends, scalebars, etc). Any print layout chart can be customised by right-clicking the chart and selecting “Item Properties“. This will open a panel with settings like position, size, frame, and other standard options. All the magic happens when you click the “Setup Plot” button inside this panel:

This exposes the vast array of styling and charting options available for use. If you’re an existing user of the DataPlotly QGIS plugin, you’ll recognise that these are the same settings you have available when creating interactive plots alongside the main map canvas. Every setting is now available for use in print layouts!

To grab the beta, head over to https://github.com/ghtmtt/DataPlotly/releases/tag/v3.9-beta and download the DataPlotly.zip file. Then, inside QGIS, select the Manage and Install Plugins option from the Plugins menu. Click on the “Install from ZIP” section, and point the dialog at your downloaded DataPlotly.zip file. Click “Install Plugin“, and then restart QGIS for good measure. When QGIS restarts you should see the new chart icon inside the print layout designer.

Note that you’ll need a recent QGIS release for this to work correctly — either QGIS 3.8.3 or 3.4.12. (The print layout functionality may not be compatible with earlier releases, as we’ve had to fix several issues inside QGIS itself to get this feature working as designed!). 

We are actively seeking feedback and user testing on this beta release. If you encounter any issues, just head over to https://github.com/ghtmtt/DataPlotly/issues and let us know.

We’ll be further refining this functionality, with the goal of releasing the final non-beta version of the plugin to coincide with the upcoming QGIS 3.10 release.

Happy charting!

Recently, we’ve been working on an exciting development which is coming soon in QGIS 3.10… support for Geospatial PDF exports! This has been a long-desired feature for many QGIS users, and it was only made possible thanks to a group of financial backers (listed below). In this post, we’re going to explore these new features and how they improve your QGIS PDF outputs.

Geospatial PDFs can now be created either by exporting the main QGIS map canvas, or by creating and exporting a custom print layout. For instance, when you select the “Save Map as PDF” option from the main QGIS window, you’ll see a new group of Geospatial PDF related options:

At its most basic, Geospatial PDF is a standard extension to the PDF format which allows for vector spatial datasets to be embedded in PDF files. If the “Include vector feature information” checkbox is ticked when creating a Geospatial PDF output, then QGIS will automatically include all the geometry and attribute information from features visible within the page. So if we export a simple map to PDF, we’ll get an output file which looks just like any old regular PDF map output…

…but, we can also pull this PDF back into QGIS and treat it just like any other vector data source! In the screenshot below we’re using the Identify tool to query on of the polygons and see all the attribute information from the original source layer.

This ability adds a lot of value to PDF exports. Anyone who has ever been supplied a non-spatial PDF as a “spatial dataset” will attest to the frustrations these cause… but if you create proper Geospatial PDFs instead, then there’s no loss of the valuable underlying spatial information or feature attributes! Furthermore, if these PDFs are opened within Acrobat Reader, tools are enabled which allow users to query features interactively.

Another nice benefit which comes with Geospatial PDF output is that layers can be interactively toggled on or off in the PDF viewer. The screenshot below shows a Geospatial PDF file created from a simple QGIS map. On the left we have a list of the layers in the PDF, each of which can be turned on or off inside the PDF viewer!

The really nice thing here is that, thanks to the underlying smarts in the GDAL library which is responsible for the actual Geospatial PDF creation, the PDF renders identically to our original QGIS map. While labels turn on and off alongside their corresponding map layer, they are still correctly stacked in the exact same way as you see in the QGIS window. Furthermore, the created PDFs keep labels and vector features as vector artwork… so there’s absolutely no loss in quality when zooming in to the map! These files look GREAT!

On that same note… the sponsorship allowed us to tackle another related issue, which is that in previous QGIS versions PDF (or SVG) exports would always export every single vertex from any visible feature! Ouch! This meant that if you had a complex polygon boundary layer, you would potentially be creating a PDF with millions of vertices per feature, even though most of these would be overlapping and completely redundant at the exported map’s scale. Now, QGIS automatically simplifies vector features while exporting them (using an appropriate, effectively invisible, level of simplification). The dramatically reduces the created file sizes and speeds up opening them and navigating them in other applications (especially Inkscape). (There’s an option at export time to disable this simplification, if you have a specific reason to!).

Creating Geospatial PDFs from print layouts gives even more options. For a start, whenever a print layout is exported to Geospatial PDFs, we ensure that the created PDF correctly handles stacking of layers alongside any other print layout items you have. In the image below we see a custom print layout which includes interactive layer visibility controls. If a layer is toggled, it’s hidden only from the map item — all the other surrounding elements like the title, north arrow and scalebar remain visible:

That’s not all though! When exporting a print layout to Geospatial PDF, QGIS also hooks into any map themes you’ve setup in your project. If you select to include these themes in your output, then the result is magical! The screenshot below shows the export options for a project with a number of themes, and we’ve chosen to export these themes in the PDF:

Opening the resultant PDF shows that our layer control on the left now lists the map themes instead of individual layers. Viewers can switch between these themes, changing the visibility of layers and their styling to match the QGIS map theme from the project! Additionally, you can even expand out a theme and expose layer-by-layer visibility control. Wow! This means you could create a single PDF output file which includes an environmental, social, cadastral, transport, …. view of your map, all in the one file.

Lastly, there’s even control for fine-tuning the combination of layers which are exposed in the output PDF file and which ones should be toggled on and off together. In the screenshot below we’ve opted to group the “Aircraft” and “Roads” map layers into a single logical PDF group called “Transport”. 

The resultant PDF respects this, showing an entry in the interactive layer tree for “Transport” which toggles both the aircraft and roads layers together:

So there you go — the power of Geospatial PDF, coming your way in QGIS 3.10!

One semi-related benefit of this work is that it gave us an opportunity to rework how “layered” exports from print layouts are created. This has had a significant flow-on impact on the existing ability to create layered SVG outputs from QGIS. Previously, this was a rather fragile feature, which created SVGs with lots of issues – overlapping labels, incorrectly stacked layers, and last-but-not-least, non-descriptive layer names! Now, just like Geospatial PDF exports, the layered SVG exports correctly respect the exact look of your map, and have much more friendly, descriptive layer names:

This should significantly reduce the amount of housekeeping required when working on these layered SVG exports. 

This work was funded by:

• Land Vorarlberg
• Municipality of Vienna
• Municipality of Dornbirn
• Biodiversity Information Service for Powys and BBNP Local
• Kanton Zug
• Canton de Neuchâtel
• Canton de Thurgovia

North Road are leading experts in extending the QGIS application to meet your needs. If you’d like to discuss how you can sponsor development of features or fixes which you want in QGIS, just contact us for further details!

Currently, there’s around 20 persons in Bucharest working on QGIS development during the Contributors Meeting. And because the name of the hackfest is changed from developers meeting to contributors meeting, I now feel welcome too (as a non-coding contributor). So what can I do, as a non-coding QGIS fan? Write documentation! I just started with […]

Sorry, this entry is only available in French.