Related Plugins and Tags

QGIS Planet

QGIS 3D Tiles – thanks to Cesium Ecosystem Grant!

We’ve recently had the opportunity to implement a very exciting feature in QGIS 3.34 — the ability to load and view 3D content in the “Cesium 3D Tiles” format! This was a joint project with our (very talented!) partners at Lutra Consulting, and was made possible thanks to a generous ecosystem grant from the Cesium project.

Before we dive into all the details, let’s take a quick guided tour showcasing how Cesium 3D Tiles work inside QGIS:

What are 3D tiles?

Cesium 3D Tiles are an OGC standard data format where the content from a 3D scene is split up into multiple individual tiles. You can think of them a little like a 3D version of the vector tile format we’ve all come to rely upon. The 3D objects from the scene are stored in a generalized, simplified form for small-scale, “zoomed out” maps, and in more detailed, complex forms for when the map is zoomed in. This allows the scenes to be incredibly detailed, whilst still covering huge geographic regions (including the whole globe!) and remaining responsive and quick to download. Take a look at the incredible level of detail available in a Cesium 3D Tiles scene in the example below:

Where can you get 3D tile content?

If you’re lucky, your regional government or data custodians are already publishing 3D “digital twins” of your area. Cesium 3D Tiles are the standard way that these digital twin datasets are being published. Check your regional data portals and government open data hubs and see whether they’ve made any content available as 3D tiles. (For Australian users, there’s tons of great content available on the Terria platform!).

Alternatively, there’s many datasets available via the Cesium ion platform. This includes global 3D buildings based on OpenStreetMap data, and the entirety of Google’s photorealistic Google Earth tiles! We’ve published a Cesium ion QGIS plugin to complement the QGIS 3.34 release, which helps make it super-easy to directly load datasets from ion into your QGIS projects.

Lastly, users of the OpenDroneMap photogrammetry application will already have Cesium 3D Tiles datasets of their projects available, as 3D tiles are one of the standard outputs generated by OpenDroneMap.

Why QGIS?

So why exactly would you want to access Cesium 3D tiles within QGIS? Well, for a start, 3D Tiles datasets are intrinsically geospatial data. All the 3D content from these datasets are georeferenced and have accurate spatial information present. By loading a 3D tiles dataset into QGIS, you can easily overlay and compare 3D tile content to all your other standard spatial data formats (such as Shapefiles, Geopackages, raster layers, mesh datasets, WMS layers, etc…). They become just another layer of spatial information in your QGIS projects, and  you can utilise all the tools and capabilities you’re familiar with in QGIS for analysing spatial data along with these new data sources.

One large drawcard of adding a Cesium 3D Tile dataset to your QGIS project is that they make fantastic 3D basemaps. While QGIS has had good support for 3D maps for a number of years now, it has been tricky to create beautiful 3D content. That’s because all the standard spatial data formats tend to give generalised, “blocky” representations of objects in 3D. For example, you could use an extruded building footprint file to show buildings in a 3D map but they’ll all be colored as idealised solid blocks. In contrast, Cesium 3D Tiles are a perfect fit for a 3D basemap! They typically include photorealistic textures, and include all types of real-world features you’d expect to see in a 3D map — including buildings, trees, bridges, cliffsides, etc.

What next?

If you’re keen to learn even more about Cesium 3D Tiles in QGIS, you can check out the recent “QGIS Open Day” session we presented. In this session we cover all the details about 3D tiles and QGIS, and talk in depth about what’s possible in QGIS 3.34 and what may be coming in later releases.

Otherwise, grab the latest QGIS 3.34 and start playing…. you’ll quickly find that Cesium 3D Tiles are a fun and valuable addition to QGIS’ capabilities!

Our thanks go to Cesium and their ecosystem grant project for funding this work and making it possible.

QGIS 3.28 improvements for working with ESRI formats and services

The QGIS 3.28 release is an extremely exciting release for all users who work in mixed software workplaces, or who need to work alongside users of ESRI software. In this post we’ll be giving an overview of all the new tools and features introduced in 3.28 which together result in a dramatic improvement in the workflows and capabilities in working with ESRI based formats and services. Read on for the full details…!

Before we begin, we’d like to credit the following organisations for helping fund these developments in QGIS 3.28:

  • Naturstyrelsen, Denmark
  • Provincie Gelderland, Netherlands
  • Uppsala Universitet, Department of Archaeology and Ancient History
  • Gemeente Amsterdam
  • Provincie Zuid-Holland, Netherlands

FileGeodatabase (GDB) related improvements

The headline item here is that QGIS 3.28 introduces support for editing, managing and creating ESRI FileGeodatabases out of the box! While older QGIS releases offered some limited support for editing FileGeodatabase layers, this required the manual installation of a closed source ESRI SDK driver… which unfortunately resulted in other regressions in working with FileGeodatabases (such as poor layer loading speed and random crashes). Now, thanks to an incredible reverse engineering effort by the GDAL team, the open-source driver for FileGeodatabases offers full support for editing these datasets! This means all QGIS users have out-of-the-box access to a fully functional, high-performance read AND write GDB driver, no further action or trade-offs required.

Operations supported by the GDAL open source driver include:

  • Editing existing features, with full support for editing attributes and curved, 3D and measure-value geometries
  • Creating new features
  • Deleting features
  • Creating, adding and modifying attributes in an existing layer
  • Full support for reading and updating spatial indexes
  • Creating new indexes on attributes
  • “Repacking” layers, to reduce their size and improve performance
  • Creating new layers in an existing FileGeodatabase
  • Removing layers from FileGeodatabases
  • Creating completely new, empty FileGeodatabases
  • Creating and managing field domains

On the QGIS side, the improvements to the GDAL driver meant that we could easily expose feature editing support for FileGeodatabase layers for all QGIS users. While this is a huge step forward, especially for users in mixed software workplaces, we weren’t happy to rest there when we  had the opportunity to further improve GDB support within QGIS!

So in QGIS 3.28 we also introduced the following new functionality when working with FileGeodatabases:

FileGeodatabase management tools

QGIS 3.28 introduces a whole range of GUI based tools for managing FileGeodatabases. To create a brand new FileGeodatabase, you can now right click on a directory from the QGIS Browser panel and select New – ESRI FileGeodatabase:

After creating your new database, a right click on its entry will show a bunch of available options for managing the database. These include options for creating new tables, running arbitrary SQL commands, and database-level operations such as compacting the database:

You’re also able to directly import existing data into a FileGeodatabase by simply dragging and dropping layers onto the database!

Expanding out the GDB item will show a list of layers present in the database, and present options for managing the fields in those layers. Alongside field creation, you can also remove and rename existing fields.

Field domain handling

QGIS 3.28 also introduces a range of GUI tools for working with field domains inside FileGeodatabases. (GeoPackage users also share in the love here — these same tools are all available for working with field domains inside this standard format too!) Just right click on an existing FileGeodatabase (or GeoPackage) and select the “New Field Domain” option. Depending on the database format, you’ll be presented with a list of matching field domain types:

Once again, you’ll be guided through a user-friendly dialog allowing you to create your desired field domain!

After field domains have been created, they can be assigned to fields in the database by right-clicking on the field name and selecting “Set Field Domain”:

Field domains can also be viewed and managed by expanding out the “Field domains” option for each database.

Relationship discovery

Another exciting addition in QGIS 3.28 (and the underlying GDAL 3.6 release) is support for discovering database relationships in FileGeodatabases! (Once again, GeoPackage users also benefit from this, as we’ve implemented full support for GeoPackage relationships via the “Related Tables Extension“).

Expanding out a database containing any relationships will show a list of all discovered relationships:

(You can view the full description and details for any of these relationships by opening the QGIS Browser “Properties” panel).

Whenever QGIS 3.28 discovers relationships in the database, these related tables will automatically be added to your project whenever any of the layers which participate in the relationship are opened. This means that users get the full experience as designed for these databases without any manual configuration, and the relationships will “just work”!

Dataset Grouping

Lastly, we’ve improved the way layers from FileGeodatabases are shown in QGIS, so that layers are now grouped according to their original dataset groupings from the database structure:

Edit ArcGIS Online / Feature Service layers

While QGIS has had read-only support for viewing and working with the data in ArcGIS Online (AGOL) vector layers and ArcGIS Server “feature service” layers for many years, we’ve added support for editing these layers in QGIS 3.28. This allows you to take advantage of all of QGIS’ easy to use, powerful editing tools and directly edit the content in these layers from within your QGIS projects! You can freely create new features, delete features, and modify the shape and attributes of existing features (assuming that your user account on the ArcGIS service has these edit permissions granted, of course). This is an exciting addition for anyone who has to work often with content in ArcGIS services, and would prefer to directly manipulate these layers from within QGIS instead of the limited editing tools available on the AGOL/Portal platforms themselves.

This new functionality will be available immediately to users upon upgrading to QGIS 3.28 — any users who have been granted edit capabilities for the layers will see that the QGIS edit tools are all enabled and ready for use without any further configuration on the QGIS client side.

Filtering Feature Service layers

We’ve also had the opportunity to introduce filter/query support for Feature Service layers in QGIS 3.28. This is a huge performance improvement for users who need to work with a subset of a features from a large Feature Service layer. Unfortunately, due to the nature of the Feature Service protocol, these layers can often be slow to load and navigate on a client side. By setting a SQL filter to limit the features retrieved from the service the performance can be dramatically increased, as only matching features will ever be requested from the backend server. You can use any SQL query which conforms to the subset of SQL understood by ArcGIS servers (see the Feature Service documentation for examples of supported SQL queries).

 

What’s next?

While QGIS 3.28 is an extremely exciting release for any users who need to work alongside ESRI software, we aren’t content to rest here! The exciting news is that in QGIS 3.30 we’ll be introducing a GUI driven approach allowing users to create new relationships in their FileGeodatabase (and GeoPackage!) databases.

At North Road we’re always continuing to improve the cross-vendor experience for both ESRI and open-source users through our continued work on the QGIS desktop application and our SLYR conversion suite. If you’d like to chat to us about how we can help your workplace transition from a fully ESRI stack to a mixed or fully open-source stack, just contact us to discuss your needs.

SLYR Update — November 2022

Our SLYR tool is the complete solution for full compatibility between ArcMap, ArcGIS Pro and QGIS. It offers a powerful suite of conversion tools for opening ESRI projects, styles and other documents directly within QGIS, and for exporting QGIS documents for use in ESRI software.

A lot has changed since our last SLYR product update post, and we’ve tons of very exciting improvements and news to share with you all! In this update we’ll explore some of the new tools we’ve added to SLYR, and discuss how these tools have drastically improved the capacity for users to migrate projects from the ESRI world to the open-source world (and vice versa).

ArcGIS Pro support

The headline item here is that SLYR now offers a powerful set of tools for working with the newer ArcGIS Pro document formats. Previously, SLYR offered support for the older ArcMap document types only (such as MXD, MXT, LYR, and PMF formats). Current SLYR versions now include tools for:

Directly opening ArcGIS Pro .lyrx files within QGIS

LYRX files can be dragged and dropped directly onto a QGIS window to add the layer to the current project. All the layer’s original styling and other properties will be automatically converted across, so the resultant layer will be an extremely close match to the original ArcGIS Pro layer! SLYR supports vector layers, raster layers, TIN layers, point cloud layers and vector tile layers. We take great pride in just how close the conversion results are to how these layers appear in ArcGIS Pro… in most cases you’ll find the results are nearly pixel perfect!

In addition to drag-and-drop import support, SLYR also adds support for showing .lyrx files directly in the integrated file browser, and also adds tools to the QGIS Processing Toolbox so that users can execute bulk conversion operations, or include document conversion in their models or custom scripts.

ArcGIS Pro map (mapx) and project (aprx) conversion

Alongside the LYRX support, we’ve also added support for the ArcGIS Pro .mapx and .aprx formats. Just like our existing .mxd conversion, you can now easily convert entire ArcGIS Pro maps for direct use within QGIS! SLYR supports both the older ArcGIS Pro 2.x project format and the newer 3.x formats.

Export from QGIS to ArcGIS Pro!

Yes, you read that correctly… SLYR now allows you to export QGIS documents into ArcGIS Pro formats! This is an extremely exciting development… for the first time ever QGIS users now have the capacity to export their work into formats which can be supplied directly to ESRI users. Current SLYR versions support conversion of map layers to .lyrx format, and exporting entire projects to the .mapx format. (We’ll be introducing support for direct QGIS to .aprx exports later this year.)

We’re so happy to finally provide an option for QGIS users to work alongside ArcGIS Pro users. This has long been a pain point for many organisations, and has even caused organisations to be ineligible to tender for jobs which they are otherwise fully qualified to do (when tenders require provision of data and maps in ArcGIS compatible formats).

ArcGIS Pro .stylx support

Alongside the other ArcGIS Pro documents, SLYR now has comprehensive support for reading and writing ArcGIS Pro .stylx databases. We’ve dedicated a ton of resources in ensuring that the conversion results (both from ArcGIS Pro to QGIS and from QGIS to ArcGIS Pro) are top-notch, and we even handle advanced ArcGIS Pro symbology options like symbol effects!

Take a look below how even very advanced ArcGIS Pro style libraries convert beautifully to QGIS symbol libraries:

ArcMap Improvements

While we’ve been focusing heavily on the newer ArcGIS Pro formats, we’ve also improved our support for the older ArcMap documents. In particular, SLYR now offers more options for converting ArcMap annotation layers and annotation classes to QGIS supported formats. Users can now convert Annotation layers and classes directly over to QGIS annotation layer or alternatively annotation classes can be converted over to the OGC standard GeoPackage format. When exporting annotation classes to GeoPackage the output database is automatically setup with default styling rules, so that the result can be opened directly in QGIS and will be immediately visualised to match the original annotation class.

Coming soon…

While all the above improvements are already available for all SLYR license holders, we’ve got many further improvements heading your way soon! For example, before the end of 2022 we’ll be releasing another large SLYR update which will introduce support for exporting QGIS projects directly to ArcGIS Pro .aprx documents. We’ve also got many enhancements planned which will further improve the quality of the converted documents. Keep an eye on this blog and our social media channels for more details as they are available…

You can read more about our SLYR tool at the product page, or contact us today to discuss licensing options for your organisation.

 

Crowd funding alert: Point Clouds, Elevation Profiles and 3D Map view improvements!

Together with our partners at Lutra Consulting and Hobu, we have once again have collated your feature requests for even MORE point cloud and QGIS elevation improvements and are ready to start working on them!

Here’s a taster of what to expect if our latest crowd funding campaign is successful:

  • Point cloud data management, transformation and analysis via incorporation of PDAL within the QGIS “Processing” toolbox
  • Optimised handling of large datasets to speed up your 3D maps, including dynamic data loading for complex 3D scenes
  • Combine multiple LAS/LAZ point cloud files into a single “virtual” point cloud for easy data management and display
  • Elevation profile tool embedded into print layouts, totally customizable, more efficient and with the ability to export profile data to CSV and DXF
  • General improvements to QGIS 3D map views, including an enhanced 3D measuring tool, additional camera controls and improving the configuration dialog options and functionality.

To find out more, check out the ‘Detailed proposal and deliverables’ section on the main crowd funding page. You may want to get some popcorn ready, it really is exciting!

We need your help to make this work possible. If you’d like to see these enhancements, pledge to the crowdfunding campaign before October 24, 2022.

Securely accessing ArcGIS Online (AGOL) and enterprise ArcGIS Portal sites through QGIS (2022 update!)

We’re often contacted for advice regarding our recommendations for securely accessing content on ArcGIS Online (AGOL) or enterprise ArcGIS Portal deployments from within QGIS. While we ran through our recommended setup in an older post, we thought it was time for a 2022 update for our recommendations. In this version we’ve updated some of the descriptions and screenshots for newer QGIS releases, and have added explicit instructions on accessing ArcGIS Online content too.

This post details step-by-step instructions in setting up both AGOL/ArcGIS Portal and QGIS to enable this integration. First, we’ll need to create an application on the server in order to enable QGIS users to securely authenticate with the server. The process for this varies between AGOL and ArcGIS Portal, so we’ve separated the guidelines into two sections below:

ArcGIS Portal

Creating an application

Logon to the Portal, and from the “Content” tab, click the “Add Item” option. Select “An application” from the drop down list of options:

Set the type of the application as “Desktop

You can fill out the rest of this dialog as you see fit. Suggested values are:

  • Purpose: Ready to Use
  • Platform: Qt
  • URL: https://qgis.org
  • Tags: QGIS, Desktop, etc

Now – here comes a trick. Portal will force you to attach a file for the application. It doesn’t matter what you attach here, so long as it’s a zip file. While you could attach a zipped copy of the QGIS installer, that’s rather wasteful of server space! We’d generally just opt for a zip file containing a text file with a download link in it.

Click Add Item when you’re all done filling out the form, and the new application should be created on the Portal.

Registering the Application

The next step is to register the application on Portal, so that you can obtain the keys required for the OAuth2 logon using it. From the newly created item’s page, click on the “Settings” tab:

Scroll right to the bottom of this page, and you should see a “Register” button. Press this. Set the “App type” to “Native“.

Add two redirect URIs to the list (don’t forget to click “Add” after entering each!):

  1. The Portal’s public address, e.g. https://mydomain.com/portal
  2. http://127.0.0.1:7070

Finally, press the “Register” button in the dialog. If all goes well then the App Registration section in the item settings should now be populated with details. From here, copy the “App ID” and “Secret” strings, we’ll need these later:

Determine Request URLs

One last configuration setting we’ll need to determine before we fire up QGIS is the Portal’s OAuth Request and Token URLs. These are usually found by appending /sharing/rest/oauth2/authorize and /sharing/rest/oauth2/token to the end of your Portal’s URL.

For instance, if your public Portal URL is http://mydomain.com/portal, then the URLs will be:

Request URL: http://mydomain.com/portal/sharing/rest/oauth2/authorize
Token URL: http://mydomain.com/portal/sharing/rest/oauth2/token

You should be able to open both URLs directly in a browser. The Request URL will likely give a “redirect URL not specified” error, and the Token URL will give a “client_id not specified” error. That’s ok — it’s enough to verify that the URLs are correct.

When this is all done we can skip ahead to the client-side configuration below.

ArcGIS Online (AGOL)

Registering an application on AGOL must be done by an account administrator. So first, we’ll logon to AGOL using an appropriate account, and then head over to the “Content” tab. We’ll then hit the “New Item” button to start creating our application:

From the available item types select the “Application” option:

We’ll then select the “Mobile” application type, and enter “https://qgis.org” as the application URL (it actually doesn’t matter what we enter here!):

Lastly, we can enter a title for the application and complete all the metadata options as desired, then hit the “Save” button to create the application. AGOL should then take us over to the applications content page (if not, just open that page manually). From the application summary page, click across to the “Settings” tab:

Scroll right down to the “Credentials” section at the bottom of this page and click the “Register application” button:

Enter the following “Redirect URLs”, by copying each value and then hitting “Add”

  • localhost
  • http://127.0.0.1:7070
  • https://127.0.0.1:7070

Set the “Application environment” to Browser and then click “Register” button to continue:

You’ll be taken back to the application Settings page, but should now see an Client ID value and option to show the Client Secret. Copy the “Client ID” and “Client Secret” (there’s a “copy to clipboard” button for this) as we’ll need these later:

We’re all done on the AGOL side now, so it’s time to fire up QGIS!

Creating an QGIS OAuth2 Authentication Configuration

From your QGIS application, select Options from the Settings menu. Select the Authentication tab. We need to create a new authentication configuration, so press the green + button on the right hand side of the dialog. You’ll get a new dialog prompting you for Authentication details.

You may be asked to create a “Master Password” when you first create an authentication setup. If so, create a secure password there before continuing.

There’s a few tricks to this setup. Firstly, it’s important to ensure that you use the exact same settings on all your client machines. This includes the authentication ID field, which defaults to an auto-generated random string. (While it’s possible to automatically deploy the configuration as part of a startup or QGIS setup script, we won’t be covering that here!).

So, from the top of the dialog, we’ll fill in the “Name” field with a descriptive name of the Portal site. You then need to “unlock” the “Id” field by clicking the little padlock icon, and then you’ll be able to enter a standard ID to identify the Portal. The Id field is very strict, and will only accept a 7 letter string! Since you’ll need to create a different authentication setup for each individual ArcGIS Portal site you access, make sure you choose your ID string accordingly. (If you’re an AGOL user, then you’ll only need to create one authentication setup which will work for any AGOL account you want to access).

Drop down the Authentication Type combo box, and select “OAuth2 Authentication” from the list of options. There’s lots of settings we need to fill in here, but here’s what you’ll need:

  • Grant flow: set to “Authorization Code”
  • Request URL: 
    • For ArcGIS Portal services: enter the Request URL we determined in the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/authorize
    • For AGOL services: enter https://www.arcgis.com/sharing/rest/oauth2/authorize
  • Token URL:
    • For ArcGIS Portal services: enter the Token URL from the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/token
    • For AGOL services: enter https://www.arcgis.com/sharing/rest/oauth2/token
  • Refresh Token URL: leave empty
  • Redirect URL: leave as the default http://127.0.0.1:7070 value
  • Client ID: enter the App ID (or Client ID) from the App Registration information (see earlier steps)
  • Client Secret: enter the App Secret (or Client Secret) from the App Registration information (see earlier steps)
  • Scope: leave empty
  • API Key: leave empty
  • For AGOL services only: you’ll also need to set the “Token header” option under the “Advanced” heading. This needs to be “X-Esri-Authorization” (without the quotes!)

That’s it — leave all the rest of the settings at their default values, and click Save.

You can close down the Options dialog now.

Adding the Connection Details

Lastly, we’ll need to setup the server connection as an “ArcGIS Rest Server Connection” in QGIS. This is done through the QGIS “Data Source Manager” dialog, accessed through the Layer menu. Click the “ArcGIS REST Server” tab to start with, and then press “New” in the Server Connections group at the top of this dialog.

Enter a descriptive name for the connection, and then enter the connection URLs. These will vary depending on whether you’re connecting to AGOL or an ArcGIS Portal server:

ArcGIS Online Connection

  • URL: This will be something similar to “https://services###.arcgis.com/########/arcgis/rest/services/”, but will vary organisation by organisation. You can determine your organisation’s URL by visiting the overview page for any dataset published on your AGOL account, and looking under the “Details” sidebar. Copy the address from the “Source: Feature Service” link and it will contain the correct URL parameters for your organisation. (Just make sure you truncate the link to end after the “arcgis/rest/services” part, you don’t require the full service endpoint URL)
  • Community endpoint URL: https://www.arcgis.com/sharing/rest/community
  • Content endpoint URL: https://www.arcgis.com/sharing/rest/content

ArcGIS Portal Connection

  • URL:  the REST endpoint associated with your Portal, e.g. “https://mydomain.com/arcgis/rest/services”
  • Community endpoint URL (optional): This will vary depending on the server setup, but will generally take the form “https://mydomain.com/portal/sharing/rest/community”
  • Content endpoint URL (optional): This will vary depending on the server setup, but will generally take the form “https://mydomain.com/portal/sharing/rest/content”

Lastly, select the new OAuth2 configuration you just created under the “Authentication” group:

Click OK, and you’re done! When you try to connect to the newly added connection, you’ll automatically be taken to the AGOL or ArcGIS Portal’s logon screen in order to authenticate with the service. After entering your details, you’ll then be connected securely to the server and will have access to all items which are shared with your user account!

We’ve regularly use this setup for our enterprise clients, and have found it to work flawlessly in recent QGIS versions. If you’ve found this useful and are interested in other “best-practice” recommendations for mixed Open-Source and ESRI workplaces, don’t hesitate to contact us to discuss your requirements… at North Road we specialise in ensuring flawless integration between ESRI based systems and the Open Source geospatial software stack.

Announcing SLYR Community Edition

North Road are proud to announce the official release of SLYR Community Edition, a new open-source version of our powerful SLYR ESRI to Open Source compatibility suite. The Community Edition is available for download from the official QGIS plugin repository today, for QGIS versions 3.4 and above. It supports automated conversion of ESRI .style symbol databases, including conversion of markers, fills, line styles and color ramps to their closest QGIS symbology equivalent, allowing users to instantly transition their style libraries into QGIS!

If you’ve followed our work in the past, it will come as no surprise to hear that North Road are passionate about open source geospatial, and for reducing the barriers which users encounter when moving to open-source software. We see our SLYR tool as an integral part of this process, and the licensed version of the plugin currently supports automated conversion of MXD, LYR, PMF, and other ESRI-specific formats to QGIS documents.

Our intention all along has been to make this tool freely available for all users of open-source geospatial software, and to release our work under a permissive, open-source license so that other projects can take advantage of our reverse engineering efforts. That’s why we made an “open source pledge” a fundamental part of our SLYR tool development! By the terms of this pledge, exactly six months after we hit staged preset funding levels we will open-source more components of the code and update the community version of the plugin accordingly. (This approach gives motivated organisations instant access to the full functionality of the SLYR tool via a license purchase, or free access to a subset of this functionality via the community edition of the plugin. It allows us to heavily invest in further reverse-engineering efforts and improvements to the plugin, to QGIS, and to the wider open-source geospatial community.)

If you’re keen to explore transitioning your workplace from ESRI to open-source, send us an email to discuss what we can offer! North Road staff have years of experience in implementing open-source geospatial solutions within commercial workplaces, and for setting up dual commercial-and-open-source friendly environments.

Securely accessing enterprise ArcGIS Portal sites through QGIS

An updated version of this guide is available!

We were recently contacted for advice regarding our recommendations for securely accessing content on an enterprise ArcGIS Portal deployment from within QGIS. Fortunately, this setup is fully supported and works seamlessly in QGIS, thanks to the native integration of “OAuth2” authentication in QGIS!

This post details step-by-step instructions in setting up both ArcGIS Portal and QGIS to enable this integration. First, we’ll create a new desktop application on the Portal site in order to obtain the application-specific access keys for OAuth2 authentication. We’ll then create an authentication configuration in QGIS and associate this with a connection to the Portal site. Let’s dive in by doing the Portal configuration first…

Creating an application

Logon to the Portal, and from the “Content” tab, click the “Add Item” option. Select “An application” from the drop down list of options:

Set the type of the application as “Desktop

You can fill out the rest of this dialog as you see fit. Suggested values are:

  • Purpose: Ready to Use
  • Platform: Qt
  • URL: http://qgis.org
  • Tags: QGIS, Desktop, etc

Now – here comes a trick. Portal will force you to attach a file for the application. It doesn’t matter what you attach here, so long as it’s a zip file. While you could attach a zipped copy of the QGIS installer, that’s rather wasteful of server space! We’d generally just opt for a zip file containing a text file with a download link in it.

Click Add Item when you’re all done filling out the form, and the new application should be created on the Portal.

Registering the Application

The next step is to register the application on Portal, so that you can obtain the keys required for the OAuth2 logon using it. From the newly created item’s page, click on the “Settings” tab:

Scroll right to the bottom of this page, and you should see a “Register” button. Press this. Set the “App type” to “Native“.

Add two redirect URIs to the list (don’t forget to click “Add” after entering each!):

  1. The Portal’s public address, e.g. https://mydomain.com/portal
  2. http://127.0.0.1:7070

Finally, press the “Register” button in the dialog. If all goes well then the App Registration section in the item settings should now be populated with details. From here, copy the “App ID” and “Secret” strings, we’ll need these later:

Determine Request URLs

One last configuration setting we’ll need to determine before we fire up QGIS is the Portal’s OAuth Request and Token URLs. These are usually found by appending /sharing/rest/oauth2/authorize and /sharing/rest/oauth2/token to the end of your Portal’s URL.

For instance, if your public Portal URL is http://mydomain.com/portal, then the URLs will be:

Request URL: http://mydomain.com/portal/sharing/rest/oauth2/authorize
Token URL: http://mydomain.com/portal/sharing/rest/oauth2/token

You should be able to open both URLs directly in a browser. The Request URL will likely give a “redirect URL not specified” error, and the Token URL will give a “client_id not specified” error. That’s ok — it’s enough to verify that the URLs are correct.

We’re all done on the Portal side now, so time to fire up QGIS!

Creating an QGIS OAuth2 Authentication Configuration

From your QGIS application, select Options from the Settings menu. Select the Authentication tab. We need to create a new authentication configuration, so press the green + button on the right hand side of the dialog. You’ll get a new dialog prompting you for Authentication details.

There’s a few tricks to this setup. Firstly, it’s important to ensure that you use the exact same settings on all your client machines. This includes the authentication ID field, which defaults to an auto-generated random string. (While it’s possible to automatically deploy the configuration as part of a startup or QGIS setup script, we won’t be covering that here!).

So, from the top of the dialog, we’ll fill in the “Name” field with a descriptive name of the Portal site. You then need to “unlock” the “Id” field by clicking the little padlock icon, and then you’ll be able to enter a standard ID to identify the Portal. The Id field is very strict, and will only accept a 7 letter string!

Drop down the Authentication Type combo box, and select “OAuth2 Authentication” from the list of options. There’s lots of settings we need to fill in here, but here’s what you’ll need:

  • Grant flow: set to “Authorization Code”
  • Request URL: enter the Request URL we determined in the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/authorize
  • Token URL: enter the Token URL from the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/token
  • Refresh Token URL: leave empty
  • Redirect URL: leave as the default http://127.0.0.1:7070 value
  • Client ID: enter the App ID from the Portal item’s App Registration information (see earlier steps)
  • Client Secret: enter the App Secret from the Portal item’s App Registration information (see earlier steps)
  • Scope: leave empty
  • API Key: leave empty

That’s it — leave all the rest of the settings at their default values, and click Save.

You can close down the Options dialog now.

Adding the Portal Connection Details

Lastly, we’ll need to setup the Portal connection as an “ArcGISFeatureServer” and “ArcGISMapServer” connection in QGIS. This is done through the QGIS “Data Source Manager” dialog, accessed through the Layer menu. Click the “ArcGIS Feature Server” tab to start with, and then press “New” in the Server Connections group at the top of this dialog.

Enter a descriptive name for the connection, and then enter the URL for the ArcGIS server REST endpoint associated with your Portal:

Lastly, select the new OAuth2 configuration you just created under the “Authentication” group:

Click OK, and you’re done! When you try to connect to the newly added connection, you’ll automatically be taken to the Portal’s logon screen in order to authenticate with the server. After entering your details, you’ll then be connected securely to the server and will have access to all items which are shared with your user account on the Portal!

You can then repeat this step for and create a similar connection under the “ArcGIS Map Server” tab.

We’ve regularly use this setup for our enterprise clients, and have found it to work flawlessly in recent QGIS versions! If you’ve found this useful and are interested in other “best-practice” recommendations for mixed Open-Source and ESRI workplaces, don’t hesitate to contact us to discuss your requirements… at North Road we specialise in ensuring flawless integration between ESRI based systems and the Open Source geospatial software stack.

QGIS Processing, Model Designer and ETL Campaign crowdfund launched!

QGIS Processing offers a rich and expandable set of algorithms which can operate on spatial data, along with a powerful Model Designer which allows users to string together these algorithms to create custom workflows.

Since its introduction in QGIS 2, the Processing framework has seen an intensive amount of development and optimisation efforts. In recent QGIS releases it offers a very user-friendly way of performing complex spatial data processing tasks, all without requiring ANY expensive third-party tools or software licenses!

At North Road we are passionate about the QGIS Processing framework, and have invested considerable effort in this framework over the past 5 years. We’re proud to announce that our latest crowd-funding campaign is focused on further expanding the capabilities and flexibility of Processing and the Processing Model Designer!

Unlike a typical crowdfunding campaign, where a specific funding target and deadline is set, we’re running this campaign a little differently. Instead, this campaign is taking the form of a “à la carte” menu of Processing enhancements. These range from small “paper-cut” style fixes, through to larger architectural improvements, and are each individually priced accordingly. We are asking backers to pick individual enhancements from this “menu of enhancements” and fund that enhancement’s development in full. In order to make this campaign affordable for a wide range of backers, we’ve included a huge range of enhancements which vary in price from smaller amounts to larger amounts.

You can read the full details of the campaign and browse the list of proposed enhancements at the campaign page.

QGIS Print Layouts Graphs and Charts Campaign – Complete!

Last week saw the exciting release of version 3 of the QGIS DataPlotly plugin, which incorporates all the work done as a result of our Print Layouts Graphs and Charts crowdfunding campaign crowd funding campaign. Now, beautiful charts and graphs are available directly within QGIS print layouts, and all it takes is the easy installation of the “DataPlotly” plugin from your QGIS install!

In this post we’ll showcase the functionality which has been added during this campaign, and which is available today in the plugin.

UI modernisation and tweaks

First up, during our work on this plugin we’ve invested some time in refreshing the plugin’s UI to ensure it follows all the widget conventions used elsewhere in QGIS. Now, the plugin blends seamlessly into your QGIS window, and all the chart setting widgets behave in just the way you’re used to. We’ve also used this opportunity to fix a number of issues the plugin had when running on hi-dpi displays (such as Apple retina displays)!

 

Dynamic charting of selected and visible features

If you’ve used earlier versions of the DataPlotly plugin, you’ll likely notice that there’s many new options shown in the screenshot above. Possibly the most exciting of these is the new “Use only selected features” and “Use only visible features” checkboxes. When either of these options are enabled, then your chart will immediately respond to changes in layer selections or changes to the visible map canvas extent respectively. Previously, interactivity in the plugin only went one way (from the chart to the canvas) – but now the charts are truly interactive, and respond dynamically to changes in the canvas too!

 

Improved handling of “data defined” settings

During the plugin refactoring, we reworked how “data defined” settings are handled within charts. If you’re not familiar with these, “data defined” settings are QGIS’ approach for exposing per-feature control over the map rendering process. In DataPlotly charts, we expose this functionality to allow per-feature control over the chart appearance (e.g., showing different scatter plot dot colors based on feature attributes). The new code uses the same code model as QGIS itself, so data-defined settings in your charts now have full access to the whole suite of QGIS expression functions and variables that you’re used to! Additionally, QGIS data-defined assistants are fully supported in the charts too. Ultimately, this enables some very advanced styling options, such as charts which dynamically change color and appearance on every page of your print atlas…

Charts in print layouts

We’ve previously covered this feature in depth, but the DataPlotly v3 release brings print-layout based charts to the masses! When a chart is inserted into a print layout, some additional options are available for controlling the plot behavior:

These new options allow you to link the chart to a map item within the layout, which lets you filter the content of the chart to only include features visible within the map. If your print layout is setup as an atlas export, you can also filter out included features to only show those which are geographically located inside the current atlas feature.

Our partners from Faunalia demonstrate this in the screencast below:

 

Saving plot configuration

An often-requested functionality previous missing from the plugin was the ability to save and restore plot configuration. Now, plot configuration is automatically saved within your QGIS project and restored when you reopen the project. You no longer have to re-create all your charts from scratch at every session (ouch!). We also added the ability to export chart configuration to XML files, allowing you to share and reuse chart configuration across projects.

Behind the scenes work

Aside from all the wonderful new features added to the plugin, we’ve extensively refactored most of the plugin backend. Unit tests and CI infrastructure have been added, which will ensure the plugin remains stable and regression-free in future releases. The code cleanup and simplification has drastically lowered the barrier of contribution to the project, and we’ve already seen new contributors adding more new features to the plugin as a result of this! (Kudos to Simon Gröchenig, who added the new “Feature subset” expression option you can see in the above screenshots!).

Project sponsors

All this work is thanks to the backers of our crowdfunding campaign. Without their contributions this work would not have been possible! In no particular order, our thanks go out to:

  • Federico Gianoli
  • Papercraft Mountains
  • Liam McCrae
  • Henry Walshaw
  • Raúl Sangonzalo
  • Ferdinando Urbano
  • pitsch-ing.ch
  • Carbon-X
  • Gabriel Diosan
  • Rene Giovanni Borella
  • Enrico Bertonati
  • Guido Ingwer
  • David Addy
  • Gerd Jünger
  • Andreas Neumann
  • Stefano Campus
  • Michael Jabot
  • Korto
  • Enrico Ferreguti
  • Carlo A. Nicolini
  • Salvatore Fiandaca
  • Alberto Grava
  • Hans van der Kwast
  • Ben Hur Pintor
  • Silvio Grosso
  • Nobusuke Iwasaki
  • Alasdair Rae
  • Manori Senanayake
  • Canton de Neuchâtel
  • Matthias Daues
  • Alteri Seculo
  • SunGIS Ltd.
  • Stu Smith
  • Keolis Rennes
  • Gabriel Diosan
  • Aiden Price
  • Giacomo Ponticelli
  • Diane Fritz
  • Gemio Bissolati
  • Claire Birnie
  • Nicolas Roelandt
  • Rocco Pispico
  • Gabriel Bengtsson
  • Birds Eye View
  • Barend Köbben
  • Roberto Marzocchi (GTER)
  • Yoichi Kayama
  • Alessandro Sarretta
  • Luca Angeli
  • Luca Bellani
  • giswelt
  • Stefan Giese
  • Ben Harding
  • Joao Gaspar
  • Romain Lacroix
  • Ryan Cooper
  • Daniele Bonaposta
  • QGIS Swedish User Group
  • Nino Formica
  • Michael Gieding
  • Amedeo Fadini
  • Andrew Hannell
  • Stefano
  • Phil Wyatt
  • Brett Edmond Carlock
  • Transitec

Keep an eye out on the North Road blog for future crowd-funding initiatives. Coming soon: a QGIS Processing grab-bag of ETL modelling improvements!

SLYR ESRI to QGIS compatibility suite – October 2019 update

Recently, staff at North Road have been hard at work on our SLYR “ESRI to QGIS compatiblity suite“, and we thought it’s time to share some of the latest exciting updates with you.

While SLYR begun life as a simple “LYR to QGIS conversion tool”, it quickly matured into a full ArcGIS compatibility suite for QGIS. Aside from its original task of converting ESRI LYR files, SLYR now extends the QGIS interface and adds seamless support for working with all kinds of ArcGIS projects and data files. It’s rapidly becoming a must-have tool for any organisation which uses a mix of ESRI and open source tools, or for any organisation exploring a transition away from ArcGIS to QGIS.

Accordingly, we thought it’s well past time we posted an update detailing the latest functionality and support we’ve added to SLYR over the past couple of months! Let’s dive in…

  • Full support for raster LYR file conversion, including unique value renderers, color map renderers, classified renderers, RGB renderers and stretched color ramp renderers:

    From ArcMap…

    …to QGIS!
  • Support for conversion of fill symbol outlines with complex offsets, decorations and dashed line templates
  • Conversion of 3D marker and simple 3D lines to their 2d equivalent, matching ArcMap’s 2D rendering of these symbol types
  • Beta support for converting map annotations and drawings, including custom text labels and reference scale support
  • Label and annotation callout support*
  • Support for converting bookmarks stored in MXD documents*
  • Support for converting ESRI bookmark “.dat” files via drag and drop to QGIS*
  • Correct conversion of OpenStreetMap and bing maps basemap layers
  • SLYR now presents users with a friendly summary of warnings generated during the LYR or MXD conversion process (e.g. due to settings which can’t be matched in QGIS)
  • Added support for MXD documents generated in very early ArcMap versions
  • We’ve added QGIS Processing algorithms allowing for bulk LYR to QLR and MXD to QGS conversion. Now you can run a batch conversion process of ALL MXD/LYR files held at your organisation in one go!
  • Greatly improved matching of converted symbols to their original ArcGIS appearance, including more support for undocumented ArcGIS symbol rendering behavior
  • Support for conversion of text symbols and label settings stored in .style databases*
  • Directly drag and drop layers and layer groups from ArcMap to QGIS to add them to the current QGIS project (maintaining their ArcGIS symbology and layer settings!)*
  • Directly drag and drop layers from ArcCatalog to QGIS windows to open in QGIS*
  • Support for ESRI MapServer layers

(*requires QGIS 3.10 or later)

Over the remainder of 2019, we’ll be hard at work further improving SLYR’s support for MXD document conversion, and adding support for automatic conversion of ArcMap print layouts to QGIS print layouts.

While SLYR is not currently an open-source tool, we believe strongly in the power of open source software, and accordingly we’ve been using a significant portion of the funds generated from SLYR sales to extend the core QGIS application itself. This has directly resulted in many exciting improvements to QGIS, which will become widely available in the upcoming QGIS 3.10 release. Some of the features directly funded by SLYR sales include:

  • A “Segment Center” placement mode for marker line symbols
  • Reworked bookmark handling in QGIS, with a greatly enhanced workflow and usability, and a stable API for 3rd party plugins and scripts to hook into
  • Improved handling of layer symbology for layers with broken paths
  • Auto repair of all other broken layers with a matching data source whenever a single layer path is fixed in a project
  • Support for managing text formats and label settings in QGIS style libraries, allowing storage and management of label and text format presets
  • A new Processing algorithm “Combine Style Databases“, allowing multiple QGIS style databases to be merged to one
  • Adding a “Save layer styles into GeoPackage” option for the “Package Layers” algorithm
  • New expression functions which return file info, such as file paths and base file names
  • Adding new options to autofill the batch Processing dialog, including adding input files using recursive filter based file searches
  • Coming in QGIS 3.12: A new option to set the color to use when rendering nodata pixels in raster layers
  • Coming in QGIS 3.12: A new “random marker fill” symbol layer type, which fills polygons by placing point markers in random locations

You can read more about our SLYR ESRI to QGIS compatibility tool here, or email [email protected] to discuss licensing arrangements for your organisation! Alternatively, send us an email if you’d like to discuss your organisations approach to open-source GIS and for assistance in making this transition as painless as possible.

QGIS Print Layouts Graphs and Charts — Beta Out Now!

Thanks to the success of our recent QGIS Print Layouts Graphs and Charts crowdfunding campaign, staff at North Road and Faunalia have been busy updating and improving the QGIS “DataPlotly” plugin with the goal of allowing beautiful charts inside your print layouts.

We’re super-excited to announce that the beta release of this new functionality is now available! With this beta installed, you’ll see a new icon in your QGIS Print Layout designer window:

Clicking this button will allow you to draw a new plot onto your print layout, just like you would any other standard layout item (like legends, scalebars, etc). Any print layout chart can be customised by right-clicking the chart and selecting “Item Properties“. This will open a panel with settings like position, size, frame, and other standard options. All the magic happens when you click the “Setup Plot” button inside this panel:

This exposes the vast array of styling and charting options available for use. If you’re an existing user of the DataPlotly QGIS plugin, you’ll recognise that these are the same settings you have available when creating interactive plots alongside the main map canvas. Every setting is now available for use in print layouts!

 

To grab the beta, head over to https://github.com/ghtmtt/DataPlotly/releases/tag/v3.9-beta and download the DataPlotly.zip file. Then, inside QGIS, select the Manage and Install Plugins option from the Plugins menu. Click on the “Install from ZIP” section, and point the dialog at your downloaded DataPlotly.zip file. Click “Install Plugin“, and then restart QGIS for good measure. When QGIS restarts you should see the new chart icon inside the print layout designer.

Note that you’ll need a recent QGIS release for this to work correctly — either QGIS 3.8.3 or 3.4.12. (The print layout functionality may not be compatible with earlier releases, as we’ve had to fix several issues inside QGIS itself to get this feature working as designed!). 

We are actively seeking feedback and user testing on this beta release. If you encounter any issues, just head over to https://github.com/ghtmtt/DataPlotly/issues and let us know.

We’ll be further refining this functionality, with the goal of releasing the final non-beta version of the plugin to coincide with the upcoming QGIS 3.10 release.

Happy charting!

QGIS 3.10 Loves GeoPDF!

Recently, we’ve been working on an exciting development which is coming soon in QGIS 3.10… support for Geospatial PDF exports! This has been a long-desired feature for many QGIS users, and it was only made possible thanks to a group of financial backers (listed below). In this post, we’re going to explore these new features and how they improve your QGIS PDF outputs.

Geospatial PDFs can now be created either by exporting the main QGIS map canvas, or by creating and exporting a custom print layout. For instance, when you select the “Save Map as PDF” option from the main QGIS window, you’ll see a new group of Geospatial PDF related options:

At its most basic, Geospatial PDF is a standard extension to the PDF format which allows for vector spatial datasets to be embedded in PDF files. If the “Include vector feature information” checkbox is ticked when creating a Geospatial PDF output, then QGIS will automatically include all the geometry and attribute information from features visible within the page. So if we export a simple map to PDF, we’ll get an output file which looks just like any old regular PDF map output…

…but, we can also pull this PDF back into QGIS and treat it just like any other vector data source! In the screenshot below we’re using the Identify tool to query on of the polygons and see all the attribute information from the original source layer.

This ability adds a lot of value to PDF exports. Anyone who has ever been supplied a non-spatial PDF as a “spatial dataset” will attest to the frustrations these cause… but if you create proper Geospatial PDFs instead, then there’s no loss of the valuable underlying spatial information or feature attributes! Furthermore, if these PDFs are opened within Acrobat Reader, tools are enabled which allow users to query features interactively.

Another nice benefit which comes with Geospatial PDF output is that layers can be interactively toggled on or off in the PDF viewer. The screenshot below shows a Geospatial PDF file created from a simple QGIS map. On the left we have a list of the layers in the PDF, each of which can be turned on or off inside the PDF viewer!

The really nice thing here is that, thanks to the underlying smarts in the GDAL library which is responsible for the actual Geospatial PDF creation, the PDF renders identically to our original QGIS map. While labels turn on and off alongside their corresponding map layer, they are still correctly stacked in the exact same way as you see in the QGIS window. Furthermore, the created PDFs keep labels and vector features as vector artwork… so there’s absolutely no loss in quality when zooming in to the map! These files look GREAT!

On that same note… the sponsorship allowed us to tackle another related issue, which is that in previous QGIS versions PDF (or SVG) exports would always export every single vertex from any visible feature! Ouch! This meant that if you had a complex polygon boundary layer, you would potentially be creating a PDF with millions of vertices per feature, even though most of these would be overlapping and completely redundant at the exported map’s scale. Now, QGIS automatically simplifies vector features while exporting them (using an appropriate, effectively invisible, level of simplification). The dramatically reduces the created file sizes and speeds up opening them and navigating them in other applications (especially Inkscape). (There’s an option at export time to disable this simplification, if you have a specific reason to!).

Creating Geospatial PDFs from print layouts gives even more options. For a start, whenever a print layout is exported to Geospatial PDFs, we ensure that the created PDF correctly handles stacking of layers alongside any other print layout items you have. In the image below we see a custom print layout which includes interactive layer visibility controls. If a layer is toggled, it’s hidden only from the map item — all the other surrounding elements like the title, north arrow and scalebar remain visible:

That’s not all though! When exporting a print layout to Geospatial PDF, QGIS also hooks into any map themes you’ve setup in your project. If you select to include these themes in your output, then the result is magical! The screenshot below shows the export options for a project with a number of themes, and we’ve chosen to export these themes in the PDF:

Opening the resultant PDF shows that our layer control on the left now lists the map themes instead of individual layers. Viewers can switch between these themes, changing the visibility of layers and their styling to match the QGIS map theme from the project! Additionally, you can even expand out a theme and expose layer-by-layer visibility control. Wow! This means you could create a single PDF output file which includes an environmental, social, cadastral, transport, …. view of your map, all in the one file.

Lastly, there’s even control for fine-tuning the combination of layers which are exposed in the output PDF file and which ones should be toggled on and off together. In the screenshot below we’ve opted to group the “Aircraft” and “Roads” map layers into a single logical PDF group called “Transport”. 

The resultant PDF respects this, showing an entry in the interactive layer tree for “Transport” which toggles both the aircraft and roads layers together:

So there you go — the power of Geospatial PDF, coming your way in QGIS 3.10!

One semi-related benefit of this work is that it gave us an opportunity to rework how “layered” exports from print layouts are created. This has had a significant flow-on impact on the existing ability to create layered SVG outputs from QGIS. Previously, this was a rather fragile feature, which created SVGs with lots of issues – overlapping labels, incorrectly stacked layers, and last-but-not-least, non-descriptive layer names! Now, just like Geospatial PDF exports, the layered SVG exports correctly respect the exact look of your map, and have much more friendly, descriptive layer names:

This should significantly reduce the amount of housekeeping required when working on these layered SVG exports. 

This work was funded by:

  • Land Vorarlberg
  • Municipality of Vienna
  • Municipality of Dornbirn
  • Biodiversity Information Service for Powys and BBNP Local
  • Kanton Zug
  • Canton de Neuchâtel
  • Canton de Thurgovia

North Road are leading experts in extending the QGIS application to meet your needs. If you’d like to discuss how you can sponsor development of features or fixes which you want in QGIS, just contact us for further details!

 

 

QGIS Print Layouts Graphs and Charts — target reached!

We’ve just passed the extended deadline for our recent QGIS Print Layouts Graphs and Charts campaign, and the great news is that thanks to a large number of generous backers we’ve successfully hit the target for this campaign! This has only been possible thanks to the tireless work of the QGIS community and user groups in promoting this campaign and spreading the word.

The Print Layouts Graphs and Charts campaign is a joint effort with our friends at Faunalia, so we’ll soon be starting work together on all the wonderful new functionality heading to the QGIS DataPlotly plugin as a result. The work will be commencing late June, just after the QGIS 3.8.0 final release. Keep an eye out for further updates on the development from this time! You can read more about what’s coming in detail at the campaign page.

We’d like to take this opportunity to extend our heartfelt thanks to all the backers who have pledged to support this project:

  • Federico Gianoli
  • Papercraft Mountains
  • Liam McCrae
  • Henry Walshaw
  • Raúl Sangonzalo
  • Ferdinando Urbano
  • pitsch-ing.ch
  • Carbon-X
  • Gabriel Diosan
  • Rene Giovanni Borella
  • Enrico Bertonati
  • Guido Ingwer
  • David Addy
  • Gerd Jünger
  • Andreas Neumann
  • Stefano Campus
  • Michael Jabot
  • Korto
  • Enrico Ferreguti
  • Carlo A. Nicolini
  • Salvatore Fiandaca
  • Alberto Grava
  • Hans van der Kwast
  • Ben Hur Pintor
  • Silvio Grosso
  • Nobusuke Iwasaki
  • Alasdair Rae
  • Manori Senanayake
  • Canton de Neuchâtel
  • Matthias Daues
  • Alteri Seculo
  • SunGIS Ltd.
  • Stu Smith
  • Keolis Rennes
  • Gabriel Diosan
  • Aiden Price
  • Giacomo Ponticelli
  • Diane Fritz
  • Gemio Bissolati
  • Claire Birnie
  • Nicolas Roelandt
  • Rocco Pispico
  • Gabriel Bengtsson
  • Birds Eye View
  • Barend Köbben
  • Roberto Marzocchi (GTER)
  • Yoichi Kayama
  • Alessandro Sarretta
  • Luca Angeli
  • Luca Bellani
  • giswelt
  • Stefan Giese
  • Ben Harding
  • Joao Gaspar
  • Romain Lacroix
  • Ryan Cooper
  • Daniele Bonaposta
  • QGIS Swedish User Group
  • Nino Formica
  • Michael Gieding
  • Amedeo Fadini
  • Andrew Hannell
  • Stefano
  • Phil Wyatt
  • Brett Edmond Carlock
  • Transitec

 

QGIS Print Layouts Graphs and Charts – campaign deadline extended!

If you’re a regular reader of this blog, it won’t surprise you to hear that we’re very excited about adding rich charting and graph functionality to QGIS’ Print Layout designer! Alongside our friends at Faunalia, we’re currently running a crowd funding campaign to make this a reality.

So, while the required funds weren’t raised within our original April 30 deadline, we’ve decided to extend this campaign by an additional 30 days in the hopes that the users and organisations from the wider QGIS community will jump onboard and pledge the remaining required funds.

This missing feature is a large gap in QGIS printing capabilities, so we’re counting on you to show your support and spread the word to your local user groups, QGIS users, and any organisations you know of who rely on QGIS and would love to see its inbuilt reporting capabilities levelled up!

QGIS (and SLYR!), now with Hash Lines support

Thanks to an anonymous corporate sponsor, we’ve recently had the opportunity to add a new Hashed Line symbol type for QGIS 3.8. This allows for a repeating line segment to be drawn over the length of a feature, with a line-sub symbol used to render each individual segment.

There’s tons of options available for customising the appearance and placement of line hashes. We based the feature heavily off QGIS’ existing “Marker Line” support, so you can create hashed lines placed at set intervals, on line vertices, or at the start/end/middle of lines. There’s options to offset the lines, and tweak the rotation angle of individual hashes too. Added to QGIS’ rich support for “data defined” symbol properties, this allows for a huge range of new symbol effects.

E.g. using a data defined hash length which increases over the length of a feature gives us this effect:

Or, when using a complex line sub-symbol for rendering each hash, we can get something like this:

Or even go completely “meta” and use a hashed line sub symbol for the hash line itself!

With the right combination of symbol settings and QGIS draw effects you can even emulate a calligraphic pen effect:

Or a chunky green highlighter!

This same corporate sponsor also funded a change which results in a huge improvement to the appearance of both rotated hashed lines and marker lines. Previously, when marker or hash lines were rendered, the symbol angles were determined by taking the exact line orientation at the position of the symbol. This often leads to undesirable rendering effects, where little “bumps” or corners in lines which occur at the position of the symbol cause the marker or hash line to be oriented at a very different angle to what the eye expects to see.

With this improvement, the angle is instead calculated by averaging the line over a specified distance either side of the symbol. E.g. averaging the line angle over 4mm means we take the points along the line 2mm from either side of the symbol placement, and use these instead to calculate the line angle for that symbol. This has the effect of smoothing (or removing) any tiny local deviations from the overall line direction, resulting in much nicer visual orientation of marker or hash lines.

It’s easiest to show the difference visually. Here’s a before image, showing arrow markers following a line feature. Pay specific attention to the 3rd and last arrow, which seem completely random oriented due to the little shifts in line direction:

With new smoothing logic this is much improved:

The difference is even more noticeable for hashed lines. Here’s the before:

…and the after:

Suffice to say, cartographers will definitely appreciate the result!

Lastly, we’ve taken this new hash line feature as an opportunity to implement automatic conversion of ESRI hash line symbols within our SLYR ESRI to QGIS conversion tool. Read more about SLYR here, and how you can purchase this tool for .style, .lyr and .mxd document conversion.

QGIS Print Layouts Graphs and Charts crowdfund launched!

Ever wished QGIS had a way to insert dynamic, feature rich charts and graphs directly inside print layouts? If so, our latest crowdfunding campaign has you covered! This missing feature is a large gap in QGIS printing capabilities, so we’re planning on filling that gap by exposing the powerful QGIS “Data Plotly” plugin to allow these charts to be embedded inside your layouts, and allow them to be created and modified in a simple, interactive style.

If you’re not aware of the existing capabilities of the DataPlotly plugin, here’s a quick screencast which should get you excited about the possibilities here…

QGIS is already a reporting powerhouse, and we believe that linking DataPlotly with QGIS print layouts will boost the current functionality up an order of magnitude! To make it possible we need 8600€ pledged before 30 April 2019. North Road is collaborating on this campaign with our friends at Faunalia, and development work will be shared between the two consultancy firms.

You can help make this a reality by supporting the campaign or by sharing the page and increasing exposure to the campaign. Full details about the planned functionality and how to contribute are available at the campaign page.

Announcing our SLYR (MXD to QGIS) funding drive!

One product which North Road had the chance to develop last year, and which we are super-proud of, is our SLYR ESRI style to QGIS conversion tool. If you haven’t heard of it before, this tool allows automatic conversion of ESRI .style database files to their equivalent QGIS symbology equivalent. It works well for the most part, and now we’re keen to take this to the next stage.

The good news is that North Road have been conducting extensive research and development over the past 12 months, and we’re pleased to announce our plans for extending SLYR to support ESRI LYR and MXD documents. The LYR and MXD formats are proprietary ESRI-only formats, with no public specifications allowing their use. This is a huge issue for organisations who want to move from an ESRI environment to the open geospatial world, yet are held back by hundreds (or thousands!) of existing ESRI MXD map documents and layer styles which they currently cannot utilise outside of the ESRI software ecosystem. Furthermore, many providers of spatial data only include ESRI specific layer formatting files with their data supplies. This leaves users with no means of utilising these official, pre-defined styles in non-ESRI tools.

In order for us to continue development of the SLYR tool and unlock use of LYR and MXD formats outside of ESRI tools, we are conducting a funding campaign. Sponsors of the campaign will receive access to the tools as they are developed and gain access to official support channels covering their use. At the conclusion of this drive we’ll be releasing all the tools and specifications under a free, open-source license.

You can read the full details of the campaign here, including pricing to become a project sponsor and gain access to the tools as they develop. As a campaign launch promo, we’re offering the first 10 sponsors a super-special discounted rate (as a reward for jumping on the development early).

The mockup below shows what the end goal is: seamless, fully integrated, automatic conversion of MXD and LYR files directly within the QGIS desktop application!

On custom layout checks in QGIS 3.6, and how they can do your work for you!

Recently, we had the opportunity to implement an exciting new feature within QGIS. An enterprise with a large number of QGIS installs was looking for a way to control the outputs which staff were creating from the software, and enforce a set of predefined policies. The policies were designed to ensure that maps created in QGIS’ print layout designer would meet a set of minimum standards, e.g.:

  • Layouts must include a “Copyright 2019 by XXX” label somewhere on the page
  • All maps must have a linked scale bar
  • No layers from certain blacklisted sources (e.g. Google Maps tiles) are permitted
  • Required attribution text for other layers must be included somewhere on the layout

Instead of just making a set of written policies and hoping that staff correctly follow them, it was instead decided that the checks should be performed automatically by QGIS itself. If any of the checks failed (indicating that the map wasn’t complying to the policies), the layout export would be blocked and the user would be advised what they needed to change in their map to make it compliant.

The result of this work is a brand new API for implementing custom “validity checks” within QGIS. Out of the box, QGIS 3.6 ships with two in-built validity checks. These are:

  • A check to warn users when a layout includes a scale bar which isn’t linked to a map
  • A check to warn users if a map overview in a layout isn’t linked to a map (e.g. if the linked map has been deleted)

All QGIS 3.6 users will see a friendly warning if either of these conditions are met, advising them of the potential issue.

 

The exciting stuff comes in custom, in-house checks. These are written in PyQGIS, so they can be deployed through in-house plugins or startup scripts. Let’s explore some examples to see how these work.

A basic check looks something like this:

from qgis.core import check

@check.register(type=QgsAbstractValidityCheck.TypeLayoutCheck)
def my_layout_check(context, feedback):
  results = ...
  return results

Checks are created using the @check.register decorator. This takes a single argument, the check type. For now, only layout checks are implemented, so this should be set to QgsAbstractValidityCheck.TypeLayoutCheck. The check function is given two arguments, a QgsValidityCheckContext argument, and a feedback argument. We can safely ignore the feedback option for now, but the context argument is important. This context contains information useful for the check to run — in the case of layout checks, the context contains a reference to the layout being checked. The check function should return a list of QgsValidityCheckResult objects, or an empty list if the check was passed successfully with no warnings or errors.

Here’s a more complete example. This one throws a warning whenever a layout map item is set to the web mercator (EPSG:3875) projection:

@check.register(type=QgsAbstractValidityCheck.TypeLayoutCheck)
def layout_map_crs_choice_check(context, feedback):
  layout = context.layout
  results = []
  for i in layout.items():
    if isinstance(i, QgsLayoutItemMap) and i.crs().authid() == 'EPSG:3857':
      res = QgsValidityCheckResult()
      res.type = QgsValidityCheckResult.Warning
      res.title='Map projection is misleading'
      res.detailedDescription='The projection for the map item {} is set to Web Mercator (EPSG:3857) which misrepresents areas and shapes. Consider using an appropriate local projection instead.'.format(i.displayName())
      results.append(res)

  return results

Here, our check loops through all the items in the layout being tested, looking for QgsLayoutItemMap instances. It then checks the CRS for each map, and if that CRS is ‘EPSG:3857’, a warning result is returned. The warning includes a friendly message for users advising them why the check failed.

In this example our check is returning results with a QgsValidityCheckResult.Warning type. Warning results are shown to users, but they don’t prevent users from proceeding and continuing to export their layout.

Checks can also return “critical” results. If any critical results are obtained, then the actual export itself is blocked. The user is still shown the messages generated by the check so that they know how to resolve the issue, but they can’t proceed with the export until they’ve fixed their layout. Here’s an example of a check which returns critical results, preventing layout export if there’s no “Copyright 2019 North Road” labels included on their layout:

@check.register(type=QgsAbstractValidityCheck.TypeLayoutCheck)
def layout_map_crs_choice_check(context, feedback):
  layout = context.layout
  for i in layout.items():
    if isinstance(i, QgsLayoutItemLabel) and 'Copyright 2019 North Road' in i.currentText():
      return

  # did not find copyright text, block layout export
  res = QgsValidityCheckResult()
  res.type = QgsValidityCheckResult.Critical
  res.title = 'Missing copyright label'
  res.detailedDescription = 'Layout has no "Copyright" label. Please add a label containing the text "Copyright 2019 North Road".'
  return [res]

If we try to export a layout with the copyright notice, we now get this error:

Notice how the OK button is disabled, and users are forced to fix the error before they can export their layouts.

Here’s a final example. This one runs through all the layers included within maps in the layout, and if any of them come from a “blacklisted” source, the user is not permitted to proceed with the export:

@check.register(type=QgsAbstractValidityCheck.TypeLayoutCheck)
def layout_map_crs_choice_check(context, feedback):
  layout = context.layout
  for i in layout.items():
    if isinstance(i, QgsLayoutItemMap):
      for l in i.layersToRender():
        # check if layer source is blacklisted
        if 'mt1.google.com' in l.source():
          res = QgsValidityCheckResult()
          res.type = QgsValidityCheckResult.Critical
          res.title = 'Blacklisted layer source'
          res.detailedDescription = 'This layout includes a Google maps layer ("{}"), which is in violation of their Terms of Service.'.format(l.name())
          return [res]

Of course, all checks are run each time — so if a layout fails multiple checks, the user will see a summary of ALL failed checks, and can click on each in turn to see the detailed description of the failure.

So there we go — when QGIS 3.6 is released in late February 2019, you’ll  have access to this API and can start making QGIS automatically enforce your organisation policies for you! The really neat thing is that this doesn’t only apply to large organisations. Even if you’re a one-person shop using QGIS, you could write your own checks to  make QGIS “remind” you when you’ve forgotten to include something in your products. It’d even be possible to hook into one of the available Python spell checking libraries to write a spelling check! With any luck, this should lead to better quality outputs and less back and forth with your clients.

North Road are leading experts in customising the QGIS application for enterprise installs. If you’d like to discuss how you can deploy in-house customisation like this within your organisation, contact us for further details!

Edit Features “In Place” crowdfund — made it to QGIS 3.4!

Well, thanks to the resounding success of our QGIS edit-in-place crowdfunding campaign, we’ve been frantically smashing away at our keyboards in an attempt to reward the QGIS community by sneaking this feature in a whole 4 months earlier than originally promised! And, we’re very proud to announce, that this exciting new feature has been implemented and will be included in the upcoming QGIS 3.4 release (due late October 2018). So go ahead — grab one of the nightly pre-release of QGIS 3.4 and checkout the results.

This wouldn’t have been possible without the rapid response to the campaign and the generosity of our wonderful backers:

(In addition to these backers, we’ve also received numerous anonymous donations to this feature from many other individuals — while we can’t list you all publicly, you’re also in our thanks!)

 

Keep an eye on this blog for other upcoming QGIS crowdfunding campaigns targeted at QGIS 3.6 and beyond… we’ve got lots more exciting work planned for these releases!

 

Edit Features “In Place” crowdfund — target reached!

Well, the final pledges have been tallied and we’re very proud to announce that our latest crowd funding campaign has been a roaring success!

We’ve been completely blown away by the response to this campaign. Thanks to some incredibly generous backers and donors, we’ve been able to hit the campaign target with plenty of time to spare. As a result, we’ll be pushing hard to reward the generosity of the community by trying to sneak this feature in for the upcoming QGIS 3.4 release (instead of the originally promised 3.6 release)! You can read more about what we’re adding at the campaign page.

We’d like to take this opportunity to extend our heartfelt thanks to all the backers who have pledged to support this project:

In addition to these backers, we’ve also received numerous anonymous donations to this feature from many other individuals — while we can’t list you all publicly, you’re also in our thanks!

Stay tuned for more updates to come as work proceeds on this feature…

Back to Top

Sustaining Members