New release for QField : 3.4 “Ebo”
Oslandia is the main partner of OPENGIS.ch around QField. We are proud today to forward the announcement of the new QField release 3.4 “Ebo”.
Main highlights
A new geofencing framework has landed, enabling users to configure QField behaviors in relation to geofenced areas and user positioning. Geofenced areas are defined at the project-level and shaped by polygons from a chosen vector layer. The three available geofencing behaviours in this new release are:
- Alert user when inside an area polygon;
- Alert user when outside all defined area polygons and
- Inform the user when entering and leaving an area polygons.
In addition to being alerted or informed, users can also prevent digitizing of features when being alerted by the first or second behaviour. The configuration of this functionality is done in QGIS using QFieldSync.
Pro tip: geofencing settings are embedded within projects, which means it is easy to deploy these constraints to a team of field workers through QFieldCloud. Thanks Terrex Seismic for sponsoring this functionality.
QField now offers users access to a brand new processing toolbox containing over a dozen algorithms for manipulating digitized geometries directly in the field. As with many parts of QField, this feature relies on QGIS’ core library, namely its processing framework and the numerous, well-maintained algorithms it comes with.
The algorithms exposed in QField unlock many useful functionalities for refining geometries, including orthogonalization, smoothing, buffering, rotation, affine transformation, etc. As users configure algorithms’ parameters, a grey preview of the output will be visible as an overlay on top of the map canvas.
To reach the processing toolbox in QField, select one or more features by long-pressing on them in the features list, open the 3-dot menu and click on the process selected feature(s) action. Are you excited about this one? Send your thanks to the National Land Survey of Finland, who’s support made this a reality.
QField’s camera has gained support for customized ratio and resolution of photos, as well as the ability to stamp details – date and time as well as location details – onto captured photos. In fact, QField’s own camera has received so much attention in the last few releases that it was decided to make it the default one. On supported platforms, users can switch to their OS camera by disabling the native camera option found at the bottom of the QField settings’ general tab.
Wait, there’s more
There are plenty more improvements packed into this release from project variables editing using a revamped variables editor through to integration of QField documentation help in the search bar and the ability to search cloud project lists. Read the full 3.4 changelog to know more, and enjoy the release!
Contact us !
A question concerning QField ? Interested in QField deployment ? Do not hesitate to contact Oslandia to discuss your project !
New release for QField : 3.3 “Darién”
Oslandia is the main partner of OPENGIS.ch around QField. We are proud today to forward the announcement of the new QField release 3.3 “Darién”. This release introduces a brand new plugin framework that empowers users to customize and add completely new functionalities to their favourite field application.
The plugin framework comes with other new features and improvements for this release, detailed below.
Main highlights
One of the biggest feature additions of this version is a brand new drawing tool that allows users to sketch out important details over captured photos or annotate drawing templates. This was a highly requested feature, which is brought to all supported platforms (Android, iOS, Windows, macOS, and, of course, Linux) with the financial support of the Swiss QGIS user group.
Also landing in this version is support for copying and pasting vector features into and from the clipboard. This comes in handy in multiple ways, from providing a quick and easy way to transfer attributes from one feature to another through matching field names to pasting the details of a captured feature in the field into a third-party messenger, word editing, or email application. Copying and pasting features can be done through the feature form’s menu as well as long pressed over the map canvas. Moreover, a new feature-to-feature attributes transfer shortcut has also been added to the feature form’s menu. Appreciation to Switzerland, Canton of Lucerne, Environment and Energy for providing the funds for this feature.
The feature form continues to gain more functionalities; in this version, the feature form’s value map editor widget has gained a new toggle button interface that can help fasten data entry. The interface replaces the traditional combo box with a series of toggle buttons, lowering the number of taps required to pick a value. The German Archaeological Institut – KulturGutRetter sponsored this feature.
Other improvements in the feature form include support for value relation item grouping and respect for the vector layer attributes’ « reuse last entered value » setting.
Finally, additional features include support for image decoration overlay, a new interface to hop through cameras (front, back, and external devices) for the ‘non-native’ camera, the possibility to disable the 3-finger map rotation gesture, and much more.
User experience improvements
Long-time users of QField will notice the new version restyling of the information panels such as GNSS positioning, navigation, elevation profile, and sensor data. The information is now presented as an overlay sitting on top of the map canvas, which increases the map canvas’ visibility while also achieving better focus and clarity on the provided details. With this new version, all details, including altitude and distance to destination, respect user-configured project distance unit type.
The dashboard’s legend has also received some attention. You can now toggle the visibility of any layer via a quick tap on a new eye icon sitting in the legend tree itself. Similarly, legend groups can be expanded and collapsed directly for the tree. This also permits you to show or hide layers while digitizing a feature, something which was not possible until now. The development of these improvements was supported by Gispo and sponsored by the National Land Survey of Finland.
Plugin framework
QField 3.3 introduces a brand new plugin framework using Qt’s powerful QML and JavaScript engine. With a few lines of code, plugins can be written to tweak QField’s behaviour and add new capabilities. Two types of plugins are possible: app-wide plugins as well as project-scoped plugins. To ensure maximum ease of deployment, plugin distribution has been made possible through QFieldCloud! Amsa provided the financial contribution that brought this project to life.
Our partner OPENGIS.ch will soon offer a webinar to discover how QField plugins can help your field (and business) workflows by allowing you to be even more efficient in the field.
Users interested in authoring plugins or better understanding the framework, can already visit the dedicated documentation page and a sample plugin implementation sporting a weather forecast integration.
A question concerning QField ? Interested in QField deployment ? Do not hesitate to contact Oslandia to discuss your project !
Topography and Topology in and around QGIS
Since 2018 and the arrival of Loïc Bartoletti, Oslandia has accelerated its focus on topography and topology within and around QGIS.
Two questions have driven this focus:
- How to draw directly in GIS by integrating drawing tools inspired by the CAD world into QGIS.
- How to integrate plugins for topographic calculations directly into QGIS.
To address this, Oslandia has worked on several fronts: training, developing open-source plugins, and improvements in QGIS.
1- Plugins
The following plugins were developed by Oslandia or partners, with contributions from Oslandia:
- landsurveycodesimport : https://gitlab.com/Oslandia/qgis/landsurveycodesimport
A plugin for surveyors working with the point coding methodology in topography. - QompliGIS : https://gitlab.com/Oslandia/qgis/qompligis
A tool to ensure your dataset or plan complies with a specification. - Total Open Station : https://github.com/totalopenstation
An open-source project initiated by Italian and French archaeologists, it is a tool to convert various formats of survey field books. - Topaze : https://gitlab.com/azimut-fr/qgis/topaze
A topometric calculation tool developed with Jean-Marie Arsac (Azimut). It was a proof of concept that such calculations can be done within a GIS.
These plugins can be used at different stages of a project. They can be used all together or only those needed and integrated into workflows.
2- Improvements on QGIS
Oslandia also focuses on improving the core of QGIS. Last years, our teams have worked on:
— Integration of shape tools: circles, ellipses, rectangles, regular polygons, etc.
— Improvement of snapping tools.
— Enhancement of Z and M coordinate support.
— Improvement of topological tools (relationships between geometries).
Coming soon is the possibility to use geometry and topology validation and correction plugins directly in QGIS processing tools, developed by Jacky Volpès and Loïc Bartoletti.
3- Training
Oslandia is QUALIOPI certified and offers a training program around QGIS and QField:
- Drawing: Transition from CAD to GIS with QGIS: https://oslandia.com/formations/qgis3-dessiner-avec-qgis/
- Topography with QGIS (LSCI, Topaze, QompliGIS, Total Open Station)
- QField Training
- Deployment of QField Cloud
« In 2023, 89 people were trained by Oslandia, who recommend our training at 90.9%.»
4- And QField ?
Since our partnership with OPENGIS.ch, Oslandia offers QField Cloud server deployment services, training, and QField support.
5- Coming Soon!
Several technical posts are being prepared: how to open CAD files in a GIS? What are the differences between QField and LSCI? You will find them on our website in the coming weeks. 
Additionally, we are preparing a white paper on the topic of migrating from CAD to QGIS, which we should release in September.
Stay tuned!
Software quality in QGIS
According to the definition of software quality given by french Wikipedia
An overall assessment of quality takes into account external factors, directly observable by the user, as well as internal factors, observable by engineers during code reviews or maintenance work.
I have chosen in this article to only talk about the latter. The quality of software and more precisely QGIS is therefore not limited to what is described here. There is still much to say about:
- Taking user feedback into account,
- the documentation writing process,
- translation management,
- interoperability through the implementation of standards,
- the extensibility using API,
- the reversibility and resilience of the open source model…
These are subjects that we care a lot and deserve their own article.
I will focus here on the following issue: QGIS is free software and allows anyone with the necessary skills to modify the software. But how can we ensure that the multiple proposals for modifications to the software contribute to its improvement and do not harm its future maintenance?
Self-discipline
All developers contributing to QGIS code doesn’t belong to the same organization. They don’t all live in the same country, don’t necessarily have the same culture and don’t necessarily share the same interests or ambitions for the software. However, they share the awareness of modifying a common good and the desire to take care of it.
This awareness transcends professional awareness, the developer not only has a responsibility towards his employer, but also towards the entire community of users and contributors to the software.
This self-discipline is the foundation of the quality of the contributions of software like QGIS.
However, to err is human and it is essential to carry out checks for each modification proposal.
Automatic checks
With each modification proposal (called Pull Request or Merge Request), the QGIS GitHub platform automatically launches a set of automatic checks.
Example of proposed modification
Result of automatic checks on a modification proposal
The first of these checks is to build QGIS on the different systems on which it is distributed (Linux, Windows, MacOS) by integrating the proposed modification. It is inconceivable to integrate a modification that would prevent the application from being built on one of these systems.
The tests
The first problem posed by a proposed modification is the following “How can we be sure that what is going to be introduced does not break what already exists?”
To validate this assertion, we rely on automatic tests. This is a set of micro-programs called tests, which only purpose is to validate that part of the application behaves as expected. For example, there is a test which validates that when the user adds an entry in a data layer, then this entry is then present in the data layer. If a modification were to break this behavior, then the test would fail and the proposal would be rejected (or more likely corrected).
This makes it possible in particular to avoid regressions (they are very often called non-regression tests) and also to qualify the expected behavior.
There are approximately 1.3 Million lines of code for the QGIS application and 420K lines of test code, a ratio of 1 to 3. The presence of tests is mandatory for adding functionality, therefore the quantity of test code increases with the quantity of application code.
In blue the number of lines of code in QGIS, in red the number of lines of tests
There are currently over 900 groups of automatic tests in QGIS, most of which run in less than 2 seconds, for a total execution time of around 30 minutes.
We also see that certain parts of the QGIS code – the most recent – are better covered by the tests than other older ones. Developers are gradually working to improve this situation to reduce technical debt.
Code checks
Analogous to using a spell checker when writing a document, we carry out a set of quality checks on the source code. We check, for example, that the proposed modification does not contain misspelled words or “banned” words, that the API documentation has been correctly written or that the modified code respects certain formal rules of the programming language.
We recently had the opportunity to add a check based on the clang-tidy tool. The latter relies on the Clang compiler. It is capable of detecting programming errors by carrying out a static analysis of the code.
Clang-tidy is, for example, capable of detecting “narrowing conversions”.
Example of detecting “narrowing conversions”
In the example above, Clang-tidy detects that there has been a “narrowing conversion” and that the value of the port used in the network proxy configuration “may” be corrupted. In this case, this problem was reported on the QGIS issues platform and had to be corrected.
At that time, clang-tidy was not in place. Its use would have made it possible to avoid this anomaly and all the steps which led to its correction (exhaustive description of the issue, multiple exchanges to be able to reproduce it, investigation, correction, review of the modification), meaning a significant amount of human time which could thus have been avoided.
Peer review
A proposed modification that would validate all of the automatic checks described above would not necessarily be integrated into the QGIS code automatically. In fact, its code may be poorly designed or the modification poorly thought out. The relevance of the functionality may be doubtful, or duplicated with another. The integration of the modification would therefore potentially cause a burden for the people in charge of the corrective or evolutionary maintenance of the software.
It is therefore essential to include a human review in the process of accepting a modification.
This is more of a rereading of the substance of the proposal than of the form. For the latter, we favor the automatic checks described above in order to simplify the review process.
Therefore, human proofreading takes time, and this effort is growing with the quantity of modifications proposed in the QGIS code. The question of its funding arises, and discussions are in progress. The QGIS.org association notably dedicates a significant part of its budget to fund code reviews.
More than 100 modification proposals were reviewed and integrated during the month of December 2023. More than 30 different people contributed. More than 2000 files have been modified.
Therefore the wait for a proofreading can sometimes be long. It is also often the moment when disagreements are expressed. It is therefore a phase which can prove frustrating for contributors, but it is an important and rich moment in the community life of a free project.
To be continued !
As a core QGIS developer, and as a pure player OpenSource company, we believe it is fundamental to be involved in each step of the contribution process.
We are investing in the review process, improving automatic checks, and in the QGIS quality process in general. And we will continue to invest in these topics in order to help make QGIS a long-lasting and stable software.
If you would like to contribute or simply learn more about QGIS, do not hesitate to contact us at [email protected] and consult our QGIS support proposal.
QGIS Snapping improvements
A few months ago, we proposed to the QGIS grant program to make improvements to the snap cache in QGIS. The community vote selected our project which was funded by QGIS.org. Developments are now mostly finished.
In short, snapping is crucial for editing geospatial features. It is the only way to ensuring they are topologically related, ie, connected vertices have exactly the same coordinates even if manual digitizing on screen is imprecise by nature. Snapping correctly supposes QGIS have in memory an indexed cache of the geometries to snap to. And maintainting this cache when data is modified, sometimes by another user or database logic, can be a real challenge. This it exactly what this work adresses.
The proposal was divided into two different tasks:
- Manage circular dependencies
- Relax the snap cache index build
Manage cicular data dependencies
Data dependencies
Data dependency is an existing feature that allows you to configure QGIS to reload layers (and their snapping cache) when a layer is modified.
It is useful when you store your data in a database and you set up triggers to maintain consistency between the different tables of your data model.
For instance, say you have topological informations containing lines and nodes. Nodes are part of lines and lines go through nodes. Then, you move a node in QGIS, and save your modifications to the database. In order to keep the data consistent, a trigger updates the geometry of the line going through the modified node.
Node 2 is modified, Line 1 is updated accordingly
QGIS, as a database client, has no information that the line layer currently displayed in the canvas needs to be refreshed after the trigger. Although the map canvas will be up to date, because QGIS fetches data for display without any caching system, the snapping cache is not and you’ll end up with ghost snapping highlights issues.
Snapping highlights (light red) differ from real line (orange)
Defining a dependency between nodes and lines layers tells QGIS that it has to refresh the line layer when a node is modified.
Dependencies configuration: Lines layer will be refreshed whenever Nodes layer is modified
It also have to work the other way, modifying a line should update the nodes to ensure they still are on the line.
Circular data dependencies
So here we are, lines depend on nodes which depend on lines which depend on nodes which…
That’s what circular dependencies is about. This specific behavior was previously forbidden and needed a special way to deal with it. Thanks to this recent development, it is now possible.
It’s also possible to add the layer itself as one of its own dependencies. It helps dealing with specific cases where one feature modification could lead to a modification of another feature in the same layer (to keep consistency on road networks for instance).
Road 2 is modified, Road 1 is updated accordingly
This feature is available in the next QGIS LTR version 3.10.
Relax the snapping cache index build
If you work in QGIS with huge projects displaying a lot of vector data, and you enable snapping while editing these data, you probably already met this dialog:
Snap indexing dialog
This dialog informs you that data are currently being indexed so you can snap on them while you will edit feature geometry. And for big projects, this dialog can last for a really long time. Let’s work on speeding it up!
What’s a snap index?
Let’s say you want to move a line and snap it onto another one. While you drag your line with the mouse, QGIS will look for an existing geometry beneath the mouse cursor (with a certain pixel tolerance) every time you move your mouse. Without spatial index, QGIS will have to go through every geometry in your layer to check if the given geometry is beneath the cursor position. This would be very ineffective.
In order to prevent this, QGIS keeps an index where vector data are stored in a way that it can quickly find out what geometry is beneath the mouse cursor. The building of this data structure takes time and that is what the progress dialog is about.
Firstly: Parallelize snap index build
If you want to be able to snap on all layers in your project, then QGIS will have to build one snap index for each layer. This operation was made sequentially meaning that if you have for instance 20 layers and the index building last approximatively 3 seconds for each, then the whole index building will last 1 minute. We made modifications to QGIS so that index building could be done in parallel. As a result, the total index building time could theoretically be 3 seconds!
4 layers snap index being built in parallel
However, parallel operations are limited by the number of CPU cores of your machine, meaning that if you have 4 cores (core i7 for instance) then the total time will be up to 4 times faster than when the building is sequential (and last 15 seconds in our example).
Secondly: relax the snap build
For big projects, parallelizing index building is not enough and still takes too much time. Futhermore, to reduce snap index building, an existing optimisation was to build the spatial index for a specific area of interest (determined according to the displayed area and layer size). As a consequence, when you’ve done waiting for an index currently building and you move the map or zoom in/out, you could possibly trigger another snap index building and wait again.
So, the idea was to avoid waiting at all. Snap index is now built whenever it needs to (when you first enable snapping, when you move or zoom) but the user doesn’t have to wait for the build to be over and can continue what it was doing (creating feature, moving…). Snapping highlights will be missing when the index is currently being built and will appear gradually as soon as they finished. That’s what we call the relaxing mode.
No waiting dialog, snapping highlights appears as soon as snap index is ready
This feature has been merged into current QGIS master and will be present in future QGIS 3.12 release. We keep working on this feature in order to make it more stable and efficient.
What’s next
We’ll continue to improve this feature in the coming days, if you have the chance to test it and encounter issues please let us know on the QGIS tracker. If you think about a missing feature or just want to know more about QGIS, feel free to contact us at [email protected]. And please have a look at our support offering for QGIS.
Many thanks to QGIS grant program for funding these new features. Thanks also to all the people involved in reviewing the code and helping to better understand the existing mechanism.
QGIS Versioning now supports foreign keys!
QGIS-versioning is a QGIS and PostGIS plugin dedicated to data versioning and history management. It supports :
- Keeping full table history with all modifications
- Transparent access to current data
- Versioning tables with branches
- Work offline
- Work on a data subset
- Conflict management with a GUI
QGIS versioning conflict management
In a previous blog article we detailed how QGIS versioning can manage data history, branches, and work offline with PostGIS-stored data and QGIS. We recently added foreign key support to QGIS versioning so you can now historize any complex database schema.
This QGIS plugin is available in the official QGIS plugin repository, and you can fork it on GitHub too !
Foreign key support
TL;DR
When a user decides to historize its PostgreSQL database with QGIS-versioning, the plugin alters the existing database schema and adds new fields in order to track down the different versions of a single table row. Every access to these versioned tables are subsequently made through updatable views in order to automatically fill in the new versioning fields.
Up to now, it was not possible to deal with primary keys and foreign keys : the original tables had to be constraints-free. This limitation has been lifted thanks to this contribution.
To make it simple, the solution is to remove all constraints from the original database and transform them into a set of SQL check triggers installed on the working copy databases (SQLite or PostgreSQL). As verifications are made on the client side, it’s impossible to propagate invalid modifications on your base server when you “commit” updates.
Behind the curtains
When you choose to historize an existing database, a few fields are added to the existing table. Among these fields, versioning_ididentifies one specific version of a row. For one existing row, there are several versions of this row, each with a different versioning_id but with the same original primary key field. As a consequence, that field cannot satisfy the unique constraint, so it cannot be a key, therefore no foreign key neither.
We therefore have to drop the primary key and foreign key constraints when historizing the table. Before removing them, constraints definitions are stored in a dedicated table so that these constraints can be checked later.
When the user checks out a specific table on a specific branch, QGIS-versioning uses that constraint table to build constraint checking triggers in the working copy. The way constraints are built depends on the checkout type (you can checkout in a SQLite file, in the master PostgreSQL database or in another PostgreSQL database).
What do we check ?
That’s where the fun begins ! The first thing we have to check is key uniqueness or foreign key referencing an existing key on insert or update. Remember that there are no primary key and foreign key anymore, we dropped them when activating historization. We keep the term for better understanding.
You also have to deal with deleting or updating a referenced row and the different ways of propagating the modification : cascade, set default, set null, or simply failure, as explained in PostgreSQL Foreign keys documentation .
Nevermind all that, this problem has been solved for you and everything is done automatically in QGIS-versioning. Before you ask, yes foreign keys spanning on multiple fields are also supported.
What’s new in QGIS ?
You will get a new message you probably already know about, when you try to make an invalid modification committing your changes to the master database
Error when foreign key constraint is violated
Partial checkout
One existing Qgis-versioning feature is partial checkout. It allows a user to select a subset of data to checkout in its working copy. It avoids downloading gigabytes of data you do not care about. You can, for instance, checkout features within a given spatial extent.
So far, so good. But if you have only a part of your data, you cannot ensure that modifying a data field as primary key will keep uniqueness. In this particular case, QGIS-versioning will trigger errors on commit, pointing out the invalid rows you have to modify so the unique constraint remains valid.
Error when committing non unique key after a partial checkout
Tests
There is a lot to check when you intend to replace the existing constraint system with your own constraint system based on triggers. In order to ensure QGIS-Versioning stability and reliability, we put some special effort on building a test set that cover all use cases and possible exceptions.
What’s next
There is now no known limitations on using QGIS-versioning on any of your database. If you think about a missing feature or just want to know more about QGIS and QGIS-versioning, feel free to contact us at [email protected]. And please have a look at our support offering for QGIS.
Many thanks to eHealth Africa who helped us develop these new features. eHealth Africa is a non-governmental organization based in Nigeria. Their mission is to build stronger health systems through the design and implementation of data-driven solutions.
