Input App helps to identify burial grounds in a citizen science project in the Netherlands.

“QGIS is my Number One tool for all my work. Through a podcast, I heard about Mergin and the Input App - I immediately installed it and have been playing around with it ever since. It has worked wonderfully and flawlessly. We haven’t had any problem with it so far. It was the perfect solution for this specific project!”

21st-century tools reveal 2000 years of our past history

Konan Pruiksma, born in the Netherlands, is an archaeologist and GIS (Geographic Information System) Specialist experienced in Data Analysis, Relational Databases and Geostatistics, who is making a career in exploring our cultural heritage – in particular the vast wealth buried in the fields of the Netherlands. As an employee of Tijdlab, he was approached by Heritage Quest (Erfgoed Gezocht) and Leiden University/Erfgoed Gelderland to assist in their citizen science project, in which volunteers participate in archaeological research on the Veluwe and Utrechtse Heuvelrug.

Area of already well-known burial mounds created in QGIS

Secrets of Burial Mounds

The Dutch landscape is dotted with ancient burial mounds, Celtic fields and cart tracks, some dating back to the 3rd and 2nd millennium BC. Bodies were first cremated and then the ashes were buried in these mounds. In the Middle Ages, mounds were also used for navigational purposes and even for gallows. The ashes of important people were probably honoured with a solitary mound, whereas other mounds contained the ashes of whole families or of many individuals.

Two burial mounds

Aims of the Heritage Quest Project

The aims of the Heritage Quest project are two-fold, focusing on two views of the concept of citizen science:

1. The public viewpoint - to introduce as many citizens as possible to an awareness of the unique archaeological heritage in the Netherlands literally under their feet. This encourages better protection and conservation of this ancient and fragile heritage. Citizen science lessons are even planned for the classroom, to make children aware of their archaeological heritage and how to conduct scientific research.

2. The scientific viewpoint - to collect as much information as possible about the as yet unknown archaeological treasure trove. With the aid of LiDAR map data, huge areas become visible, which were previously hidden by vegetation and thus undiscovered. Later, with the participation of citizen volunteers, a large amount of field data can be collected, which archaeologists simply do not have the manpower to gather.

Volunteers using Input App in the field

LiDAR Assists in New Discoveries

LiDAR maps – high-resolution models of ground elevation created by a laser scanner, GPS and INS systems mounted on a small aircraft – are made accessible to the project by the Dutch government. Over the past 2 years, these LiDAR maps of areas of suspected burial mounds were analysed by over 6 500 volunteers, even by children, who searched over 600 000 maps and identified many thousands of possible new discoveries. In addition to burial mounds, Celtic fields (agricultural fields about 2 500 years old) and deep linear depressions left by the wheels of carts or wagons are clearly visible in the sandy soil of the Utrechtse Heuvelrug and Veluwe on LiDAR maps.

This manual work of many volunteers has another great benefit. The identified objects from LiDAR maps were used as a teaching dataset for a neural network that could potentially do a similar task automatically in the near future for different sites.

LiDAR image showing burial mounds and tracks

As every map was inspected by at least 15 different participants, the difference in the probability of potential barrows become clear immediately; some hills are identified by all participants, while others only by a few. It is probable that hills recognised by more people have a higher chance of being burial mounds and not natural hillocks. Following this reasoning, there are about 6 000 hills that have a high potential of being ancient burial mounds. However, this needs to be checked in the field.

Volunteers Gather Data in the Field

In summer 2021, the field work began and this is where Mergin and Input App came into play.

“For this project, I immediately thought of Mergin and Input App and it worked from the get-go,”

says Konan Pruiksma regarding the ideal tools he chose for data collection in the field. Volunteers install the Input App to their phones and see the potential burial mounds or Celtic fields on the map. They navigate to a point of interest, digitise the point and fill in the required information, such as photos and notes, in a form. Such field work will continue to be carried out over the next year or more.

Burial mounds to be verified and form to fill in by volunteers

Once they have all the information about the location in Input App, they synchronise the data back to the Mergin Cloud. Konan, as a field manager, sees that data are synchronised on Mergin Cloud Dashboard, as well as who made the changes and when. If he needs to update anything, he can do it even from his office and let volunteers refresh the map. The collected data are stored in the PostGIS database via a docker container for further analysis. Konan uses QGIS Plugin for Mergin to download the field data gathered and to analyse them. When the point has been confirmed as a possible burial mound in the field a certain number of times, it is removed from the volunteers’ maps on Input App and reported to the professional team of archaeologists.

“Input App is user-friendly and can be used on any smartphone. This makes it possible to be used by volunteers with only minimal instruction. Which is great, because with over 6 000 locations to be inspected, we need all the help we can get!” explains Konan. “The App furthermore enables volunteers and archaeologists alike to locate the barrows which are usually hidden beneath vegetation. In the field, these low rises are often poorly visible, which is the reason they have not been identified until now. Before we used Input App, it often took us a very long time just to be able to locate the hill highlighted on the LiDAR map.”

Konan doing analysis of collected field data on his laptop

Information Gleaned from Soil of Burial Mounds

This team further investigates the mound by borehole surveying and the removal of soil samples. The mounds are generally not excavated, only if there is a danger of them being destroyed. Archaeologists prefer to keep these archaeological remains intact.

By coring with a 7 cm auger, a thin soil profile can be extracted from the barrow. In this way, archaeologists can get a small glimpse of the different layers that are present beneath the soil without destroying them by excavation. This provides invaluable information about our prehistoric ancestors. Radiocarbon (or Carbon-14) dating can provide accurate dating of the contents of these prehistoric mounds.

Konan says, “We previously did not know that there were so many burial mounds in the Netherlands!” He explains how information is obtained from the soil, without the need for excavation: “If we find charcoal, it is almost 100 % sure that it is a burial mound. We can learn what the burial rituals were of our ancestors, how they lived and what they ate.”

Conclusion

There are inestimable benefits of collecting and interpreting data accurately. The vast number of burial mounds in the Netherlands would not be able to be detected were it not for the assistance of volunteers, combined with LiDAR maps and tools such as QGIS or Input App. At present, there is a team of 20 volunteers at work on the Veluwe project, but this number should increase in the future, as more citizens become interested in doing citizen science in their environment. As Input App is user-friendly and very intuitive, a minimum amount of training is needed for volunteers, many of whom are students or senior citizens not au fait with digital technology.

Download Mergin Maps Today

The next round of crowdfunding to improve point cloud and elevation data in QGIS is here! Similar to the previous crowdfunding campaign, we have teamed up with North Road and Hobu to improve elevation and point cloud data in QGIS.

Last year, we added point cloud data to QGIS, thanks to the great support from the community. Based on numerous feedback and suggestions we had over the past year, we have decided to run another campaign to introduce some new tools and enhance user experience when dealing with elevation and point cloud data. Below is a list of those features we intend to introduce. For more details see the crowdfunding page.

Profile tool

A tool for drawing profiles of elevations is essential when working with 3D data, yet it has long been missing from QGIS. While there are several plugins for QGIS that add this missing functionality, unfortunately they all have limitations when it comes to ease of use, performance, integration with the rest of the user interface and wealth of functionality. The goal is to add a profile tool that would cover a wide variety of use cases, and be available immediately out-of-the-box for all QGIS users.

Enhanced 2D visualisation of point cloud

A technique that is extremely useful for visualization of point clouds in the 3D map view is the Eye Dome Lighting effect, which adds subtle shading and silhouettes (when there is a greater change of elevation) making it much easier to identify various features of interest that could be otherwise hard to see. To date this feature is only available in 3D QGIS map views, but we would like to introduce it to 2D map view as well! Many GIS workflows and products are still entirely based around 2D views of data (such as digitizing of vector data), so having the extra visual clues which Eye Dome adds is also highly desirable in 2D views.

Filtering point clouds for visualisation

A common requirement of analysts when working with point clouds to work with just a subset of the dataset. This could be as simple as showing only one class of a classified dataset, using only a limited range of elevations, viewing just a particular flight line, or looking at just the first/last returns. Currently, QGIS only provides basic class-based filtering when using rendering based on classification.

Improve 3D map views

One of the limitations is the 3D map view’s window: when it gets closed the configuration of the 3D map view is discarded and any newly opened 3D map view needs to be re-configured again. This is frustrating for users and results in a lot of wasted time! We would like to fix this by keeping 3D views stored in QGIS projects, even when they are not currently being shown. Another issue related to 3D map views is the fact that they are currently only “dock widgets” that can be embedded within the main window. However, often it is preferable to be able to see the 3D view content in a big window, which is quite tricky to do in current QGIS versions (especially when the user does not have multiple displays).

Cloud-optimized point clouds

The new specification of cloud-optimized point clouds (COPC) is current being finalized (October 2021) and it is a very exciting innovation! Inspired by the hugely successful cloud-optimized GeoTIFF (COG) format, the idea is to reuse the existing LAZ format (widely used by the lidar community) and add the improvements which make the format more useful.

More point cloud data formats & more robust loading

While LAS and LAZ files (currently supported by QGIS) are the most common formats for point clouds nowadays, they are not the only ones. In order to support more formats in QGIS we do not need to worry about technical details of each format thanks to the PDAL library. It comes with a multitude of drivers for various formats and services - see https://pdal.io/stages/readers.html . In QGIS we need to provide a user interface to allow people to get other additional formats to work. Some drivers are always available in PDAL, others are optional and may need extra third-party libraries in order to work. We welcome feedback from the community regarding drivers they consider important.

About the crowdfunding campaign

Please visit the crowdfunding page for more details about the campaign!

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

Android has forced app to use Scoped Storage for all app related data. If you are using Input app on Android please read carefully for the upcoming update (1.1) of the app in the Google Play Store.

What is Scoped Storage?

In summary, apps need to use a specific part of folders on Android devices to store app related data.

Currently, Input stores your QGIS project and some other settings (e.g. grid shift projection) on /Internal storage/INPUT. With the new Android requirements, the app related data should be stored on /Internal storage/Android/data/uk.co.lutraconsulting.

Update process

For the 1.1 release of Input on Android, there will be an extra process. This process will be a on-off action. When you launch the app after the upgrade, it will copy the data from /Internal storage/INPUT to /Internal storage/Android/data/uk.co.lutraconsulting. Depending on the size of your projects, this can take a couple of minutes. During the process you will see a screen similar to this one:

Best practices

The upgrade and copy process should work smoothly. But we suggest to take the following actions to ensure you will not lose any data during the process:

• Sync all you changes: before upgrading the app, open Input and sync all your local changes to Mergin.

• Storage space: if you work with several projects and large volume of data, make sure you have enough storage. The process will make a copy of your existing /Internal storage/INPUT without deleting it. So, you need at least the size of /Internal storage/INPUT storage available.

Troubleshooting

In case you have encountered any issues, you can take the following steps to fix the problem manually:

• Lack of storage space:
  • Transfer the data through USB cable to your PC.
  • Make a back up of data on your PC.
  • Delete /Internal storage/INPUT on your Android phone/tablet
  • Transfer the data from PC through USB to /Internal storage/Android/data/uk.co.lutraconsulting
  

• Missing data: the migration process does not delete /Internal storage/INPUT folder. It will rename it /Internal storage/INPUT_migrated. Similar to step above, you can copy the data to the PC and move them to /Internal storage/Android/data/uk.co.lutraconsulting. Alternatively, you can use a file browser app on your device to copy files around.

• If you use Input app on a shared device, the migration process will transfer all the project data from /Internal storage/INPUT and marks the folder as /Internal storage/INPUT_migrated. Therefore, when the next user starts up the app, no data will be present. To fix the issue, you need to manually move the data from /Internal storage/INPUT_migrated to /Internal storage/Android/data/uk.co.lutraconsulting (as described above) for the other users on the device.

Need help

If you need further help, please join us on the community chatroom and we will be able to help you with the upgrade issue (or other Input/Mergin related problems.)

Precision farming opens an opportunity to increase the gain for a farmer up to 20% with fewer trees.

Making a sustainable profit from farming can be something like tossing a coin. However, today the advancements in digital technology provide the incredible benefit of reducing the unpredictability of farming in general. In pecan farming in particular, the proper know-how of a combination of multiple factors such as soil composition, environmental impact, climate, irrigation and sunlight management are vital for success. Hard data is essential for innovating key decisions. Riaan Burger, a South African farmer, successfully uses open source GIS software to tackle the issues of collecting and utilising such vital data.

Riaan Burger on his pecan farm.

Becoming a Farmer

Riaan Burger’s journey started back in 2017 when, after 20 years’ working as an electrical engineer, he decided to embark on his own business in pecan nut farming. He is a self-taught farmer and co-owner of one of the biggest pecan farms in the district of Weenen, South Africa. At first, Riaan had only a rough idea of how to improve his newly bought 24 hectares of planted orchard.

After reading books on soil structure, type, depth and chemical analyses, he had soil samples taken to create a soil sample map. In his orchard, the trees were planted in 10x10 metres geometric layouts. After 20 years, the trees were already creating shade. Riaan knew that sunlight penetration is important and that pecan trees are susceptible to fungus, so he decided to take a closer look.

“The previous owner of the farm had already started chopping out every second tree. I was in the position where I could compare the yield from areas where trees had been chopped down, to areas where they had not been felled,” explains Riaan.

A photo of the farm taken by drone and initial stage of farming - soil maps.

Riaan realised that understanding the ambient conditions and mapping them to yield results was the right way to go. This was the beginning of his exciting journey in precision farming.

Collecting Data from the Field

It was no easy task for Riaan to get a complete picture. For two years, he only recorded the yield per day, and the relevant orchard block. Later, he used the services of a local consulting company which provided him with surveying applications to help him to set up a basic workflow.

“I record the weight of pecan nuts from each individual tree and I plot it on the map,” Riaan says. “And then afterwards I can see the different cultivars of trees, the weight of nuts they produced and the area in which they were produced. I previously had soil samples taken, so I’ve got soil sample maps. Now I can overlay a yield map over the soil sample map and correlate them”

Despite working well with the surveying app, Riaan was still in need of assistance to prepare data tables and then to export them to Excel sheets and to request map creation. After the consulting company moved from a third-party app to the in-house build solution, things got worse for Riaan: “It was a generic tool and a difficult app to work with. Displaying data was cumbersome. I wanted something that was quick and easy again. Ideally, I would like to capture data on the iPad and have it directly linked to my laptop – that is how I think it should work.”

So Riaan discovered the Input and Mergin suite, a surveying app based on QGIS and a cloud service for data synchronisation, respectively. Despite only a basic knowledge of QGIS and the necessity to learn and experiment, the effort immediately paid off!

“I’ve got Input, I’ve got a Mergin account and I’ve got QGIS. I now have full control of the process. I don’t need to fund a consultant and find that by 8 o’clock at night he has not yet responded to my query, and so I have to back it up with a WhatsApp call,” Riaan explains. “This is something I created entirely myself. I don’t need to ask someone to change this map, or add that icon, modify the legend or change the styling, I am in control of all of it! And I think there is a sense of satisfaction when one gets things going for oneself.”

A farming project in QGIS desktop and Input app UI.

With the Input app, Riaan has gained control of the workflow – something not before experienced. He has gained the freedom to set the project according to his own individual needs. Without much effort, not only the data but the styling, an important factor for him, are synchronised.

“In QGIS I created a style where each bubble represents the kilograms of the yield and the colour represents the variety,” says Riaan. “Then I saw in Input that there is suddenly an extra page on my iPad. So now I have the bubbles and bubble colours on the iPad instantly, which I did not have in previous apps. Previously, I had to create it the long way around, but now it is suddenly on the iPad automatically. I enjoy having that available! In the past, I only had that information by the end of the harvest.”

Cutting Down Trees

The energy invested in creating workflow, capturing and analysing data paid off handsomely for Riaan. All his initial doubts were suddenly eliminated. He has all the information essential for making any important decision right at his fingertips.

“The first time you fell trees, you have nightmares! How on earth you can be cutting your income and destroying 20 years of growth? Now, with the necessary proof at hand, it is easier in your mind to motivate that tree felling is actually your source of income, that it is the right thing to do,” says Riaan. “You can see that the yield from the remaining trees is more than double than before. The trees are healthier, because of allowing sunlight in. Moreover, you also get less pressure from fungus, because pecan trees are easily susceptible to fungus. Now that you have good ventilation, because you have a draught through your orchard, you can see on the map what your exposure to fungus is.”

Orchard with higher density of trees (top) and lower density (bottom).

Riaan continues: “I am not a sole owner, I have a partner. Now I have the data available, so that when we have a meeting I can explain to him this is what I did, this is what happened and this is the result. If you go to a bank manager for a loan, I mean, he really frowns at you when you tell him you are cutting down your trees which produce your income. Then you can explain to him why it is for the better.”

Increase in yield per tree after reduction of tree density.

Harvest Data Analysis and Beyond

The benefits of collecting and interpreting data carefully are limitless. Cutting down trees is just one example of the many advantages of careful data gathering and interpretation.

“Now I see that certain cultivars perform better than others this year,” adds Riaan. “So I can see if they consistently perform better or worse. I can decide, based on these results, what to do: if I need to add more nitrogen or improve soil pH, or whatever the case may be.”

User-friendly tools tailor-made to specific needs, such as the Input and Mergin suite, provided Riaan a way of collecting and analysing his specific type of farming data. His insights for necessary improvements also developed over the past few years.

“At first I did not have the distinction between individual trees and I even said to my partner” continues Riaan, “if only I’d started doing this in the very first season, I would have had 5 years of data to compare – but I didn’t. I will definitely continue to use this suite. It is valuable and I have been using it only for harvest. However, I see now that I can start recording where I have cut down a tree, so I have a record of when and where the tree was cut. I can also record my irrigation data.”

In spite of initial teething problems and the difficult decisions he had to take, Riaan’s success is proof that he has found the right way to go for his own specific pecan farming needs. He has even become an evangelist within the local farming community.

“I am not the only pecan farmer in our district. I am the biggest pecan farmer, but not the only one. I am trying to spread the word around. For example, my neighbour still does not want to cut down his trees. But I am getting there, showing him photos, proving that you have to be aggressive to open up the orchard to get sunlight in – and here is the proof that it has been working for me!”

Download Mergin Maps Today

Input app has reached a new milestone. Earlier in September, we have released version 1.0 of the app with many new features and enhancements.

Make syncs faster

When you work in a team with several collaborators adding data and photos to the project, the size of the project can get really big. Every time you try to sync your changes, photos from all users will be transferred to your phone. This can take a long while when there are several hundreds of photos collected by other collaborators.

Image below illustrates the difference between having selective sync or the default behaviour (selective sync is disabled):

The configuration file for enabling the selective sync is stored in mergin-config.json which should be placed in the root of your folder. By opening the file in a text editor and adding the following option, the sync will be enabled for photos within your project root folder:

{ "input-selective-sync": true }

Currently, the editing of the file is manual and through the text editor, but we plan to incorporate it within the Mergin plugin for QGIS.

To learn more about how to set up selective sync for your project, you can see the example project. Read more on our help pages for extra configuration options.

One too many!

It is often the case that you have a set of spatial features and you want to record some parameters every now and then. For example, there is a GIS layer representing the manholes and the surveyors carry out regular inspections of the manholes. Instead of duplicating the manhole layer and recording each inspection, you can create a non-spatial table and store each inspection as a new line.

Another use-case for such a feature is that you’d like to attach multiple photos to a single feature.

To learn more about how to configure these types of projects in QGIS you can see the example projects (manhole example and multiple photos example). The documentation pages describes the logic and process in QGIS in more details.

Accuracy metadata

In addition to the display of the accuracy bubble in the app, we have recently added a whole set of new variables to capture the GPS accuracy, e.g. horizontal and vertical accuracy, ground speed and many more. See the help pages to find out how you can set up those variable within your form. Alternatively, you can clone the example project on Mergin website.

In addition to capturing GPS metadata, this feature can be used for geo-fencing: for example, you can only allow users to edit/capture data when they are physically (i.e. their GPS location) within a certain area:

Join our community

If you have any questions, would like to interact with the rest of community or want to give us your feedback, you can join the Slack community channel.

If you would like to add a new feature or have suggestions to improve the app, do not hesitate to contact us on [email protected]

Read the guest post and congratulate Francesco Bursi, who successfully completed GSOC 2021 project to add virtual raster provider for QGIS with help of mentors Martin Dobias and Peter Petrik.

In this year’s Google Summer of Code (GSoC), I decided to work on the native QGIS raster calculator. Martin Dobias and Peter Petrik volunteered to mentor my work. I’ve been studying Civil Engineering and GeoInformatics at the University of Padua; here I had the opportunity to work both with a lot of GIS software including QGIS. I enjoyed working with QGIS almost immediately because of the possibility to perform complex analysis with a few clicks or with few python commands. Being passionate about programming and enthusiastic about Open Source, I realized that having the possibility to work together with some experienced developers and with an active community was really a great and unique opportunity, so I decided to apply to the GSoC.

Virtual Raster Provider

The existing raster calculator is a powerful tool to perform map algebra that outputs a raster layer, before this work it was possible to take advantage of this tool only by saving the output of this operation as a file. The aim of this year GSoC was to allow users to perform their analysis without creating a new derived raster and taking up disk space and therefore have the result as an on-the-fly computational layer.

Let’s jump to an example and let’s say I want to compute the Chanopy Height Model (CHM), subtracting the Digital Terrain Model (DTM) from the Digital Surface Model (DSM).

I also want to perform some other analysis on the DTM since I want to compute the ideal elevation value for a particular tree planting (disclaimer: the elevation value used is example purposes only, moreover when planting trees you should take into account a lot of factors like slope, aspect, latitude. QGIS, by the way, can really be helpful in this kind of analysis). To do so I will start from the same data and I will create different on-the-fly layers for each calculation, in order to avoid the creation of different files I can take advantage of the new checkbox added to the raster calculator dialog. The computation of CHM is performed in the next screencast and the output layer name is, of course, CHM.

I’ll end up with a new raster layer (CHM) that can be styled as a normal raster and that is not written as an output file to the disk. For some further analysis, from the DTM, I want to obtain the portion of the area with an elevation between 150 and 350 metres above the datum. By applying the following expression to DTM I’ll end up with a raster that has a value of 1 where conditions specified by the expression is TRUE and it will have value of 0 otherwise.

("dtm@1" > 150) AND ("dtm@1" < 350)

I did not select the output layer name intentionally. The resulting layer will be named after the expression used to generate the layer.

Conditional Statement

I also had the opportunity to improve the raster calculator capabilities by adding the possibility to write expressions that involve conditional statements. Taking the already used example, let’s imagine I want to compute the CHM only for the areas of the DTM that are between 150 and 350 metres above the datum. It’s now possible to write an expression as the following one:

if ( ("dtm@1" > 150) AND ("dtm@1" < 350), CHM, -10)

This expression will output a raster with values of the CHM where the conditions are met and value of -10 if the conditions are not met. Since this is a final result of our analysis I’ll store this output as a file to the disk in the form of a GeoTIFF. I’d like to outline that the CHM used in the expression above and in the next screencast is an onn-the-fly computed raster, so it is possible to:

• Take advantage of the virtual raster provider (on-the-fly computed raster) in other analysis with the raster calculator (and with other analysis tools);
• Store the on-the-fly computed raster as a file.

Conclusion

I had fun and I struggled working with QGIS, but I learned a lot of new and interesting things. My pull requests were met with several constructive comments, suggestions and feedback. Some suggestions can be a starting point for future improvements.

• An enhancement for the feature I’ve developed can be the possibility to take advantage of OpenCL acceleration as it has also been suggested in the dev mailing list;
• Another enhancement that concerns the raster calculator and only partially the virtual raster provider would be the possibility to support the creation of output raster with multiple bands with the declaration of multiple formulas. I hope to continue to contribute to the QGIS project in the future.

Significant time saved when route maps distributed with Input and Mergin.

This case study was originally written in Czech. The Czech version can be found here.

Every year, teams of volunteers walk door-to-door through the Czech town of Litomyšl collecting charitable donations. Event organisers define routes for the various volunteer teams by marking-up paper maps with pens. The process has a number of issues both in the making and usage of the maps which organisers worked to overcome by making the maps digital using open source GIS software.

Maps were developed using QGIS and made available on volunteers’ phones using the Input app. Volunteers are now able to easily orientate themselves on maps which clearly show their routes. Organisers have reduced the time it takes to update routes and distribute these to volunteers.

Veronika Peterková works for the Litomyšl Parish Charity, a non-profit organisation providing health and social services to people in need since 1993.

Veronika describes the charity’s activities: “We provide home medical services and nursing care to the residents of Litomyšl and its surrounding villages. This includes helping families where the healthy development of a child is at risk and providing respite stays for clients who are otherwise cared for by their families at home. We provide care for about 1000 clients a year.”

She added: “We also coordinate the activities of volunteers who visit the elderly, help with tutoring children and with various leisure and cultural activities.”

One of the parish charity’s biggest fundraising events is the “Tříkrálová sbírka” (Three Kings Collection), a door-to-door carol-singing collection taking part around the 6th of January each year.

Volunteers participating in the Three Kings Collection.

“The Three Kings Collection is the largest national volunteer event in the Czech Republic. In the Litomyšl region alone, nearly 300 volunteers are involved each year with the carol-singers collecting over 500,000 Czech crowns (~20,000 EUR) in sealed boxes. The proceeds are intended to help the sick, the disabled, the elderly, mothers with children in need and other in-need groups in the local area.” Veronika explains.

The Three Kings Collection is organised by Caritas Czech Republic and at least 10% of its proceeds are allocated for humanitarian aid abroad.

The Challenge

Veronika is responsible for planning routes for the carol-singers so they efficiently visit households in the Litomyšl area. Singers are split into groups and paper maps are provided which show groups which households to visit.

An example of previous paper maps, image courtesy of Farní charita Litomyšl.

The above maps were produced by printing screenshots from a national web mapping provider and marking-up printouts for each of the 50 teams using marker pens.

This method proved to have a number of issues as Veronika describes: “On maps of larger areas, house numbers were not always visible due to the scale. This made it even harder for coordinators not familiar with the area to orient themselves, leading to confusion. Coordinators also found it hard to keep the maps dry and undamaged during unfavourable weather. If new groups signed-up afterwards or others opted-out, we’d have to redo/redivide the areas which would be very time-consuming as the maps would need to be marked-up manually once again.”

The Solution and Implementation

Veronika wanted to try a new solution for organising the 2021 Three Kings Collection with the goal of making volunteer tasks clearer and less reliant on paper maps. She wanted the new solution to allow her to:

• reduce work through the reuse of maps in future Three Kings Collection events
• easily update maps if new groups sign in/out and areas need editing
• allow carol singers to see exactly where they are on the map
• gradually replace paper maps while still allowing the use of paper maps where preferred
• group and colour buildings to be visited on the computer
• record a building’s use (e.g. commercial) to direct volunteers more effectively
• clearly show how areas are assigned so anyone can see who is responsible for a given area

In addition, Veronika wanted the solution to be affordable and work offline without volunteers needing internet connectivity in the field.

Peter Petrík, a regular participant of the Litomyšl Three Kings Collection suggested Veronika try using the Input app for coordinating the collection in 2021. Peter works for Lutra Consulting, the company behind Input and Mergin.

He showed Veronika how to create the maps in QGIS, a free and open source mapping software. Using map data from OpenStreetMap, they created a project showing the buildings to be visited, coloured by their associated volunteer group number.

Houses grouped by team in QGIS, image courtesy of Farní charita Litomyšl.

The styled map was uploaded to Mergin, a collaborative mapping platform, making it readily available for viewing interactively on volunteer’s phones using the Input mobile app. Both QGIS and Input integrate closely with Mergin which meant that maps could be adjusted in QGIS with the resulting changes being visible to volunteers shortly thereafter.

Outcomes

Veronika reflects on the solution: “The solution met all our requirements and the maps we’ve prepared can easily be reused in upcoming events, saving us time. The fact that the new maps were made publicly accessible means volunteers can just download them using Input which makes distributing and updating them very easy.”

Volunteer routes and position information shown in Input, screenshot courtesy of Farní charita Litomyšl.

She adds: “All the districts we wanted to visit were distinguished from each other by colour and we were also pleased to be able to clearly mark the areas not to be visited like industrial areas by colouring them in grey.”

Unfortunately COVID meant that Veronika’s plans changed as she explains: “Using these new methods we were able to prepare for the 2021 Three Kings Collection in a short time. Unfortunately however, the COVID situation meant we could not go out on the streets to use the new maps as intended. We hope that in 2022 we’ll be able to more closely evaluate the positives and negatives of the field aspect of the project.”

She adds: “We already see it’s now much easier to allocate areas of the town to our volunteers in a clear and fair manner using QGIS. Producing printed maps for those who prefer them is also now easy and the maps look much more professional. Those who only wanted to use the Input app could see the same information as on the paper maps, but had the advantage of being able to pinpoint their exact location and clearly see the house numbers of each building.”

Example printed map created for volunteers wanting also paper maps, image courtesy of Farní charita Litomyšl.

She concludes: “Overall we found the solution user-friendly, and appreciated being able to discuss the process with Lutra Consulting who helped us solve issues as required. About a third of our volunteers are interested in using Input, which I consider positive.”

The Litomyšl Parish Charity are on Facebook and Instagram.

Download Mergin Maps Today

The case study presents the C++ development of QGIS Desktop to support rendering of 3D results produced by TUFLOW’s 3D capable solver: TUFLOW FV (10 minute read)

Introduction

TUFLOW is a suite of advanced 1D/2D/3D computer simulation software for flooding, urban drainage, coastal hydraulics, sediment transport, particle tracking and water quality. With over 30 years of continuous development, TUFLOW is internationally recognised as one of the industry leaders for hydraulic modelling accuracy, speed and workflow efficiency.

Lutra Consulting Ltd is a leader in software development for pre- and post-processing of hydraulic and meteorological results in open-source QGIS. We also work on mobile data collection Input App and GIS data synchronization service Mergin

In 2019 the TUFLOW team commissioned us to develop post-processing support for their TUFLOW Flexible Mesh format for QGIS 3.12. The format is 3D stacked mesh, which consists of multiple stacked 2D unstructured meshes each extruded in the vertical direction (levels) by means of a vertical coordinate.

At that time QGIS only supported 2D meshes that defined results on vertices and faces. We had been keen to extend the capabilities of the software stack to support 3D mesh data for a long time so this was an exciting opportunity. Part of the task was also to include rendering support for TUFLOW model results on the QGIS 3D view. The delivery of the project was within one QGIS release cycle (less than 6 months time for users to use it on their projects!)

Contact us at [email protected] if you’d like to discuss the benefits of integrating your flood modelling software more tightly with QGIS or you have some custom QGIS development in mind.

C++ Development Process: From requirement to delivery

Communicate project with the community first

When doing a substantial change in the QGIS codebase, the developer needs to write a technical specification of the QGIS changes for community discussion. QGIS Core Developers (which Lutra is a part of) can give valuable feedback to the overall technical approach and the wider community can raise some usability issues or enhancement proposals. Most importantly, each part of the QGIS code has its lead maintainers, for example Martin Dobias, our CTO, is the maintainer of QGIS 3D code and Peter Petrik is the maintainer of the Mesh layer code. It is a good practice to address the maintainers’, users’ and other developers’ concerns and feedback to ensure the feature can be implemented in QGIS.

So after a thorough discussion about the requirements with the TUFLOW team and analysis of the existing tools for post-processing and display of the TUFLOW FV format we came up with the QGIS Enhancement: Support of 3D layered meshes

The community reaction was very positive and supportive. Time to start coding!

MDAL to support TUFLOW FV NetCDF format

Mesh Data Abstraction Library MDAL is a C++ library for handling unstructured mesh data. It provides a single data model for multiple supported data formats. MDAL is used by QGIS for data access for mesh layers. If you want QGIS to support your data format, you need to have a driver in MDAL that implements it.

We added support for 3D stacked meshes and the TUFLOW FV format in MDAL. When we develop features in MDAL, we focus on quality code, so

• all changes have a proper code review,
• all code has fully automated tests with more than 90% coverage target
• the documentation and manual testing is done after coding

To implement the TUFLOW FV driver for 3D stacked meshes, we added a new API/interface in MDAL, so we needed to follow up with the QGIS changes in QgsMeshLayer and MDAL data-provider.

QGIS C++ Development to support stacked meshes and visualization in 3D

The implementation of large feature changes is best to split into smaller but self-consistent parts. For example the first pull request added the basic support for the new 3D stacked meshes. Each pull request we do has a screenshot or gif/video with the new functionality preview, follows QGIS Coding Standards, has unit tests where necessary and includes the documentation for the functions/classes added in the public interface. Once the request is merged, the features are next day available in nightly builds on all platforms for testing!

Final Steps: feedback, testing, documentation and presentation

When all the features were in QGIS master, the TUFLOW team used windows nightly builds to test the new features and provide feedback. After a small number of iterations, all issues were resolved and implementation signed.

Shortly the new official QGIS release was published and we started promotion of the new features on our social media channels. Also, the features developed under this contract were promoted in the visual QGIS changelog.

• Mesh Layer in 3D
• Mesh Reference Time
• Mesh Vector Streamlines

Benefits for TUFLOW to support QGIS Core C++ Development:

• Reduced development and maintenance costs for tools such as the TUFLOW Viewer QGIS Plugin since the new features are part of the QGIS core
• By being part of the QGIS ecosystem it provides opportunities to approach QGIS users in the flooding and coastal modeling industry to use TUFLOW software
• As a project sponsor, the requirements of the new features meet the present and future needs of the TUFLOW user base.
• At the beginning of the project Lutra showed all the current relevant capabilities of QGIS ecosystem, allowing TUFLOW to be aware of the latest and greatest features
• Allowed TUFLOW to solve upstream bugs in QGIS or MDAL due to the open-source nature of the projects

Benefits for TUFLOW users:

Key benefits made available to TUFLOW users include:

• Being able to work with TUFLOW models using open source GIS on all major operating systems
• A full GIS application to support their data pre-processing
• Logical and intuitive workflows
• Visualisation and post-processing of TUFLOW results natively in QGIS via mesh layer
• The development allows interactive plotting features for 3D results, such as 3D profiles and curtains that can be easily extracted, providing an improved user experience
• Ability to use all native QGIS support and development channels in addition to TUFLOW support
• Integration of internal workflows with powerful native QGIS features including projection support, GDAL/OGR integrations, background maps support (e.g. vector tiles) and printed flood maps.

Further Reading

• TUFLOW webpage
• MDAL and Mesh Layer
• TUFLOW Viewer QGIS Plugin

Do you have any questions or would like to see a demo of the QGIS Mesh Layer? Contact us at [email protected] or schedule a demo call calendly.com/saber-razmjooei/15min

Key words

QGIS, migration, optimised, speed up, fast, hydraulic modelling, water, 2D, 3D, open-source, cost reduction, software development, TUFLOW, TUFLOW FV

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

Mesh Data Abstraction Library MDAL proudly joins the OSGeo Community program

We would like to share the excellent news that the OSGeo incubation committee kindly accepted the MDAL project into its Community Program. MDAL is an MIT Licensed C++ Library which integrates into your QGIS installation and is backed by the Lutra Consulting team and the wider community.

MDAL is a low-level library which allows reading unstructured mesh data from various formats. If you’ve never seen a QGIS Mesh Layer, then check out my presentation at FOSS4G 2019.

We kicked-off the MDAL project at the time we were migrating the Crayfish plugin to QGIS version 3. The Crayfish project started in late 2012, has more than 120,000 downloads and opened the way engineers and data scientists to visualize their hydro and meteo-data in QGIS.

QGIS uses MDAL to visualize, analyze and even modify data on unstructured meshes which are used by many hydraulic / numercial modelling solvers. In addition to providing abstraction for complex mesh data structures, MDAL also allows vectoral (e.g. wind speed) and temporal data to be visualised effectively and easily in QGIS.

The MDAL project (and therefore the wider community) has benefitted from generous sponsors such as TUFLOW, 3Di, Hydrotech, DHI, FLO-2D, Artelia, Federal Ministry of Austria for Agriculture, Regions and Tourism and many others. If you’d like to see your formats or workflows supported by QGIS then please consider becoming a sponsor or contact Peter at [email protected]

Right now we’re working on some new mesh frame editing tools for QGIS 3.22 (due for release in autumn 2021) and of course support for additional result file formats.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

We’d like to share some exciting news with you about our cloud-based geo-data synchronisation service, Mergin.

In this post we’ll talk both about Mergin the online managed service (Mergin Cloud) and also about the software stack that powers it (the Mergin Software Stack).

Mergin CE

The Mergin Software Stack has been developed and maintained at Lutra Consulting over the past 3 years to power our Mergin Cloud service and has been maturing nicely in production.

We believe a sync service for supporting field-based GIS activities has been a missing piece of the open geospatial puzzle, and as strong advocates of open source software, we’re now sharing the Mergin Software Stack with the community. Therefore, on the 14th of June we released Mergin CE on GitHub. The CE stands for Community Edition.

The release means a fully open solution for field data collection and synchronisation is now possible using QGIS, Input app and Mergin CE.

Mergin CE is released under the AGPL licence and we are open to contributions from others. Looking forward to seeing what the open source community has to offer!

Mergin CE gives you the freedom to deploy, host and manage your own Mergin server on your own infrastructure, giving you complete control over your data. Mergin CE comes without commercial support.

Our main efforts are still very much focussed on the continuous improvement of Mergin Cloud, making it an awesome fully managed service for our customers.

We also now provide Mergin EE (Enterprise Edition) for those who want an on-premises deployment but with extra features like Active Directory integration, commercial support, and/or prefer a licence other than AGPL.

Changes to Mergin Cloud

Releasing Mergin CE got us taking another look at Mergin Cloud’s Community (free of charge) tier. That’s why from today we’re updating Mergin Cloud’s Terms of Service so its free tier can no longer be used to store projects for commercial use. Mergin accounts storing projects for commercial use should now purchase a paid subscription after their initial 14 day evaluation period.

A common surveying setup is a single paid account (providing extra storage for projects) and a handful of free tier accounts used by surveyors for collecting field data. This is still possible because the new Terms only require that the Mergin account hosting/storing/owning the commercial project has a paid subscription.

For example, consider two users: Fred (who uses a free account) and Penny (who uses a paid account). Penny is permitted to use the project Penny/Survey for commercial purposes as it resides on her paid account. Fred (who uses a free account) is permitted to collaborate on the commercial project Penny/Survey as it resides on Penny’s paid account. However, Fred may not use the project Fred/Survey2 for commercial use as it resides on his free account.

In light of the release of Mergin CE, we feel it is fair to encourage those using Mergin Cloud’s free tier for commercial gain to support us with one of our affordable subscriptions. The link also has a number of frequently asked questions relating to this change.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

We are excited to announce that the geodiff library has finally reached version 1.0. We have started to develop geodiff back in 2019 as a part of our efforts to allow synchronisation of changes between the Input mobile app and Mergin platform.

At the core, geodiff library provides functionality to:

• compare a pair of GeoPackage databases and create “diff” files containing changes between them
• apply a “diff” file to a GeoPackage database
• rebase changes in a “diff” file
• invert, concatenate diffs and other utility functions

Thanks to the above low-level operations, any changes to data stored in spatial/non-spatial tables in GeoPackages can be easily transferred to others and applied. And thanks to the “rebase” functionality - inspired by source code management systems like git - we can automatically merge changes from multiple users capturing data offline in Input/Mergin (see our recent blog post that covers rebasing for more).

The library is written in C++, providing stable C API and offering Python bindings as well (look for pygeodiff package in pip). It also comes with a command line interface tool geodiff covering all major features. The whole package has a very permissive MIT license.

Support for drivers

Initially, geodiff library only worked with SQLite / GeoPackage files. This has changed with the version 1.0 - geodiff supports drivers, allowing use of different database backends to compare and apply diffs. In the 1.0 release we have added PostGIS driver in addition to SQLite/GeoPackage driver.

This means that users can compare tables or apply diffs in PostGIS databases using the same APIs as with GeoPackages. And not only that - diff files are compatible across different drivers. That means it is possible to take a diff file from a GeoPackage and apply it to PostGIS database!

Using the PostGIS driver we were able to create mergin-db-sync tool as a companion to Mergin platform. With DB sync, one can keep a local PostGIS database always in sync with a project in Mergin, supporting automatic transfer of changes from Mergin to PostGIS and the other way round as well - from PostGIS to back Mergin.

Try it

The library is hosted on GitHub in lutraconsulting/geodiff repository. We would love to hear your feedback!

Stay tuned for more!

As announced earlier, next week we will be open sourcing Mergin, our platform for easy sharing of spatial data in teams (whether they are in office or in the field). If you have not heard about Mergin platform yet, please have a look at the Mergin website, try Mergin plugin for QGIS and Input app, a mobile app based on QGIS for iPhone/iPad and Android devices. Since the initial release in early 2019, Mergin and Input have been used by thousands of users around the world.

At Lutra Consulting, we are dedicated to improving free and open source software for geospatial. We will be releasing Mergin as open source to solve another missing piece in the puzzle, providing open source end-to-end solution for mobile data capture for QGIS users. Watch our blog and Twitter for further updates!

People often ask whether the Mergin service and the Input app can deal with multiple team members doing edits at the same time.

They happily find out that the answer is yes - it is one of the core features that greatly simplifies life of our users. In this post we will shed some light on how things work behind the scenes.

Getting started

Let’s think of a simple project directory that needs to be synchronised between multiple desktop or mobile users, containing just two files:

• a QGIS project file my-project.qgz that sets up map layers, styling, …
• a GeoPackage file my-data.gpkg containing all GIS data

Our sample GIS data will contain a tree survey table, containing location, species and age of various trees:

When users edit data in my-data.gpkg, the traditional cloud storage solutions (such as Dropbox, Google Drive, Microsoft OneDrive and others) simply copy the modified files there. They do not understand the file content though - so if two people modify the same file, they have no way of knowing how to merge changes together. In the worse case, when two versions of the same file are uploaded, they keep just the version which was synchronised last. Or slightly better, they resort to creation of conflicting copies which need to be manually merged later. As one can imagine, merging and consolidating modifications from multiple GeoPackages back to one copy is a slow, tedious and error-prone job.

the Mergin service has been designed to understand spatial data, especially GeoPackages that are becoming the most popular format to store vector & attribute data. This is thanks to the open source geodiff library that we have developed while working on Mergin.

Synchronising data using “diffs”

The first trick is that synchronisation of GeoPackage files between Mergin server and clients (Input app, QGIS or other apps) only transfers actual changes in tables (“diffs” in technical jargon).

Our Mergin project with the tree survey has been prepared and downloaded by users. Jack did a field survey and he added or updated some rows in the survey table (changes highlighted in yellow and green):

After pressing sync button, his changes are detected and uploaded to Mergin, encoded as a list of changes to the survey table:

Another user, Jill, also downloaded the tree survey project to her mobile device prior to Jack’s changes. When Jill synchronises the project to get the latest version, the changes as uploaded by Jack are downloaded and applied to her local copy of the project, getting the same data as seen by Jack.

At this point, the advantage of uploading/download only changes in tables may not seem obvious besides saving some network bandwidth… Read on to learn how this is used to support multi-user editing.

Merging changes from multiple users

So far we have expected that Jill does not have any pending changes to sync, so that was easy. Now let’s assume that Jill has also done some changes on her device:

Here comes the more tricky part - how do we merge changes from Jack and Jill back to a single table:

In Mergin, cases that require merging changes from multiple users are handled by the “rebase” operation, a concept we have borrowed from version control systems for source code.

Let’s assume that Jack has synchronised his changes first. Later, when Jill synchronises her changes, a couple of things happen on her device before uploading the changes: Jill’s changes will be temporarily undone, Jack’s changes get applied, and finally Jill’s changes are re-applied after being rebased on top of Jack’s changes.

What does it mean to rebase someone’s changes? There are a couple of possible edit conflicts that could happen between rows of a database table with matching IDs (insert/insert, update/delete, delete/delete, update/update). These edit conflicts need to be resolved in order to be able to merge changes from multiple users.

In our example, both Jack and Jill have added a row with ID = 4. This is not allowed, and therefore Jill’s new row ID will get changed to ID = 5 (any unused ID would do). As a result, here’s how the merged table will look at the end - combining changes of both users:

If both Jack and Jill modified the same row (the update/update edit conflict), we can only accept one edit automatically. The conflicting edit of the other user is written to a special conflict file and uploaded to Mergin, so no data gets lost, and the conflict can be later inspected by the project admin. Fortunately, this kind of conflict does not happen often if the survey work is well planned to avoid users simultaneously modifying the same features within the GeoPackage data.

What if conflict files appear

There are some cases when automatic merging is not supported. In those cases, Mergin is unable to find out details about changes in the data file(s) and has to resort to creation of a conflicting copy which gets uploaded to Mergin project along the original data file(s). In particular the problems may appear when:

• Other format than GeoPackage is used for data storage (e.g. shapefiles)
• Database or table structure is changed (e.g. adding new columns or new tables)

In the future, these limitation may be removed, but at this point it is good to keep them in mind.

If you plan to change structure of the survey tables and the project is already being used on multiple devices, it may be a good idea to create a new Mergin project with the modified database structure and instruct users to switch to the new project. Otherwise conflict files may start popping up as long as some users have older version of the project, adding more manual work to collate data.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

It happens that when you collect data and then checking it in the office on your desktop using QGIS, the points are misplaced and shifted. This blog post explains the possible root cause of this issue? (Read time 5 min)

This article follows the help document about the projections and coordinate reference system handling in QGIS.

The accuracy of the captured points is affected by two main factors

• GPS receiver and accuracy
• Selection of coordinate reference systems used in the project

Lets take a closer look at both problems in context of field surveys.

GPS accuracy

GPS accuracy depends on the quality of your GPS receiver and the number of visible GPS satellites your receiver can use at the moment of capturing the point.

When you load the map in Input app, the bottom bar contains the GPS marker with a coloured dot. The dot can have the following three colours:

• green: The actual accuracy is below the threshold
• orange: The actual accuracy is above the threshold
• red: The GPS is unavailable

For different use cases, acceptable accuracy is different. So the threshold for the colour scheme could be adjusted in the Input app settings. Always adjust the settings based on your project needs and check the GPS accuracy when capturing data.

The GPS receiver itself can be either improved by usage of device with a better internal hardware or usage of powerful external GPS receivers If you want to use the external GPS receiver, read the following help article for setup.

Projection problems

First of all, there could be multiple problems with setup of coordinate reference systems and transformations. To solve it, you need to load the project in QGIS, using Mergin Plugin and check for various situations as described in this help article.

Input app checks for missing datum shift files or other transformation problems and notify you once at the project load by following message.

To be able to use the datum grid shift in your mobile app:

• Copy the proj folder from your QGIS working directory (see help page for different platforms)
• Paste the folder to your survey project downloaded fromt he Mergin server
• Sync the changes

For our example with the British national grid on Windows, the grid file is called OSTN15_NTv2_OSGBtoETRS.gsb and if the Mergin project is in C:\users\qgis\myproject then it should be copied to C:\users\qgis\myproject\proj directory and synchronised by the Mergin service to the Input app.

Notes:

• This is one-off process. Input app will transfer the grid shift datum to its own working directory on your mobile device.
• There might be more than one datum grid shift in your QGIS working directory. Use the appropriate one for your survey project.

You can find a short summary of this article on the Input app help pages

In this post, we will explore the new feature in the Input app which allows you to record and display photo’s geotag information (e.g. direction, longitude and latitude).

Photos’ metadata

When users take photos in the field, they often would like to record extra information which can help convey a better understanding and awareness of the surroundings. Recent mobile devices allow users to record photo direction, location, time stamp, camera settings, etc when taking photos. This information is often optional or restricted by default (due to privacy) within the mobile settings.

For surveying and data collection, this information can add extra dimension and context. For example, if you are surveying a bat nesting, it is useful to know the directions of the photos you have taken. This will help identifying the site easily in the subsequent site visit.

Geotag information is also useful metadata to have attached to your photos. There are other GIS or non-GIS applications which can read and interoperate the information.

In the recent version of the Input app, we have added a feature which allows you record and display the geotag information. Combined with QGIS styling, you can create a very informative symbology to display the information while you are capturing it in the field.

Before you start

To be able to capture geotag information, you need to enable your Camera app to use location. For that:

• in Android, from Settings > Apps and notifications > (see all) Camera and under Permission ensure you allow Location

  

• in iOS, from Settings > Privacy > Location Services > Camera and the Precise Location is enabled.

  

Quick start

If you want to record photos with directions as shown below, you can follow the following steps:

• Log into Mergin (or sign up if you have not yet registered)
• Brows to photo survey project
• On the top right-click on Clone to make a copy of your project in your account. You can choose a different project name:

• Download and install the Input app

• Select My projects and log in using your Mergin credentials

• Download the newly created project from the earlier step

• Start recording points and add photos, you should see photo direction

More details

The project configuration is done in QGIS. The form has been set up to allow you take photos using Attachment widget. To learn more about different form edit tools in Input app and how to set the up in QGIS, see the Input app help pages.

There are multiple fields to extract geotag information from the photo, as soon as you add the photo either using the camera or the gallery:

• Direction: read_exif_img_direction(@project_home + '/' + "photo")
• Latitude: read_exif_latitude(@project_home + '/' + "photo")
• Longitude: read_exif_longitude(@project_home + '/' + "photo")

As noted, all the above functions take the path to the photo (@project_home + '/' + "photo") and return different metadata related to the image.

In addition to the form, the layer styling has been modified to resemble the direction of the camera and field of view.

To set the camera icon and direction of the camera:

• For the point styling, select SVG Marker
• From the list of SVGs, select the camera icon. (ensure to embed the SVG to the project, otherwise it will appear as a question mark in the Input app.)
• For the Rotation, select direction field

To add field of view:

• Add another symbol layer to your point
• For Symbol layer type select Geometry Generator
• For Geometry type select Polygon / MultiPolygon
• For the expression, type: wedge_buffer(center:=$geometry,azimuth:= "direction",width:=45,outer_radius:=0.0008)
• For Symbol layer type, select Shapeburst Fill

Feedback and suggestions

Input, a field data collection app based on QGIS. Input makes field work easy with its simple interface and cloud-based sync. Available on Android and iOS.

If you would like to add a new feature or suggestions to improve the app, do not hesitate to contact us on [email protected]

Just few weeks ago, QGIS 3.18 has been released - the first version to include support for point cloud data, thanks to the great support from the QGIS community in our joint crowdfunding campaign with North Road and Hobu.

We have received a lot of feedback from users and we would like to summarise the most common problems people have faced, and add a couple of tips on useful features that are otherwise easy to overlook.

1. I am unable to load LAS/LAZ files

If your QGIS installation does not recognize .las or .laz files (which are the two point cloud formats that we currently support in QGIS), the most likely source of the problem will be that the PDAL library needed for reading LAS/LAZ is missing in your installation. Unfortunately not all installers include it at this point. Particularly on Windows, there are several options how to install QGIS, and only one choice is the right one for PDAL support. At the time of writing (April 2021), you need to download installer from section called “Standalone installers from OSGeo4W testing packages (MSI)”, here’s a screenshot of what to look for:

On macOS the official all-in-one installers include point cloud support. On Linux, PDAL support depends on the particular distribution/packages, but probably most of them include point PDAL library.

2. Point cloud is completely flat in 3D

It may happen that you load a point cloud layer, then open the 3D map view and the point is displayed as a flat surface like on this screenshot:

The reason for this is that the 3D renderer is not enabled for your point cloud layer. To enable 3D renderer, open the Layer Styling panel (there’s F7 shortcut to open it!), then switch to the second tab (“3D View” - with a cube icon) and change the “No Rendering” option to some other option - for example “Classification” in case your point cloud is classified. You should then see your point cloud in 3D.

3. Point cloud is rendered twice - in 3D and in 2D (“flat”)

Commonly when people open 3D view with a point cloud, they may see the point cloud rendered twice, like in the following screenshot:

The reason is that both 3D rendering and 2D rendering of point cloud is enabled, and therefore the layer is also rendered as a 2D map texture on top of terrain (which is flat by default). An easy way how to fix this is to set 2D rendering of the point cloud layer to “Extent Only” in Layer Styling panel (in the first tab):

If the dashed rectangle marking the extent is still bothering you, it is possible to change the line symbol to use white (or transparent) colour.

Hopefully in near future we would address unexpected behaviour and layers with a 3D renderer defined would not be rendered as 2D.

4. I still can’t see my point cloud in 3D view

It could happen that if your point cloud is for a small area, yet the elevation of points is relatively high: when you first open 3D view or when you click “Zoom Full” button, the view may get zoomed too close and the actual point cloud data may be behind the camera. Try zooming out a bit to see if it helps. (This is a bug in QGIS - at this point “zoom full” ignores extra entities and only takes into account terrain.)

5. Enable “Eye Dome Lighting” in 3D view

For a much better 3D perception of your point cloud, try clicking the “Options” button (with a wrench icon) in the toolbar of 3D view and enable “Show Eye Dome Lighting” in the pop-up menu. This will apply extra post-processing that adds slight shading based on the positions of nearby points, and adds silhouettes when there is a sudden change in depth:

As soon as you zoom into your point cloud more, to the point when individual points can be seen, the eye dome lighting effect will start to disappear. You can try experimenting with the point size (in Layer Panel, 3D View tab) - increasing the point size will help.

- A view with point size 2.0 pixels

- The same view with point size 6.0 pixels

6. Try the new camera navigation mode in 3D view

In QGIS 3.18 we have added a new “Walk Mode” camera navigation mode that is much better suited for inspection of point clouds (compared to the default “Terrain Based” camera navigation mode). Open 3D map view configuration dialogue, pick “Camera & Skybox” tab and set it here:

7. Use elevation scaling and offset to your advantage

Sometimes it is useful to modify offset and/or scale of the elevation of points (Z values). For example, if the point elevations do not match your other 3D data, or maybe you have source data where X,Y coordinates are in meters and Z coordinate is in feet!

Another case when this can be useful, is when your point cloud data are further away from the terrain and the default “Terrain Based” navigation mode does not work nicely - it expects that data are near the terrain, and the camera rotates around a point terrain, which may feel strange when browsing point clouds. A workaround is to apply offset to the point cloud layer to move the points near the terrain. For example, this is a point cloud which is roughly at 200 meters elevation (the grey plane is the terrain):

When an offset of -200 is applied to the point cloud in Layer Styling panel, data show up much closer to the terrain and camera navigation feels more natural:

8. Try circular points in 2D maps

By default QGIS draws points as squares as this is the fastest option. But for higher quality output you may want to change point symbol style to circle in Layer Styling panel, which makes things look a little less jagged:

– using squares – using circles

At this point we always use square points in 3D views - in the future we will likely offer circular points in 3D views as well.

9. Give us your feedback

We would love to hear from you about your experience with point clouds in QGIS so far, what features you are missing or what problems you have encountered - feel free to drop us a mail at [email protected]. If you think you have found a bug, best to file an issue for it in the QGIS GitHub repository.

With the QGIS 3.18 release, we know we are only at the beginning of a long journey to provide great point cloud support to the users. The decreasing cost of laser scanning (LIDAR) hardware and increasing availability of photogrammetric methods means that we will be be seeing point cloud data in GIS more and more often. There is still a lot of functionality missing for efficient work with raw point clouds, for easy processing of data and we are still using only a fraction of what the PDAL point cloud library offers. We are dedicated to provide first class support for point clouds in QGIS - however this cannot be done without funding. Therefore, if you would like to help us to push point clouds in QGIS to the next level, please do not hesitate to contact us at [email protected].

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

Based on the feedback from users, it appeared that packaging and creating a stand-alone project for using on the Input app were sometimes cumbersome and confusing. Existing projects have links to several layers and those layers can be spread over different network drives or folders.

With the new release of the Mergin plugin for QGIS, we have introduced a project wizard, which can help users to package and upload the project to the Mergin service. There is also a new toolbar to simplify your workflow.

Easy access to Mergin tools

The plugin now comes with a new toolbar. The common tasks (syncing the project/data and checking the status of the project) are now available from the toolbar.

Project wizard

To create a new project, you can start from a blank project or alternatively open an existing project. You can then select Create Mergin Project from the toolbar:

A new window will appear which should give you three options (the last two options are available only if you have an existing project open):

• New basic QGIS project: if you are new to QGIS, this is a good starting point. With this option, a project will be created. Within the project there will be a survey layer (a point layer) and background map (OpenStreetMap).

• Package current QGIS project: this option will create a copy of your project and copies all the files to a single folder. The wizard tries to guess each format and offers users three options to package the layer, keep as is (i.e. the layer will be referenced as is in the new project) or ignore (the layer will not be included in the new project). The default for each layer type is as: the web services (e.g. WMTS, XYZ tiles, vector tile layers) will be referenced in the new project as they are. Vector layers will be all written to Geopackage format (each vector in one Geopackage database). Raster layers will be copied as they are. The layers will be referenced in the new project accordingly.

• Use current QGIS project as is: this is for cases when you have already a stand-alone folder with your projects packaged.

In the next window, you will be prompted to assign a project name and select a path where your project folder and associated files will be generated (this option is only available for the first two choices):

After the wizard, the new project will be created locally and on the Mergin server.

Validation and status check

It is recommended to run the project status after changing your layers and project. This will help getting a list of pending changes and also see any warnings or validations of your project. The warnings are related to restructuring of a Geopackage layer (adding/removing a field or addding/removing a layer in a Geopackage database). Validations can be linked to missing layer or availability of a layer when working offline:

Reporting issues

If you have any issues or suggestion to improve the plugin, you can file a ticket on the Github project repository.

What next

Your project is ready to be used in Input. Get the app for your Android or iOS device. Log into the Mergin service and download the project(s) created in the above section on your device.

QGIS 3.18 is finally here. This will be the first release of QGIS with native support for point cloud data. This work was made possible by generous contributions from the community.

(Data from UGKK SR, made by Tibor Lieskovsky)

Contributors

When we announced the campaign in August 2020, the response was overwhelming and within weeks, we managed to reach and then exceed the amount required.

Below is the list of contributors in no particular order:

Mapfly, Ujaval Gandhi from Spatial Thoughts, National Land Survey of Finland, Daniel Löwenborg, BNHR, Imapct GIS, Andreas Neumann, Kanton Solothurn, Switzerland, City of Vevey, Mapping Automation, Service de la géomatique du canton de Neuchâtel, Hans van der Kwast, 3DGeoCloud, Rudaz+Partner AG, Leonard Gouzin, TileDB, Inc.

We wish to also thank the many anonymous contributors who do not appear in this list.

Note: if you have contributed to the campaign but your name does not appear here, you may not have selected the option to be listed in the campaign form. Please contact us if you’d like to be listed.

In addition to financial contributions, we’d like to extend our gratitude to all those who helped spread the word, helped with the testing and provided feedback and sample data.

New features

This is a brief summary of new features from our joint work with North Road and Hobu:

• Introduction of a new map layer type: a point cloud layer
• Load LAS or LAZ files (either by drag’n’drop or by opening files in Data Sources Manager)
• Load EPT datasets by pointing QGIS to their ept.json file (currently only supporting datasets on the local drive)
• Support for rendering point cloud data in 2D and 3D map views
• Apply various rendering styles in 2D and 3D:
  • “Attribute by Ramp” - draw data based on a single attribute and a color ramp (similar to “Graduated” styling for vector layers)
  • “RGB” - draw data using colors assigned to the points (combining red/green/blue attributes)
  • “Classification” - draw data using different colors for different classes (ground, buildings, vegetation, …), also allowing display of only desired classes
  • “Extent only” (2D only) - draw only bounding box of the point cloud
  • “Single color” (3D only) - draw all points with a single color
• Set size and shape of points
• Manually adjust scaling and offset of elevation (Z values) - if needed to match with elevation of other data
• Point cloud layer properties dialog to see metadata of the point clouds
• Identify tool supports point cloud layers and shows all attributes of picked points
• 2D and 3D views only render a subset of the point cloud for the best performance for the given view (for geeks - this is thanks to indexing to octree data structure, using EPT format written by untwine tool packaged with QGIS)
• Optimize the quality and performance of the 3D view using point budget configuration, which limits maximum amount of point rendered at any time (set to 1 million by default)
• Enable eye-dome lighting in 3D views for much better depth perception of point clouds
• New “Walk mode” camera navigation in 3D views - there is now a switch between the original “Terrain based” navigation mode and the new mode, which allow easier navigation through point cloud data

Please note that as this is the initial release (with over 10 thousand lines of new code related to point clouds), there may be still some rough edges here and there, or some data may not load or display correctly. In case you encounter any issues with the new functionality, please let us know - do not hesitate to create a new QGIS issue

(Data from Helsinki City)

Future work

This has been the start of a larger effort to bring full support for point cloud data into QGIS. We, in collaboration with North Road and Hobu are developing requirements for integrating point cloud data processing and analysis, more data formats, better visualisation, profile tools etc. in future releases of QGIS.

If you’re interested in helping shape those requirements or funding such features, please contact us at [email protected].

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

In the past months, we have been busy adding support for point cloud data in QGIS (3.18). Read more on how to install and test this feature under MS Windows.

Installation

To be able to use this feature, you need extra packages and also the latest OSGeo4W installer.

Update

Since the original post, Jürgen Fischer has created a stand-alone MSI installer. You can download the new installer from the QGIS website. The installer is only for 64-bit platforms and does not support MS Windows 7.

Note that there have been several regressions and bugs with the first release of QGIS 3.18.0. The issues are being addressed and soon there will be an updated version available. The above link is only for those who are eager to test the point cloud data in QGIS.

Installation

To be able to use this feature, you need extra packages and also the latest OSGeo4W installer.

Note: This is a completely revamped and different packaging system than the current OSGeo4W installer. To avoid any clash with your current installation, it is recommended to use different paths for temporary download files and installation of the new packages. The new packaging only supports 64-bit platform.

1- Download and run the NEW OSGeo4W installer

2- Select the Advanced install and pick qgis-dev from the list of packages

Special thanks to Jürgen Fischer for his hard work on preparing the new packages for Windows.

Testing

Once installation is completed, try to run QGIS from the installation path (e.g. C:\OSGeo4W64\bin\qgis-dev.bat). You should be able to load LAS\LAZ file to your map from the Browser panel or the Data Source Manager.

The point cloud data can be visualised in 2D and 3D map canvas.

Please test and let us know if you encounter any problems when loading, viewing or styling point cloud data. The best way to do that is to create a new issue on GitHub: https://github.com/qgis/QGIS/issues

QGIS 3.18 will be released later this week (February 19), so grab your copy of QGIS today and give it a try, so that we can fix any remaining issues before the release!

Troubleshooting

Problem: I am unable to add any LAS/LAZ point cloud file

Solution: Ensure you have used the correct installer linked above. Development builds of QGIS in the ordinary OSGeo4W installer DO NOT include support for point clouds.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

The case study presents the implementation of the QGIS FLO-2D Plugin project (5 minute read)

Introduction

FLO-2D is one of the most widely used commercially available flood models. FLO-2D is capable of simulating urban flooding in high resolution including storm drain systems.

In 2016 the FLO-2D team invited us to develop a set of tools for optimising the flood model build process in QGIS. The resulting plugin allows hydraulic models to be built quickly by leveraging the wide range of tools available in the QGIS ecosystem.

The plugin can be downloaded by following the guidance in the plugin documentation

As the use of open source GIS grows within the water engineering sector, Lutra Consulting develops and maintains MDAL which makes the visualisation and post-processing of time-varying numerical model results possible in QGIS.

Contact us at [email protected] if you’d like to discuss the benefits of integrating your flood modelling software more tightly with QGIS.

Overview of Previous GUI tools: FLO-2D GDS

FLO-2D’s focus on urban modeling requires large datasets that may include several million grid cells. Each cell has between 4 and 10 attributes so datasets can often be in the range of several gigabytes.

Previously, Grid Developer System (GDS) was used to pre-process the spatial data used by FLO-2D. Being a 32-bit application, the GDS was only able to load up to 4 gigabytes of data and its programming framework was also no longer maintained by Microsoft. Modellers could only be built on Windows PCs.

All development and maintenance of the GDS was carried out by the FLO-2D team.

Behind the Implementation

Guiding principles behind the new QGIS Flo-2D plugin

We assessed the tools used at the time and in close collaboration with the FLO-2D team, optimised the workflow from the users’ point of view with emphasis on speed and simplicity.

To keep things simple for the users, we aimed to provide a level of abstraction so users would no longer need to be involved with the internal structure of FLO-2D solver input files. They could instead focus on real-world aspects affecting their models.

Together with the FLO-2D team we designed a solution that was based on 3 core ideas:

• Modellers will use native QGIS tools and point/line/polygon layers to define real-world objects (e.g. domain, boundary conditions, levees, …)
• These real-world objects will be converted automatically into the data structures required by the FLO-2D solver (although expert users can still modify these data structures if they so wish).
• Additional productivity tools will be provided to allow users to speed-up time consuming tasks

Features of the plugin

Some of the key features of the solution are listed here:

• Easy creation and handling of new models (all model data sits in a single GeoPackage file)
• User friendly and intuitive digitizing and manipulation of model components (through simple GIS layers and dedicated tools) including:
  • 1D domain
  • Boundary and initial conditions
  • 1D channels and cross-sections
  • Levees
  • Rainfall
  • Infiltration areas
  • Storm drains
• Various options for obtaining grid data such as elevation, roughness, reduction factors etc. from different sources (raster layers, external vector layers etc.)
• Plotting profiles and editing time series data
• Automated tools for schematizing input layers into the GDS format
• Importing basic data from HEC-RAS models
• Import / export functionality between GeoPackage and GDS format (*.DAT files)
• Running external FLO-2D tools (FLO-2D Pro engine etc.) directly from QGIS

Benefits for the FLO-2D developers:

Benefits for the developers of FLO-2D include:

• Reduced development and maintenance costs, since much of the heavy lifting of the FLO-2D plugin is done by QGIS itself
• By being part of the QGIS ecosystem, gaining opportunities to approach QGIS users in the flood risk industry to use FLO-2D software
• The FLO-2D plugin is developed on GitHub, allowing the latest development technologies such as continuous integration, automatic testing and issue tracking to be used
• Ability to solve upstream bugs in QGIS or MDAL due to the open-source nature of the projects

Benefits for FLO-2D users:

Some of the benefits realised by FLO-2D customers include:

• Being able to work with their FLO-2D models using open source GIS on all major operating systems
• A full GIS application to support their data pre-processing
• Logical and intuitive workflows
• FLO-2D results can now be visualised and post-processed natively in QGIS via mesh layer
• Ability to use all native QGIS support and development channels in addition to FLO-2D support
• Integration of internal workflows with powerful native QGIS features including projection support, GDAL/OGR integrations, background maps support (e.g. vector tiles), printed flood maps, etc.
• Ability analyze results via QGIS’ Crayfish plugin and produce graphs and outputs

The plugin in action!

Further Reading

• FLO-2D webpage
• MDAL and Mesh Layer

Do you have any questions or would like to see demo of QGIS Mesh Layer? Contact us at [email protected] or schedule a demo call calendly.com/saber-razmjooei/15min

Key words

QGIS, plugin, python, migration, optimised, speed up, fast, hydraulic modelling, water, 2D, open-source, cost reduction, software development

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.

Preparing reports with lots of survey photos takes time - this plugin automates the process.

Nowadays, it’s really easy to take georeferenced photos on site visits. Tools like Input can capture photos, descriptions and location information and bringing it all into GIS is straight-forward. However.. I recently discovered that people still spend significant amounts of time organising their photos into reports outside of GIS. This small plugin automates the process of getting the photos and their descriptions into Microsoft Word.

For this blog post I knocked-up a simple survey project based on the Field notes Mergin template. Points can have a photo, title and description amongst other fields. No comments on my gardening skills please!

When I sync the data back to QGIS, the attribute table looks something like this:

We’ll now use the HTML Table Exporter plugin to export the layer as an HTML table (this will let us get it into Word).

First install the plugin.

It’s an experimental plugin so you’ll first need to check the Show also experimental plugins option under Plugins > Manage and Install Plugins… > Settings.

When you have installed the plugin, open it using this button or via Plugins > HTML Table Exporter > Export table as HTML.

You should now see this:

Set Table to the layer you want to export. The image scaling option is described later, leave it at a small setting for the time being. Click OK to export and tell the plugin where to save the HTML file.

Now the data should be out of QGIS.. the next steps are in.. MS Word :o

Right click on the exported html file and open it with Microsoft Word:

Let’s see what it looks like:

OK.. we’re getting somewhere! At this point you’ll want to:

• Enable Print Layout mode in Word so you can see what the printed page will look like
• Set the page orientation as desired (I chose landscape in the image above)
• Experiment with image rotation if required (Right click photo > Size and position > Rotation)

If playing with rotation, just focus on rotating a single image, we’ll batch rotate later as required. For now you want to get a feel of whether you want to adjust the scale factor in QGIS (to make the images smaller or larger) to save you having to resize them individually in Word.

I’ve decided to increase the scale factor from 10% to 15% so will now re-export.

Beware that Word has an exclusive lock on the html file when it’s open so you need to close it in Word before you can export it again from QGIS.

I settled for 10% in the end so I could get multiple images on each page in portrait mode. After removing the columns I didn’t want, the table in Word now looks like this:

To rapidly rotate images by 90 degrees, rotate the first one using Right click on photo > Size and position > Rotation then select subsequent photos and press the F4 key. This method is described in more detail here as well as other Word batch image rotation methods.

My document is almost finished. There are just a few small issues to iron out. Currently, the images are referenced by the Word document, not embedded. This means if I email the document to someone, the images will be missing. Let’s fix that by embedding the images in the word document.

First save the document as a Word document in Word’s native format (e.g. *.docx).

Next, locate the Edit Links to Files option:

Select all the linked images (the shift and arrow keys help here) and check the Save picture in document option and click OK:

Save the document, your photos should now be embedded within the document.

If you find your word document gets huge, you can use the method here to quickly batch compress all images in the document.

Input is a free and open source field data collection and mobile GIS app based on QGIS.

If this guide saved you some time and you feel like doing something awesome for us, a review of Input on the Apple App Store or Google Play Store would be really well appreciated.