Last update:
Mon Mar 27 05:00:26 2023
•
•
•
•
This is a guest post by Mickael HOARAU @Oneil974
As an update of the tutorial from previous years, I created a tutorial showing how to make a simple and dynamic color map with charts in QGIS.
In this tutorial you can see some of interesting features of QGIS and its community plugins. Here you’ll see variables, expressions, filters, QuickOSM and DataPlotly plugins and much more. You just need to use QGIS 3.24 Tisler version.
Here is the tutorial.
Today’s post is a follow-up and summary of my mapping efforts this December. It all started with a proof of concept that it is possible to create a nice looking snowfall effect using only labeling:
After a few more iterations, I even included the snowflake style in the first ever QGIS Map Design DLC: a free extra map recipe that shows how to create a map series of Antarctic expeditions. For more details (including project download links), check out my guest post on the Locate Press blog:
If you want to just use the snowflake style in your own projects, the easiest way is to grab the “Snowy Day” project from the QGIS hub (while the GeoPackage is waiting for approval on the official site, you can get it from my Dropbox):
The project is self-contained within the downloaded GeoPackage. One of the most convenient ways to open projects from GeoPackages is through the browser panel:
From here, you can copy-paste the layer style to any other polygon layer.
To change the snowflake color, go to the project properties and edit the “flake_color” variable.
Happy new year!
The Central Institution for Meteorology and Geodynamics (ZAMG) is Austrian’s meteorological and geophysical service. And as such, they have a large database of historical weather data which they have now made publicly available, as announced on 28th Oct 2021:
The new ZAMG Data Hub provides weather and station data, mainly in NetCDF and CSV formats:
I decided to grab a NetCDF sample from their analysis and nowcasting system INCA. I went with all available parameters for a period of one day (the data has a temporal resolution of one hour) and a bounding box around Vienna:
The loading screen of QGIS 3.22 shows the different NetCDF layers:
After adding the incal-hourly layer to QGIS, the layer styling panel provides access to the different weather parameters. We can switch between these parameters by clicking the gradient icon next to the parameter names. Here you can see the air temperature:
And because the NetCDF layer is time-aware, we can also use the QGIS Temporal Controller to step through the hourly measurements / create an animation:
Make sure to grab the latest version of QGIS to get access to all the functionality shown here.
The QGIS 3.22 release is just around the corner, and we’d love to introduce you to one of the exciting changes included in this version. Just like all QGIS major releases, 3.22 brings a whole swag of improvements designed to enhance and simplify your mapping needs. In this post we’ll be highlighting one of these improvements — “Annotation Layers”. Before we dive further in, we need to extend our heartfelt thanks to the Swiss QGIS User Group who provided the funding required for this work. The Swiss User Group has invested heavily in cartographic enhancements to QGIS over many years, and it’s great to see this tradition continue for the 3.22 release!
So… let’s dive into Annotation Layers in QGIS 3.22, and what this new functionality allows you to do! Before we can get started, we’ll need to enable the new “Annotations” toolbar (it’s hidden by default). You can do this through the “View” – “Toolbars” menu by checking the “Annotations Toolbar” option. This will show a new toolbar, full of useful actions for creating and working with annotation layers:
Annotation layers can contain a whole mix of different annotation types. In QGIS 3.22 this includes markers, lines, polygons and text. (We anticipate this list will grow steadily in future QGIS releases!)
Creating an annotation is as easy is picking the desired item type (marker, line, polygon or text), and then drawing directly onto your map:
All the usual QGIS shortcuts for creating features apply when creating annotation items. A line or polygon annotation is drawn by left-clicking once for each vertex, with a final right mouse click to complete the shape. Snapping can be enabled while you draw, you can use the “Advanced Digitizing Tools” to precisely place vertices, and even switch the drawing tools to the streaming mode for completely free-form shapes!
While annotations are created in this free-form way, every annotation item is still completely geo-referenced and tied to a particular geographic location. Moving your map, changing the scale or changing projection won’t cause your annotations to jump around the map. Rather, they’ll be locked in place to the location you’ve drawn them:
Unlike features in a traditional vector layer, it’s easy to interactively style annotation items on an item-by-item basis. (This is what makes them a great tool to help quickly create beautiful maps!). All the item appearance properties are set through the QGIS “Layer Styling” dock, which allows you to interactively tweak an item’s appearance without any annoying dialog windows getting in your way. Annotation items support all of QGIS’ rich symbology and labelling options, including layer effects (drop shadows, inner/outer glows), blending modes (multiply, overlay, etc), opacity, and even data-driven symbol property overrides!
Unlike traditional vector or raster (or mesh, or point cloud!) layers, an annotation layer isn’t linked to any external dataset. They exist within a single QGIS project only, so they are perfect for project-specific markups and cartographic notes. By default, any newly created annotation item will be placed on top of all your other map content. (Formally, they are added into a “main annotation layer” which is always drawn on the very top of the map.) This main annotation layer is a special beast, and you won’t see it listed alongside all the other layers in your project. If you need to interact with it, say to change a layer-level setting like the opacity or blend mode, just select the “Main Annotation Layer Properties” option from the annotations toolbar:
In addition to this special main annotation layer, it’s also possible to create your own secondary annotation layers. From the annotations toolbar, select the “New Annotation Layer” option and a brand new annotation layer will be added to your project. Unlike the main annotation layer, you’ll see any secondary annotation layers appear in the list of your project’s layers. Just like any other map layer, you can toggle their visibility, rename them, and even reposition them to control whether the annotations show above or below particular layers in your map!
(Note that whenever you have a secondary annotation layer selected as the current layer, any newly drawn annotation items will be placed into this layer instead of the main annotation layer.)
So there we go — that’s a quick dive into annotation layers in QGIS 3.22, and some of the new cartographic capabilities they unlock for your maps! We see this as the starting point of an exciting journey, and North Road developers have many plans for further enhancement of annotation layers in future QGIS releases. (And naturally, we’ve also added full support for ArcGIS drawings to QGIS annotation layer conversion to our SLYR ESRI conversion toolkit!) If you’ve something specific you’d love to see added to annotations layers, just contact us at [email protected] to discuss sponsorship options.
Kappazunder is the city of Vienna’s database created during their recent mobile mapping campaign. Using vehicle-mounted Lidar and cameras, they collected street-level Lidar and street view images.
Yesterday, they published a first sample dataset, containing one trajectory on data.gv.at. The download contains documentation, vector data (.shp), images (.jpg), and point clouds (.laz):
The shapefiles contain vehicle location updates, photo locations, and areas describing the extent of the point clouds. Since the shapefile lack .prj files, we need to manually specify the correct CRS (EPSG:31256 MGI / Austria GK East).
The vehicle location updates and photo locations contain timestamps as epoch. However, the format is a little special:
To display a human-readable timestamp, I therefore used the following label expression:
format_date( datetime_from_epoch( "epoch_s"*1000), 'HH:mm:ss')
Adding these labels also reveals that the whole trajectory is just 2 minutes long. This puts the download size of over 5GB into perspective. The whole dataset will be massive.
The .laz files are between 100 and 200MB, each. There are four .laz files, even though the previously loaded point cloud extent areas only suggested three:
Loading the .laz files for the first time takes a while and there seem to be some issues – either on the user end (me) or in the files themselves. Trying to load content of the ept_ folders only results in very few points and multiple “invalid data source” errors:
For the few point that are loaded, it looks like the height information is available:
Update on 2021-10-01: I’ve reported the data loss issue and Martin Dobias has provided a first work-around that makes it possible to view the data in QGIS:
The street view images are published as cubemaps. Here’s a sample of the side view:
Sorry, this entry is only available in Dutch.
One of the new features in QGIS 3.20 is the option to trim the start and end of simple line symbols. This allows for the line rendering to trim off the first and last sections of a line at a user configured distance, as shown in the visual changelog entry.
This new feature makes it much easier to create decorative label callout (or leader) lines. If you know QGIS Map Design 2, the following map may look familiar – however – the following leader lines are even more intricate, making use of the new trimming capabilities:
To demonstrate some of the possibilities, I’ve created a set of four black and four white leader line styles:
You can download these symbols from the QGIS style sharing platform: https://plugins.qgis.org/styles/101/ to use them in your projects. Have fun mapping!
Today’s post is a video recommendation. In the following video, Alexandre Neto demonstrates an exciting array of tips, tricks, and hacks to create an automated Atlas map series of the Azores islands.
Highlights include:
1. A legend that includes automatically updating statistics
2. A way to support different page sizes
3. A solution for small areas overshooting the map border
You’ll find the video on the QGIS Youtube channel:
This video was recorded as part of the QGIS Open Day June edition. QGIS Open Days are organized monthly on the last Friday of the month. Anyone can take part and present their work for and with QGIS. For more details, see https://github.com/qgis/QGIS/wiki#qgis-open-day
Friday 29 January, Richard and Raymond will do an Ask Me Anything (in Dutch) on release management, and how to advise clients on which releases they should deploy. And the meaning of FF, difference between LR and LTR, etc. It’s in Dutch, but if you want to join in anyway: https://meet.jit.si/qgisnl-release-amaQGIS
Sorry, this entry is only available in Dutch.
Even with all their downsides, CSV files are still a common data exchange format – particularly between disciplines with different tech stacks. Indeed, “How to Specify Data Types of CSV Columns for Use in QGIS” (originally written in 2011) is still one of the most popular posts on this blog. QGIS continues to be quite handy for visualizing CSV file contents. However, there are times when it’s just not enough, particularly when the number of rows in the CSV is in the range of multiple million. The following example uses a 12 million point CSV:
To give you an idea of the waiting times in QGIS, I’ve run the following script which loads and renders the CSV:
from datetime import datetime
def get_time():
t2 = datetime.now()
print(t2)
print(t2-t1)
print('Done :)')
canvas = iface.mapCanvas()
canvas.mapCanvasRefreshed.connect(get_time)
print('Starting ...')
t0 = datetime.now()
print(t0)
print('Loading CSV ...')
uri = "file:///E:/Geodata/AISDK/raw_ais/aisdk_20170701.csv?type=csv&xField=Longitude&yField=Latitude&crs=EPSG:4326&"
vlayer = QgsVectorLayer(uri, "layer name you like", "delimitedtext")
t1 = datetime.now()
print(t1)
print(t1 - t0)
print('Rendering ...')
QgsProject.instance().addMapLayer(vlayer)
The script output shows that creating the vector layer takes 02:39 minutes and rendering it takes over 05:10 minutes:
Starting ... 2020-12-06 12:35:56.266002 Loading CSV ... 2020-12-06 12:38:35.565332 0:02:39.299330 Rendering ... 2020-12-06 12:43:45.637504 0:05:10.072172 Done :)
Rendered CSV file in QGIS
Panning and zooming around are no fun either since rendering takes so long. Changing from a single symbol renderer to, for example, a heatmap renderer does not improve the rendering times. So we need a different solutions when we want to efficiently explore large point CSV files.
The Pandas data analysis library is well-know for being a convenient tool for handling CSVs. However, it’s less clear how to use it as a replacement for desktop GIS for exploring large CSVs with point coordinates. My favorite solution so far uses hvPlot + HoloViews + Datashader to provide interactive Bokeh plots in Jupyter notebooks.
hvPlot provides a high-level plotting API built on HoloViews that provides a general and consistent API for plotting data in (Geo)Pandas, xarray, NetworkX, dask, and others. (Image source: https://hvplot.holoviz.org)
But first things first! Loading the CSV as a Pandas Dataframe takes 10.7 seconds. Pandas’ default plotting function (based on Matplotlib), however, takes around 13 seconds and only produces a static scatter plot.
Loading and plotting the CSV with Pandas
hvPlot to the rescue!
We only need two more steps to get faster and interactive map plots (plus background maps!): First, we need to reproject the lat/lon values. (There’s a warning here, most likely since some of the input lat/lon values are invalid.) Then, we replace plot() with hvplot() and voilà:
Plotting the CSV with Datashader
As you can see from the above GIF, the whole process barely takes 2 seconds and the resulting map plot is interactive and very responsive.
12 million points are far from the limit. As long as the Pandas DataFrame fits into memory, we are good and when the datasets get bigger than that, there are Dask DataFrames. But that’s a story for another day.
If you are following QGIS topics on social media, you may have already seen this but if you don’t, I recommend having a look at Tim Sutton’s most recent adventures in building dashboards with QGIS:
The dashboard is built using labeling and geometry generator functionality. This means that they work in the QGIS application map window as well as in layouts. As hinted at in the screenshot above, the dashboard can show information about whole layers as well as interactive selections.
Here’s a full walk-through Tim published yesterday:
You can follow the further development via Tim’s tweets or the dedicated Github issue (where you can even find an example QGIS dashboard project in a GeoPacakge for download).
The 2016 post More icons & symbols for QGIS still regularly makes it to the top 10 list of posts by visitors. I wouldn’t attribute this popularity to the quality of this particular post, however. Instead, it’s a pretty clear sign that QGIS users are actively searching for more styling resources to add to their installations.
When it comes to styling resources, the person to follow right now is clearly Klas Karlsson who’s been keeping a steady stream of styling-related posts coming to Twitter:
Additionally, he’s the master-mind behind QGIS Hub, a – currently prototypical – platform for sharing styling resources and print layout templates:
If you are interested in sharing styling resources, head over there. Similarly, if you want to lend a hand developing QGIS Hub, get in touch!
We have recently been working for the French Space Agency ( CNES ) who needed to store and visualize satellite rasters in a cloud platform. They want to access the image raw data, with no transformation, in order to fullfill deep analysis like instrument calibration. Using classic cartographic server standard like WMS or TMS is not an option because those services transform datasets in already rendered tiles.
We chose to use a quite recent format managed by GDAL, the COG (Cloud Optimize Geotiff) and target OVH cloud platform for it provides OpenStack, a open source cloud computing platform.
A COG file is a GEOTiff file which inner structure is tiled, meaning that the whole picture is divided in fixed size tile (256 x 256 pixels for instance) so you can efficiently retrieve parts of the raster. In addition to the HTTP/1.1 standard feature range request, it is possible to get specific tiles of an image through the network without downloading the entire raster.
We used a service provided by OpenStack, called Object Storage to serve the COG imagery. Object storage allows to store and retrieve file as objects using HTTP GET/POST requests.
Web Coverage Service standard could have been an option. A WCS server can serve raw data according to a given geographic extent. It’s completely possible to deploy a container or a VPS (Virtual Private Server) running a WCS Server in a cloud plateform. The main advantages of the COG solution over WCS Server is that you don’t have to deal with the burden of deploying a server, like giving it ressources, configuring load balancing, handle updates, etc…
The beauty of COG solution is its simplicity. It is only HTTP requests, and everything else (rendering for instance) is done on the client side.
Here are the different steps you’d have to go through if you’re willing to navigate in a big raster image directly from the cloud.
First, let’s generate a COG file
Install your openstack-client, it can be achieved easily with Python pip install command line
Next, configure your openstack client in order to generate an athentification token. To do so you need to download your project specific openrc file to setup your environment)
You are now good to push you COG file to the cloud instance
Before starting QGIS, 2 environment variables need to be set. (replace xxxx_my_token_xxxx with the token you’d just come to generate)
It can also be done directly from the QGIS Python console by setting those variable using the os.environ.
Finally, add a cloud raster data source in in QGIS
You can now navigating into your image directly reading it from the cloud
© CNES 2018, Distribution Airbus DS
While panning in the map, QGIS will download only few tiles from the image in order to cover the view extent. The display latency that you could see in the video depends essentially on:
Note that the white flickering that you could see when you move in the map and the raster is refreshed should be removed in next version of QGIS according to this QEP.
Thanks so much to the GDAL and QGIS contributors for adding such a nice feature ! It brings lots of possibilities for organizations that have to deal with great number of big raster and just want to explore part of it.
We are already thinking about further improvments (ease authentification, better integration with processing…), so if you’re willing to fund them or just want to know more about QGIS, feel free to contact us at [email protected]. And please have a look at our support offering for QGIS.
If you’re a follower of North Road’s social media accounts, it probably comes as no surprise to hear that we love sharing regular tips and useful suggestions about working with spatial data on these channels. Recently, we realised we were close to a new milestone of 4k followers of our Twitter stream. This milestone gave us the perfect excuse to give something back to all these followers! So we ran a little promo, where we promised that if we hit 4k followers we’d implement a new feature in QGIS, as determined by a popular vote:
*GIVE AWAY*
We’re nearly at 4K followers! ?
We love you guys, so as a thank-you to our wonderful community for sharing and liking our tweets we want to give YOU something! Vote for your favourite #QGIS feature below and we’ll add the most popular when we hit 4,000 followers!— North Road (@northroadgeo) August 8, 2020
Given we’ve now reached the target milestone, it was our turn to deliver! So we’re proud to introduce a new feature coming in QGIS 3.16: the ability to control exactly where labels are positioned along line feature! Let’s take a look!…
In previous QGIS releases, whenever labels are enabled for a line layer these labels will automatically gravitate towards the center of the line features:
In certain situations, it’s desirable for these labels to be positioned in specific positions along these line features (e.g. at the start of lines, or at the end of lines). With this new feature gift to the community, QGIS users have to option to precisely control the label placement for these lines:
There’s new options for placing the labels at the start of lines, end of lines, or at a custom offset along the line.
This option can be super-handy when combined with Rule Based Labelling, as it allows you to have different labels at the start versus end of lines:
So there we go! Another great QGIS cartography enhancement, heading your way in QGIS 3.16, and just a little thank you back to the community for all your support of North Road.
And if you’re not following us on social media yet, it’s a great to start… because who knows when our next little giveaway will be?!
QWAT est une application open source de gestion des réseaux d’eau potable émanant des collectivités de Pully, le SIGE à Vevey, Morges et Lausanne.
QGEP est son homologue dédiée à la gestion des eaux usées et pluviales, initiée par le groupe utilisateur QGIS Suisse.
L’échange de données entre institutions est une pierre angulaire des politiques de l’eau. Ces échanges se basent sur des formats d’échanges standardisés. Ainsi les Cantons de Fribourg (format aquaFRI) ou de Vaud (format SIRE) conditionnent certaines subventions publiques à la transmission des données selon des formats pré-définis et permettent à ces échelons administratifs d’avoir une vision globale des réseaux humides.
Dans le cadre d’une expérimentation des outils QWAT (eau potable) et QGEP (eaux usées), Charentes Eaux a souhaité mettre en œuvre des extensions dédiées au standard d’échange de données sur les réseaux d’eau Français, le Géostandard Réseaux d’adduction d’eau potable et d’assainissement (RAEPA) défini par la Commission de validation des données pour l’information spatialisée (COVADIS).
Oslandia a été mandaté pour mettre en œuvre des instances de QWAT et QGEP, réaliser les extensions RAEPA pour chacun de ces outils, et aider Charente Eaux à charger les données des collectivités membres de ce syndicat mixte.
https://charente-eaux.fr/le-syndicat/qui-sommes-nous/
Le travail a été publié pour QWAT sous forme d’une extension standardisée dans le dépôt l’organisation QWAT https://github.com/qwat/extension_fr_raepa/
Pour QGEP, il n’existe pas encore de fonctionnalité pour gérer d’extension, le dépôt https://gitlab.com/Oslandia/qgep_extension_raepa/ contient donc les définitions de données et de vues à rajouter manuellement au modèle de données.
La compatibilité des modèles de données a été évaluée et le choix a été fait de ne faire que des vues dédiées à l’export de données. Il est techniquement possible de faire des vues éditables pour permettre le chargement de données via ces vues depuis des fichiers suivant le gabarit de données RAEPA. Le niveau de simplification et d’agrégation des listes de valeurs rend ce travail peu générique dans l’état actuel du géostandard (v1.1), il est donc plus pertinent à ce stade de réaliser des scripts de chargement sans passer par ce pivot dans le cas de Charente-Eaux
With the recent advancements in LiDAR survey technology and photogrammetry there has been a huge demand in capturing and storing point cloud data. Point cloud data are vector in nature, but are usually orders of magnitude larger than a standard vector layer. Typical vector datasets range from thousands to millions of features, while point clouds range from millions to billions or even trillions of points. Due to this sheer number of points a completely different approach to visualise, analyse and store point clouds is needed in a GIS platform.
Integrating a point cloud viewer in a desktop GIS application adds a lot of value for users compared to a specialised and dedicated point cloud viewer:
Despite these benefits, current versions of the QGIS desktop application do not support visualisation of point cloud data. With our partners at Lutra Consulting and Hobu, we have decided to bridge this missing gap and add point cloud support in QGIS, and are launching a new crowdfunding campaign to fund this work!
Head on over to the official crowd funding page here for the full details of the campaign, and for details on how you can contribute and make this work a reality.
