Page 1 of 116 (2315 posts)

  • talks about »

Tags

Last update:
Thu Jun 4 16:00:17 2020

A Django site.

QGIS Planet

Securely accessing enterprise ArcGIS Portal sites through QGIS

We were recently contacted for advice regarding our recommendations for securely accessing content on an enterprise ArcGIS Portal deployment from within QGIS. Fortunately, this setup is fully supported and works seamlessly in QGIS, thanks to the native integration of “OAuth2” authentication in QGIS!

This post details step-by-step instructions in setting up both ArcGIS Portal and QGIS to enable this integration. First, we’ll create a new desktop application on the Portal site in order to obtain the application-specific access keys for OAuth2 authentication. We’ll then create an authentication configuration in QGIS and associate this with a connection to the Portal site. Let’s dive in by doing the Portal configuration first…

Creating an application

Logon to the Portal, and from the “Content” tab, click the “Add Item” option. Select “An application” from the drop down list of options:

Set the type of the application as “Desktop

You can fill out the rest of this dialog as you see fit. Suggested values are:

  • Purpose: Ready to Use
  • Platform: Qt
  • URL: http://qgis.org
  • Tags: QGIS, Desktop, etc

Now – here comes a trick. Portal will force you to attach a file for the application. It doesn’t matter what you attach here, so long as it’s a zip file. While you could attach a zipped copy of the QGIS installer, that’s rather wasteful of server space! We’d generally just opt for a zip file containing a text file with a download link in it.

Click Add Item when you’re all done filling out the form, and the new application should be created on the Portal.

Registering the Application

The next step is to register the application on Portal, so that you can obtain the keys required for the OAuth2 logon using it. From the newly created item’s page, click on the “Settings” tab:

Scroll right to the bottom of this page, and you should see a “Register” button. Press this. Set the “App type” to “Native“.

Add two redirect URIs to the list (don’t forget to click “Add” after entering each!):

  1. The Portal’s public address, e.g. https://mydomain.com/portal
  2. http://127.0.0.1:7070

Finally, press the “Register” button in the dialog. If all goes well then the App Registration section in the item settings should now be populated with details. From here, copy the “App ID” and “Secret” strings, we’ll need these later:

Determine Request URLs

One last configuration setting we’ll need to determine before we fire up QGIS is the Portal’s OAuth Request and Token URLs. These are usually found by appending /sharing/rest/oauth2/authorize and /sharing/rest/oauth2/token to the end of your Portal’s URL.

For instance, if your public Portal URL is http://mydomain.com/portal, then the URLs will be:

Request URL: http://mydomain.com/portal/sharing/rest/oauth2/authorize
Token URL: http://mydomain.com/portal/sharing/rest/oauth2/token

You should be able to open both URLs directly in a browser. The Request URL will likely give a “redirect URL not specified” error, and the Token URL will give a “client_id not specified” error. That’s ok — it’s enough to verify that the URLs are correct.

We’re all done on the Portal side now, so time to fire up QGIS!

Creating an QGIS OAuth2 Authentication Configuration

From your QGIS application, select Options from the Settings menu. Select the Authentication tab. We need to create a new authentication configuration, so press the green + button on the right hand side of the dialog. You’ll get a new dialog prompting you for Authentication details.

There’s a few tricks to this setup. Firstly, it’s important to ensure that you use the exact same settings on all your client machines. This includes the authentication ID field, which defaults to an auto-generated random string. (While it’s possible to automatically deploy the configuration as part of a startup or QGIS setup script, we won’t be covering that here!).

So, from the top of the dialog, we’ll fill in the “Name” field with a descriptive name of the Portal site. You then need to “unlock” the “Id” field by clicking the little padlock icon, and then you’ll be able to enter a standard ID to identify the Portal. The Id field is very strict, and will only accept a 7 letter string!

Drop down the Authentication Type combo box, and select “OAuth2 Authentication” from the list of options. There’s lots of settings we need to fill in here, but here’s what you’ll need:

  • Grant flow: set to “Authorization Code”
  • Request URL: enter the Request URL we determined in the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/authorize
  • Token URL: enter the Token URL from the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/token
  • Refresh Token URL: leave empty
  • Redirect URL: leave as the default http://127.0.0.1:7070 value
  • Client ID: enter the App ID from the Portal item’s App Registration information (see earlier steps)
  • Client Secret: enter the App Secret from the Portal item’s App Registration information (see earlier steps)
  • Scope: leave empty
  • API Key: leave empty

That’s it — leave all the rest of the settings at their default values, and click Save.

You can close down the Options dialog now.

Adding the Portal Connection Details

Lastly, we’ll need to setup the Portal connection as an “ArcGISFeatureServer” and “ArcGISMapServer” connection in QGIS. This is done through the QGIS “Data Source Manager” dialog, accessed through the Layer menu. Click the “ArcGIS Feature Server” tab to start with, and then press “New” in the Server Connections group at the top of this dialog.

Enter a descriptive name for the connection, and then enter the URL for the ArcGIS server REST endpoint associated with your Portal:

Lastly, select the new OAuth2 configuration you just created under the “Authentication” group:

Click OK, and you’re done! When you try to connect to the newly added connection, you’ll automatically be taken to the Portal’s logon screen in order to authenticate with the server. After entering your details, you’ll then be connected securely to the server and will have access to all items which are shared with your user account on the Portal!

You can then repeat this step for and create a similar connection under the “ArcGIS Map Server” tab.

We’ve regularly use this setup for our enterprise clients, and have found it to work flawlessly in recent QGIS versions! If you’ve found this useful and are interested in other “best-practice” recommendations for mixed Open-Source and ESRI workplaces, don’t hesitate to contact us to discuss your requirements… at North Road we specialise in ensuring flawless integration between ESRI based systems and the Open Source geospatial software stack.

Movement data in GIS #30: synchronized trajectory animations with QGIS temporal controller

QGIS Temporal Controller is a powerful successor of TimeManager. Temporal Controller is a new core feature of the current development version and will be shipped with the 3.14 release. This post demonstrates two key advantages of this new temporal support:

  1. Expression support for defining start and end timestamps
  2. Integration into the PyQGIS API

These features come in very handy in many use cases. For example, they make it much easier to create animations from folders full of GPS tracks since the files can now be loaded and configured automatically:

Script & Temporal Controller in action (click for full resolution)

All tracks start at the same location but at different times. (Kudos for Andrew Fletcher for recordings these tracks and sharing them with me!) To create an animation that shows all tracks start simultaneously, we need to synchronize them. This synchronization can be achieved on-the-fly by subtracting the start time from all track timestamps using an expression:

directory = "E:/Google Drive/QGIS_Course/05_TimeManager/Example_Dayrides/"

def load_and_configure(path):
    path = os.path.join(directory, filename)
    uri = 'file:///' + path + "?type=csv&escape=&useHeader=No&detectTypes=yes"
    uri = uri + "&crs=EPSG:4326&xField=field_3&yField=field_2"
    vlayer = QgsVectorLayer(uri, filename, "delimitedtext")
    QgsProject.instance().addMapLayer(vlayer)

    mode = QgsVectorLayerTemporalProperties.ModeFeatureDateTimeStartAndEndFromExpressions
    expression = """to_datetime(field_1) -
    make_interval(seconds:=minimum(epoch(to_datetime("field_1")))/1000)
    """

    tprops = vlayer.temporalProperties()
    tprops.setStartExpression(expression)
    tprops.setEndExpression(expression) # optional
    tprops.setMode(mode)
    tprops.setIsActive(True)

for filename in os.listdir(directory):
    if filename.endswith(".csv"):
        load_and_configure(filename)

The above script loads all CSV files from the given directory (field_1 is the timestamp, field_2 is y, and field_3 is x), enables sets the start and end expression as well as the corresponding temporal control mode and finally activates temporal rendering. The resulting config can be verified in the layer properties dialog:

To adapt this script to other datasets, it’s sufficient to change the file directory and revisit the layer uri definition as well as the field names referenced in the expression.


This post is part of a series. Read more about movement data in GIS.

Spatial on air: talking Python on the MapScaping Podcast

Podcasts have become huge. I’m an avid listener of podcasts myself. I particularly enjoy formats that take the time to talk about unconventional topics in detail.

My first podcast experience was on the QGIS podcast hosted by Tim Sutton in 2014. Unfortunately, it seems like the podcast episodes are not online anymore.

Recently, I had the pleasure to join the MapScaping Podcast by Daniel O’Donohue to talk about Python for Geospatial

Other guests Daniel has already interviewed include:

Another geospatial podcast I really enjoy is The Mappyist Hour by Silas and Todd. Unfortunately, it’s a bit silent there now but it’s definitely worth to listen into their episode archive. One of my favorites is Episode 9 where Linda Stevens (Hecht) discusses her career at ESRI, the future of GIS, and the role of Open Source Spatial in that future:

If you listen to and want to recommend other spatial podcasts, please share them in the comments!

QGIS Server and OGC API Features

Based on text and information from Paul Blottiere and Alessandro Pasotti (both QCooperative)

QGIS Server implements a number of OGC services, such as WMS, WFS, WCS or WMTS and extends these services where useful. Thanks to the efforts of a number of QGIS Server developers and companies, QGIS 3.10 (and 3.4 before) had been certified by the OGC for the WMS 1.3.0 service, and is also a WMS reference implementation.

alt wording

Last year in 2019, a new protocol has been developed and named OGC API Features (commonly known as WFS3). With the purpose of having an up-to-date QGIS Server, both OSGeo and QGIS.ORG have dedicated funds to work on the implementation of this brand-new service: but we wanted to do it right, so the ambition was also to reach the OGC certification!

This new protocol with REST interfaces gets rid of the XML specification to use the OpenAPI standard as well as the JSON open format instead. In other words, it’s not just another protocol to support, but a whole package of changes and fresh mechanisms to work on. It was quite a challenge!

QGIS core developers of QCooperative were remotely participating in OGC sprints to closely monitor the development of the new OGC API Features protocol. Hence, we started its implementation and a fully operational version landed in QGIS Server 3.10.

Implementation and features

As a reminder, the WFS protocol allows to query, retrieve and manipulate vector features, unlike the WMS format which provides raster outputs. OGC API Features is the natural continuity and consistently provides basic mechanisms to retrieve features and corresponding information in a specific area (the famous GetFeatureInfo request in WFS 1.X).

In addition, QGIS Server also provides transactions for the OGC API Features protocol. This means basically that we can update, insert or delete features in the underlying data. And of course, everything can be easily reached and configured through QGIS Desktop.

Yet another interesting thing to note is also the full support of the date and time filtering. Nifty!

And last, but not least, QGIS Server 3.10 provides a default HTML template with an embedded map to explore the data served by the server. There’s literally nothing to configure, it’s just there as soon as you work with the OGC API Features protocol :).

alt wording

OGC Certification

Once the implementation was completed, we started to address the OGC certification goal. To avoid unwanted regressions along the way, we first added nightly tests by updating the dedicated QGIS repository for OGC tests. From that moment, HTML reports are available day-to-day to monitor development over time.

Then, some bugfixes and backports later, we’re finally there: OGC tests are green on the development version, 3.12 and 3.10 releases. Yippee!

alt wording

Conclusion

Now that everything is in order, the last step is to start the formal OGC certification process. From now on, the dedicated QGIS OGC Team takes care of further operations.

Greetings from your new QGIS project Chair

Dear community,

First of all, I would like to thank Paolo for his work during the last two years as Chair and for the years before that, and years to come in his role as PSC member. I’m looking forward to keeping up the good work with him.

Secondly, I would like to thank all the community voting members for all the great inputs during the discussion phase of the AGM and for the fantastic participation in the voting process where we had more than an 80% turnout.

It is a pleasure to see that besides approving the more formal points (annual report, financial report, budget and auditors), the AGM approved all matters arising:

  • We now have two new honourable voting members: Harrissou Sant-anna and Nyall Dawson. Honourable members are specially designated voting members, whose position does not need to be affiliated with a country user group. Congratulations, and thank you, you are both such an inspiring example to our community!
  • Many QGIS users and contributors are geoscientists or geoinformatics specialists. As such, we need to act responsibly and serve as role models. Thanks to the approval of our new environmental policy, QGIS.ORG and the QGIS community committed ourselves to act responsibly regarding our actions and activities where it has any relevant influence on the environment. This will mainly affect our server infrastructure and our physical contributor meetings. The complete policy can be found in appendix 1 of our AGM minutes or on our website.

Beyond thanking Paolo as outgoing chair, I’d especially like to thank Tim, Andreas, Anita and Jürgen for the great work they are doing in the PSC and in general the incredible drive they have in helping to make QGIS thrive. I’m sure that with the help of our new Vice-Chair Alessandro Pasotti we’ll be able to take QGIS to even greater heights. Welcome, Alessandro!

Last, but definitely not least, I’d like to thank every single member of this amazing community for all your help documenting, coding, translating, testing, designing, teaching, supporting and in general spreading the QGIS love.

I never thought, when I first started using QGIS 0.6 in 2005 that 15 years later I would be given the honour of becoming the official face of such an amazing heart.

Have a great week, rock on QGIS!

Marco Bernasocchi

Incoming Chair of the QGIS.ORG Board

 


For Reference, here an extract of my published vision for QGIS.org:

I want to help QGIS and it’s community thrive under the value proposition of:

Making the most amazing opensource GIS that provides users with value and that meets their needs by providing great functionality and usability, being cost-effective whilst being actively supported by a vibrant and knowledgeable community.

Sharing our work under an open-source license is part of the approach by which we achieve that value proposition as it allows broad collaboration with our developers and users community.

I see FOSS as a very socially responsible way to develop software, but even more, I see the immense technological advantage that writing open-source code brings. This is why I want our focus to be on allowing both pragmatic and ideological views to respectfully coexist and enrich each other.

One of my main motivations to be part of the PSC and to make myself available as project Chair is to help QGIS keep this incredible growth rate by being even more attractive to new community members, sponsors and large/corporate users. To achieve this, the key is maintaining the right balance between sustainable processes (that guarantee the great quality QGIS has been known for) and an interesting and motivating grassroots project to ensure that QGIS remains an attractive project for volunteers to contribute to and help QGIS and its community to grow to become even more the reference [Open Source] GIS project.

TimeManager is dead, long live the Temporal Controller!

TimeManager turns 10 this year. The code base has made the transition from QGIS 1.x to 2.x and now 3.x and it would be wrong to say that it doesn’t show ;-)

Now, it looks like the days of TimeManager are numbered. Four days ago, Nyall Dawson has added native temporal support for vector layers to QGIS. This is part of a larger effort of adding time support for rasters, meshes, and now also vectors.

The new Temporal Controller panel looks similar to TimeManager. Layers are configured through the new Temporal tab in Layer Properties. The temporal dimension can be used in expressions to create fancy time-dependent styles:

temporal1

TimeManager Geolife demo converted to Temporal Controller (click for full resolution)

Obviously, this feature is brand new and will require polishing. Known issues listed by Nyall include limitations of supported time fields (only fields with datetime type are supported right now, strings cannot be used) and worse performance than TimeManager since features are filtered in QGIS rather than in the backend.

If you want to give the new Temporal Controller a try, you need to install the current development version, e.g. qgis-dev in OSGeo4W.


Update from May 16:

Many of the limitations above have already been addressed.

Last night, Nyall has recorded a one hour tutorial on this new feature, enjoy:

QGIS Annual General Meeting – 2020

Dear QGIS Community

We recently held our 2020 QGIS Annual General Meeting. The minutes of this meeting are available for all to view.

I would like to welcome our new QGIS Board Chair: Marco Bernasocchi and our new QGIS Board Vice-Chair and QGIS PSC Member, Alessandro Pasotti. In case you are not familiar with Marco and Alessandro, you can find short introductions to them below. I will continue to serve on the PSC and am pleased also to say that the project governance is in good hands with Jürgen Fischer, Andreas Neumann and Anita Graser kindly making themselves available to serve on the PSC for another two years. It is also great to know that our project founder, Gary Sherman, as well as long-term PSC member Tim Sutton continue to serve on the PSC as honorary PSC members. They both set the standard for our great project culture and it is great to have his continued presence.

QGIS has been growing from strength to strength, backed by a really amazing community of kind and collaborative users, developers, contributors and funders. I am looking forward to seeing how it continues to grow and flourish and I am excited and confident it will do so with Marco acting as the project chair and representative. Rock on QGIS!

Marco Bernasocchi (http://berna.io @mbernasocchi)

I am an open source advocate, consultant, teacher and developer. My background is in geography with a specialization in geographic information science. I live in Switzerland in a small Romansh speaking mountain village where I love scrambling around the mountains to enjoy the feeling of freedom it gives me. I’m a very communicative person, I fluently speak Italian, German, French English and Spanish and love travelling.

I work as director of OPENGIS.ch which I founded in 2011. Since 2015 I share the company ownership with Matthias Kuhn. At OPENGIS.ch LLC we (6 superstar devs and myself) develop, train and consult our client on any aspect related to QGIS.

My first QGIS (to be correct for that time QuantumGIS) ever was “Simon (0.6)” during my BSc when the University of Zurich was teaching us proprietary products and I started looking around for Open Source alternatives. In 2008, when starting my MSc, I made the definitive switch to ubuntu and I started working more and more with QGIS Metis (0.11) and ended developing some plugins and part of Globe as my Masters thesis. Since three years the University of Zurich invites me to hold two seminars on Entrepreneurship and Open Source. In November 2011 I attended my first Hackfest in Zürich where I started porting all QGIS dependencies and developing QGIS for Android under a Google Summer of Code. A couple of years and a lot of work later QField was born. Since then I’ve always tried to attend at least to one Hackfest per year to be able to feel first hand the strong bonds within our very welcoming community. In 2013 i was lucky enough to have a release named after a suggestion I saved you all from having QGIS 2.0 – Hönggerberg and giving you instead QGIS 2.0 – Dufour. In 2018 I’ve been honored to be nominated Co-chair of the QGIS PSC, since then I’ve been taking care of GitHub, the user groups, running votes, elections, doing some small work on the website, giving more talks on opensource advocacy and foremost helping in the day to day work needed to help our amazing project keep on growing.

Beside my long story with QGIS as user and passionate advocate I have a long story as QGIS service provider where we are fully committed to its stability, feature richness and sustainable development. For that in 2019 we started our own QGIS sustainability initiative financed through our support contracts.

Alessandro Pasotti (@elpaso https://www.itopen.it, https://www.qcooperative.net)

I am an open source software developer and I live in Italy. By education I’m an agronomist with some topography and pedology background, but I turned to the dark side early in my career and I started programming any kind of device that has a chip inside as soon as their price dropped low enough. I started using Linux in 1994 and after some real work as an R&D data analyst for a big pharmaceutical company I started my own small business that was making map-based web applications for the touristic market (there was no Google Map and such at that time) and it is for this reason that I discovered GRASS, Mapserver, PostGIS and finally QGIS when I needed a GIS viewer.

Over the years I’ve made minor contributions to several open source projects and I created a bunch of QGIS Python plugins, but it is from the QGIS Lisbon Hack-Fest in 2011 that I really got involved within the community and my first big contribution was a new website for the fast growing set of QGIS Python plugins (the one that it is already in production today at https://plugins.qgis.org ).

8 years ago I re-started to write some C++ code and I’m now a QGIS core developer and a proud member of this amazing community.

Regards

Paolo Cavallini (outgoing Chair)

QGIS video tutorials: election maps, hydrology, and more

Mapping spatial decision patterns, such as election results, is always a hot topic. That’s why we decided to include a recipe for election maps in our QGIS Map Design books. What’s new is that this recipe is now available as a free video tutorial recorded by Oliver Burdekin:

This video is just one of many recently published video tutorials that have been created by QGIS community members.

For example, Hans van der Kwast and Kurt Menke have recorded a 7-part series on QGIS for Hydrological Applications:

and Klas Karlsson’s Youtube channel is also always worth a follow:

For the Pythonically inclined among you, there is also a new version of Python in QGIS on the Automating GIS-processes channel:

 

(Fr) Entretien avec Vincent Picavet

Sorry, this entry is only available in French.

SLYR ESRI to QGIS compatibility suite – April 2020 update

Since the last update, our “SLYR” ESRI to QGIS compatibility suite has gained a ton of new functionality, including full support for conversion of ArcMap MXD documents (with page layouts!). In this update, we’ll explore some of the new functionality available in the tool — but instead of focusing solely on SLYR, this time we’ll also explore the enhancements we’ve been making in QGIS itself that have helped improve the quality of ArcMap document conversion.

While many of these enhancements are already available to all users of QGIS 3.12, others are exciting additions to the upcoming QGIS 3.14 release. Let’s dive in!

Improved legend customisation

One shortcoming we realised early on during our work on SLYR was that QGIS map legends just didn’t offer a comparable level of customisation as ArcMap legends. We could convert the basic layout of a legend, but we just couldn’t get the legend appearance in QGIS sufficiently close to its original appearance in the MXD file.

To address this we’ve been extending QGIS’ inbuilt legend support by adding finer control over the legend layout and spacing.

As a result, one exciting addition we’ve recently made for QGIS 3.14 is adding the ability to customise legend patch shapes and sizes on an item-by-item basis! Previously, legends in QGIS were rather boring, with all polygon layers showing as a plain rectangle and line layers as a horizontal line.

Now, users have full control over setting custom shapes for their legend patches! This makes for much more user-friendly legends, as you can now show representative shapes in your legends — e.g. a river symbol can be shown as a wiggly line, instead of an unrealistic straight horizontal line.

You can also override the size of a legend patch on an item-by-item basis too, which allows for further control over the final legend appearance. Checkout the screencast below showing both these features in action (naturally, using a legend from a converted MXD document… ArcMap users will likely recognise the fonts and patch shapes used here!)

We really wanted custom legend patch shapes to be a full first-class citizen in QGIS, so we also added support for managing them in user’s style databases. This makes it easy to setup your own libraries of custom legend shapes and share them with others. As a nice bonus, the SLYR tool even offers support for converting area and line patch shapes while converting ESRI .style databases:

Marker north arrows

Another issue we ran into while converting ArcMap page layouts was converting north arrows. QGIS used a very different approach to north arrows compared with ArcMap — in QGIS, north arrows were always based on existing SVG files, while ArcMap uses a rotated marker symbol for north arrows. Both approaches have their advantages and disadvantages, but we struggled to get good results when trying to convert ArcMap’s marker based approach to QGIS’ SVG based approach.

In the end, we weren’t happy with the result, so we took the step of implementing full support for marker symbol north arrows in QGIS 3.14. Now QGIS users have a choice of both north arrow styles — you can still create north arrows direct from SVG files, but you’ve also now got the flexibility to create them from standard marker symbols instead!

Adding support for marker based north arrows in QGIS allows us to get an perfect match when converting ESRI Page Layouts with north arrows:

Hollow and stepped line scale bars

While adding support for scale bar conversion to SLYR, we identified that some scale bar styles which are widely used in ArcMap just weren’t possible to reproduce in QGIS. Accordingly, from QGIS 3.14 on, we added native support for “Stepped line” and “Hollow” scale bar styles:

Embedded images in QGIS print layouts

ArcMap offers users the ability to directly embed images inside symbol definitions or page layouts. Whilst QGIS has offered embedded image support in symbols for a number of releases, this previously wasn’t possible to do in print layouts.

This was an issue for us while converting ArcMap page layouts, because we didn’t have any way to represent embedded images in QGIS layouts. Accordingly, for QGIS 3.14 we’ve added native support for directly embedding images (either raster images or SVG pictures) inside a page layout:

One handy consequence of this improvement is that it’s now possible to create completely self contained print layout templates for QGIS — you no longer have to separately distribute any required images (such as company logos) along with your QPT templates!

Naturally, our SLYR plugin now automatically converts any embedded images it finds in an ArcMap page layout and correctly creates a converted, embedded version of the image in the QGIS print layout.

Scalebar numeric formats

Another missing customisation in QGIS’ scale bar functionality was allowing users control over the scale bar’s number format. Previously, QGIS offered no customisation for these numbers, so you got only what QGIS decided you wanted. We improved this in QGIS 3.12 by offering users the ability to control exactly how they want their scale bar numbers to appear. There’s options for manually controlling the thousand and decimal separators, rounding, and much more:

This enhancement allowed us to get an exact match when converting ArcMap scale bars — the converted results should appear identical to their original ArcMap appearance!

Random marker fill

Until recently, one of the few remaining layer symbolisation gaps between QGIS and ArcMap was that QGIS had no symbology option for randomised marker placements for fill symbols. This was a big issue for us, because without it there just wasn’t any way that SLYR could convert layers styled with ArcMap’s “dot density renderer”  or using a marker fill’s “random offset” option.

So for QGIS 3.12, we add inbuilt support for a new “Random marker fill” symbol type:

This new symbol layer type allows for randomised (or stable seed-based) placement of markers inside polygon features. You’re given the option of either an absolute number of points to show in the feature, or a density-based count which retains its dot density regardless of the map’s scale!

Aside from being a useful symbology option in it’s own right, adding this functionality allowed us to accurately convert random markers or dot density renders from ArcMap to QGIS.

Other enhancements

The highlights above are only a small subset of the work we’ve done in QGIS to improve its interoperability with ArcMap via the SLYR plugin! Some of the other work we’ve done includes:

  • Many improvements to QGIS’ bad layer handling and automatic repair functionality. For instance, QGIS now emulates ArcMap’s helpful behaviour where ALL similar broken layer paths in a document are fixed automatically after fixing the path to one broken layer. Handling broken layer paths was a pain point for our customers, so we’ve sought to make this as painless as possible.
  • Support for plugins to handle pasting content into QGIS print layouts. We use this in SLYR to offer the ability to directly copy and paste content from ArcMap page layouts into a QGIS print layout.
  • Support for plugins to hook into the standard QGIS “Open Project” dialog, offering support for opening projects of their own custom types. We use this to allow users to directly open MXD, MXT, PMF and SXD files from the QGIS Open Project menu action.
  • We’ve worked closely with the upstream proj project, to ensure that coordinate reference systems from ESRI documents are correctly matched to known EPGS/ESRI CRS definitions in certain circumstances.

Other new features in SLYR

Aside from all the goodness we’ve explored above, the latest versions of SLYR offer TONS of new functionality for conversion of ArcMap documents, including:

  • Full support for joins and relations when converting MXD documents
  • Print layouts, including support for conversion of data driven pages to QGIS print atlases and support for multi-map page layouts using multiple data frames.
  • Support for reading MXD document metadata (and converting this to QGIS document metadata)
  • Support for dragging and dropping layers direct from ArcMap or ArcCatalog to a QGIS window, respecting all the layer styling.
  • Support for AVL style conversion
  • A new tool for dumping the full structure of MXD or LYR files to a json document. This is very handy for digging right into the full internals of the documents and for diagnosing corrupted documents.
  • Full support for conversion of vector and raster layers
  • Support for converting MXD, MXT and PMF documents
  • Support for converting ArcScene SXD documents to 2-dimensional QGIS maps

Read more are the SLYR home page, or contact us today to discuss purchasing SLYR and your licensing needs!

QGIS Grants #5: Call for Grant Proposals 2020

Dear QGIS Community,

Our previous rounds of grant proposals have always been a great success (2019, 2018, 2017, 2016). We are very pleased to announce that this year’s round of grants is now available. The call is open to anybody who wants to make a funded contribution to QGIS, subject to the call conditions outlined in the application form.

The deadline for this round is 24th May 2020.

For more details, please read the introduction provided in the application form.

We look forward to seeing all your great ideas for improving QGIS!

QGIS development on macOS

We introduced a new packaging and build environment for QGIS and its dependencies for macOS. This will bring a completely new experience for QGIS macOS developers and subsequently, the users.

Overview of the changes

What is new

In 2019, we upgraded the QGIS macOS packaging to address several issues at the time. This was made possible with the help of the QGIS project and donors who supported our work.

The upgraded system had a more transparent and automated packaging approach. But it came with its own limitation. We have been using Homebrew to fetch the QGIS dependencies and to compile only QGIS for the LTR, PR and nightly releases. The main problems with this approach are:

  • Homebrew can only support one version of a package. This will limit QGIS to be built against multiple versions of, for example GDAL or Proj libraries. In one hand, we want QGIS nightlies to use more bleeding edge versions of the dependencies (e.g. for plugin or core developers) . On the other hand, using a more stable and tested versions of the dependencies for PR or LTR are not recommended (e.g. users and organisations). OSGeo4W offers the same approach for different versions of QGIS in Windows. Unfortunately, Homebrew does not offer such an option.

  • Continuous integration has been a major part of QGIS infrastructure to ensure pull requests by developers do not break workflows in certain Operating Systems. Microsoft Windows and Linux have been well supported in the CI. But with the macOS packages, it was left to user to test and report back any bugs after Pull Requests were merged to QGIS source code.

With the recent work, we have addressed both of the above issues. QGIS and almost all of its dependencies are built from sources. It will be also possible to easily integrate the code with the current CI for macOS with GitHub Workflows. This will bring more stability and control over the way we build and package QGIS for our users, whilst helping developers to identify issues with their pull requests for macOS platform, before merging them to the QGIS source code.

QGIS development on macOS

If you want to check the latest QGIS master on macOS or tweak the code to see some new exciting features, now it should be easier with the following steps, to compile QGIS from source code an a clean machine:

1. Install XCode

  • Upgrade to latest MacOS version
  • Install latest XCode from the official AppStore
  • Open XCode and accept license
  • Install command line tools from XCode, AppStore or from command line by sudo xcode-select --install

2. Install Qt and QGIS-Deps

  • Download the latest install qgis-deps script and check the version of Qt required
  • Install the required Qt version referenced from the official Qt download area. Or alternatively download qt-version.tar.gz from deps area
  • Download latest qgis-deps package too.
  • Run install_qgis_deps.bash script to install Qt in /opt/Qt/5.14.1/clang_64 and QGIS-Deps to /opt/QGIS/qgis-deps-0.3.0 folder. You may need to use root privileges or create those folders under /opt/ and assign the right permission beforehand.

3. Install other tools

  • Install Homebrew
  • Install minimal set of packages required for macOS build tools listed in this list

You may use MacPorts, Conda or other packaging system, but by the end you need cmake, git, astyle and other useful tools to be on the system PATH

4. Download and compile QGIS

  • Open the terminal (Note that the qgis-deps package is not yet signed, so you may need to add Terminal to System Preferences -> Security & Privacy -> Privacy -> Developer Tools)
  • In your work folder (e.g. ~/Projects/), clone QGIS with git clone git@github.com:qgis/QGIS.git
  • Create a build folder mkdir -p ~/Projects/build-QGIS
  • Go to the created empty directory cd ~/Projects/build-QGIS
  • Run CMake to generate the build system (use your download qgis-deps and qt package)
    QGIS_DEPS_VERSION=0.3.0;\
    QT_VERSION=5.14.1;\
    PATH=/opt/QGIS/qgis-deps-${QGIS_DEPS_VERSION}/stage/bin:$PATH;\
    cmake -DQGIS_MAC_DEPS_DIR=/opt/QGIS/qgis-deps-${QGIS_DEPS_VERSION}/stage \
        -DCMAKE_PREFIX_PATH=/opt/Qt/${QT_VERSION}/clang_64 \
        ../QGIS
    

Note that all libraries are picked from qgis-deps and not from system /usr/lib or Homebrew’s /usr/local/ or system Frameworks /Library/Frameworks/. Especially check Proj, GDAL, sqlite3 and Python paths. You should see output similar to this:

-- QGIS version: 3.13.0 Master (31300)
-- Found OpenCL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework (found version "1.2")
-- Found OpenCL C++ headers: /Users/peter/Projects/mesh/QGIS/external/opencl-clhpp/include
-- Found GRASS 7: /opt/QGIS/qgis-deps-0.3.0/stage/grass78 (7.8.2, off_t size = 8)
-- Looking for openpty
-- Looking for openpty - found
-- Found Proj: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libproj.dylib version 6 (6.3.1)
-- Found GEOS: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libgeos_c.dylib (3.8.1)
-- Found GDAL: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libgdal.dylib (3.0.4)
-- Found Expat: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libexpat.dylib
-- Found Spatialindex: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libspatialindex.dylib
-- Found Qwt: /opt/QGIS/qgis-deps-0.3.0/stage/lib/qwt.framework (6.1.4)
-- Found LibZip: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libzip.dylib
-- Found libzip: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libzip.dylib
-- Found Sqlite3: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libsqlite3.dylib
-- Found Protobuf: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libprotobuf.dylib (found version "3.11.4")
-- Found Protobuf: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libprotobuf.dylib
-- Found ZLIB: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libz.dylib (found version "1.2.11")
-- Found zlib: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libz.dylib
-- Found PostgreSQL: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libpq.dylib
-- Found SpatiaLite: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libspatialite.dylib
-- Qt WebKit support enabled
-- Found Qt version: 5.14.1
-- Found QScintilla2: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libqscintilla2_qt5.dylib (2.11.4)
-- Found QtKeychain: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libqt5keychain.dylib
-- Found QCA: /opt/QGIS/qgis-deps-0.3.0/stage/lib/qca-qt5.framework (2.3.0)
-- Found QCA OpenSSL plugin
-- Found Libtasn1: /opt/QGIS/qgis-deps-0.3.0/stage/include
-- Pedantic compiler settings enabled
-- Found PythonInterp: /opt/QGIS/qgis-deps-0.3.0/stage/bin/python3 (found suitable version "3.7.7", minimum required is "3")
-- Found Python executable: /opt/QGIS/qgis-deps-0.3.0/stage/bin/python3
-- Found Python version: 3.7.7
-- Found Python library: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libpython3.7m.dylib
-- Found Python site-packages: /opt/QGIS/qgis-deps-0.3.0/stage/lib/python3.7/site-packages
-- Found PyQt5 version: 5.14.1
-- Found SIP version: 4.19.21
-- Found QScintilla2 PyQt module: 2.11.4
-- txt2tags not found - disabled
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Found exiv2: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libexiv2.dylib
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libhdf5.dylib;/opt/QGIS/qgis-deps-0.3.0/stage/lib/libz.dylib;/usr/lib/libdl.dylib;/usr/lib/libm.dylib (found version "1.10.0")
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
-- Found NetCDF: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libnetcdf.dylib
-- Found LibXml2: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libxml2.dylib (found version "2.9.10")
-- Looking for updwtmpx
-- Looking for updwtmpx - not found
-- Found GSL: -L/opt/QGIS/qgis-deps-0.3.0/stage/lib -lgsl -lgslcblas
-- Using PROJ 6 srs database.
-- Ctest Binary Directory set to: /Users/peter/Projects/mesh/build-QGIS/output/bin
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/peter/Projects/mesh/build-QGIS
  • Compile QGIS make -j4 and wait for [100%] mark :)
  • Now you can run QGIS with command ./output/bin/QGIS.app/Contents/MacOS/QGIS or open ./output/bin/QGIS.app

Notes:

  • If you use QT Creator, you can open already generated project to modify files from it.
  • The critical CMake variable is QGIS_MAC_DEPS_DIR and you must ensure it contains valid qgis-deps installation
  • You can add more variables to CMake command to tweak the build settings, e.g. enable 3D with ‘-DWITH_3D’, etc.
  • Consult QGIS’s [INSTALL]https://github.com/qgis/QGIS/blob/master/INSTALL) for latest developement tips

Development of the QGIS-Deps package

If you want to help with development of the qgis packages on macOS, please feel free to contact peter.petrik@lutraconsulting.co.uk. The development is taking place on public QGIS-Mac-Packager repository.

Next steps

Currently, we are carrying out the following tasks:

  • to move all-in-one QGIS bundle for LTR/PR/nighlies to use this new package of dependencies. Check out Twitter for for any news on the packages. We are aiming to have the new packaging in place for the QGIS 3.14 release, which will bring GDAL 3 and Proj 6 to macOS users

  • to sign the qgis-deps packages, so that developers do not need to tweak Privacy settings to use it for compiling QGIS and its dependencies.

PyQGIS Cookbook revision 2020

We are happy to announce that the PyQGIS Cookbook has received a complete overhaul and is now better than ever.

The PyQGIS Cookbook is a great source of information, not only for PyQGIS beginners or plugin developers, but also for C++ developers: it contains a lot of information about the internals of the QGIS API that you cannot really find anywhere else.

The main point addressed in this review were:

  1. All the code snippets have been reviewed and put under automated test in CI. Before this revision, there were 62 tests. Now there are over 300 tested snippets.
  2. A few snippets had to be updated because of changes in the QGIS API or because of the deprecation of methods (CRS handling in particular due to the proj6 switch).
  3. Textual descriptions have been edited to update the contents where the API has substantially changed.
  4. The material covering Python for QGIS Server has been reorganized and now includes new snippets and introductory texts about the new modules and OGC APIs architecture.

For the full summary, please refer to Alessandro Pasotti’s report on the PSC mailing list.

Thank you to Alessandro Pasotti for taking on this project and thank you to all our sponsor and donors who make this initiative possible!

Super-quick interactive data & parameter exploration

This post introduces Holoviz Panel, a library that makes it possible to create really quick dashboards in notebook environments as well as more sophisticated custom interactive web apps and dashboards.

The following example shows how to use Panel to explore a dataset (a trajectory collection in this case) and different parameter settings (relating to trajectory generalization). All the Panel code we need is a dict that defines the parameters that we want to explore. Then we can use Panel’s interact function to automatically generate a dashboard for our custom plotting function:

import panel as pn

kw = dict(traj_id=(1, len(traj_collection)), 
          tolerance=(10, 100, 10), 
          generalizer=['douglas-peucker', 'min-distance'])
pn.interact(plot_generalized, **kw)

Click to view the resulting dashboard in full resolution:

The plotting function uses the parameters to generate a Holoviews plot. First it fetches a specific trajectory from the trajectory collection. Then it generalizes the trajectory using the specified parameter settings. As you can see, we can easily combine maps and other plots to visualize different aspects of the data:

def plot_generalized(traj_id=1, tolerance=10, generalizer='douglas-peucker'):
  my_traj = traj_collection.get_trajectory(traj_id).to_crs(CRS(4088))
  if generalizer=='douglas-peucker':
    generalized = mpd.DouglasPeuckerGeneralizer(my_traj).generalize(tolerance)
  else:
    generalized = mpd.MinDistanceGeneralizer(my_traj).generalize(tolerance)
  generalized.add_speed(overwrite=True)
  return ( 
    generalized.hvplot(
      title='Trajectory {} (tolerance={})'.format(my_traj.id, tolerance), 
      c='speed', cmap='Viridis', colorbar=True, clim=(0,20), 
      line_width=10, width=500, height=500) + 
    generalized.df['speed'].hvplot.hist(
      title='Speed histogram', width=300, height=500) 
    )

Trajectory collections and generalization functions used in this example are part of the MovingPandas library. If you are interested in movement data analysis, you should check it out! You can find this example notebook in the MovingPandas tutorial section.

Successful crowdfunding for vector tile support in QGIS

We are delighted to announce that the crowdfunding to support Vector Tiles in QGIS has been successful!

Here is the list of donors who have supported us and the wider QGIS user by funding this feature.

List of donors

(in no particular order)

Also many thanks to those who spread the word and offered us moral support!

LTR usage survey

Back in 2018, we asked QGIS users how often they use QGIS and how often they upgrade. Today, we want to find out more about how different user groups and organisations use QGIS and particularly the LTR. You may be aware of ongoing discussions concerning potentially extending the LTR period and other potential steps to further improve user experience.

To better understand the needs of our QGIS community, we therefore invite you to our new LTR usage survey:

https://ee.kobotoolbox.org/single/::nKmWHGWv


Update as posted on the mailing list: 

Thanks to all respondents, we’ve collected over 1,600 responses to this questionnaire!

Some interesting tidbits:

  1. The LTR concept seems to be well accepted: 50% of respondents use 3.10 LTR, 38% use 3.12, and 16% use 3.4 LTR.
  2. Most respondents install patch versions but 27% do so less often than every 6 months.
  3. Most respondents install .0 releases but 17% wait until .3 or longer
  4. 6% of respondents work in organizations with more than >100 users. It will be interesting to look at those responses separately.
  5. 7% say that they still need 32bit installers

You can find the raw survey responses here:

https://drive.google.com/open?id=1lECEhusQcpDpb50MSMYP_5BDfEl4jXeV

QGIS Events Cancellation

Dear QGIS Community:

Due to the uncertainty caused by rapidly unfolding global events related to the COVID-19 virus, we have decided to cancel all in-person QGIS events until further notice. This includes the 25th QGIS Contributor meeting and User Conference that was scheduled to be held this year in Nødebo, Denmark. In the interim, we will pursue ways to meet virtually from time to time, and of course, continue working using our normal collaboration process via email and GitHub.

Thank you for your understanding,

The QGIS Team

Recording tracks using QGIS and Input app

In this post, we will use Input app to record tracks. The first step is to set up a project in QGIS.

Quick recap

In the previous tutorial, we set up a simple QGIS project and transferred it to our devices using the Mergin service.

In this tutorial we are going to set up a new survey project with more advanced features:

  • Forms with more options
  • Different background layers (aerial imagery and street map)
  • Using different geometry type of survey layer
  • Exploring other Input app functions

Software needed

To start with, you will need to install the following software applications:

In addition, you will need to register with the Mergin service. The Mergin service allows you to transfer data between your PC/laptop and mobile/table via the cloud. Note that after signing up to the Mergin service, you have to activate the account by clicking on the link sent to your email.

Configuring QGIS project

Similar to the previous tutorial, we are going to start with a blank QGIS project. All the data and the project will be stored locally on a folder called recording tracks.

First, start a QGIS project and add the OpenStreetMap layer under XYZ Tiles. You can add also add aerial imagery as an XYZ layer to your QGIS.

Project setup

Save your project as tracks in the recording tracks folder.

In addition, we need to create a Geopackage survey layer. Below are the attribute columns for your survey layer. Also note that need to select Line as geometry type.

To create a survey layer, in QGIS, from the main menu select Layer > Create Layer > New Geopackage Layer …. A new window will appear:

For Database click on and select the recording tracks folder then type survey.gpkg for the name of your database.

For Table name, type tracks.

For Geometry type, select Line.

You can select the option to Include Z dimension. The upcoming version of Input app supports capturing altitude from your device (or external GPS device tethered with your phone/tablet).

For Coordinate Reference System (CRS), select WGS 84 / Pseudo-Mercator EPSG:3857.

Add the following Fields. Note, different field types:

  • Type field type as text
  • Public field type as Boolean or bool
  • Photo field type as text
  • Date field type as Date & time

Geopackage survey layer

Click OK to create and add the layer to your QGIS session.

Styling layers and setting up forms

We are going to style the tracks layer based on Type field and use four categories:

  • Footpath
  • Byway
  • Cycle path
  • Bridle way

In the Layer Panels right-click on tracks layer and select Properties. A new window will appear. From the left panel, select Symbology. Change the styling from Single to Categorized on the top left section of the window. For the Value, select Type from the drop-down menu.

In the bottom-left section of the window, click on Classify. Then add 4 classes (by adding the green plus adjacent to Classify) for track types. In addition, we can add another class for Other types. Adjust the colours and line style accordingly:

Setting up style

To set up the form, select Attribute forms from the left panel of the layer properties window. Adjust the Widget type as below:

  • For fid, select Hidden
  • For Type, select Value map and add the 5 categories (Footpath, Byway, Cycle path, Bridle way and Other) for Value and Description section (see image below)
  • For Public, select Checkbox
  • For Photo, select Attachment and select the option for Relative path
  • For Date, select Date/Time

Setting up forms

Input app allows you to have a custom pop-up drawer window when you tap on feature. To customise that, you can change the Display settings under the layer properties window. For Display name select Type and in the HTML Map Tip section type:

# image file:///[%@project_folder%]/[% "Photo" %]

Custom pop-up

Setting up map themes

We can set up two different map themes: one with aerial imagery and one with street map. To do that, ensure you have only the following layers are set to visible in your layer panel:

  • tracks
  • Aerial images

Layer visibility

On top of the layers panel, select Manage Map Themes > Add Theme …

Creating a new theme

A new window will appear. Type Aerial photo for the name of your map theme and press OK.

Now, turn off your Aerial images layer in the layer panel and turn on the OpenStreetMap layer:

OSM theme in QGIS

You can now add a new map theme and call it Street map.

Save your project and transfer your data and project to your phone through the Mergin service as described in the previous tutorial.

If you missed a step, you can see the final version of the project here. You can also skip the above section by cloning the project in the above link.

Recording tracks using Input app

In Input app, download and open Tracks project.

Downloading a project

Map view in Input

You can easily change between different map themes in Input, by going to (… More) > Map Themes:

Selecting theme in Input

OSM map theme

To capture tracks, press Record button in the lower panel in Input. A green cross will appear, centred on your GPS location. You can capture a line in two different methods:

  • Adding points to form the line by moving the map and pressing Add Point
  • Streaming GPS location while moving to form a line. To do that, press and hold GPS icon on the bottom left of your screen (see image below)

Streaming GPS

You can change the frequency of adding points when streaming GPS location, under Settings in Input. Also note that the GPS streaming can be done for polygon survey layer.

Once you finished capturing the track, press Done and you will be presented with a form:

Forms

You can easily fill the form using the drop-down menu for Type and checkbox option for Public.

By tapping on tracks on the map, you will should be able to see a small preview window presenting Type and Photo:

Preview panel

After completion of the survey, you can synchronise your data back through the Mergin and see the results in QGIS desktop.

MovingPandas v0.3 released!

MovingPandas has come a long way since 2018 when I started to experiment with GeoPandas for trajectory data handling.

This week, MovingPandas passed peer review and was approved for pyOpenSci. This technical review process was extremely helpful in ensuring code, project, and documentation quality. I would strongly recommend it to everyone working on new data science libraries!

The lastest v0.3 release is now available from conda-forge.

All tutorials are available on MyBinder

New features include:

  • Support for GeoPandas 0.7
  • Trajectory collection aggregation functions to generate flow maps

 

Movement data in GIS #29: power your web apps with movement data using mobilitydb-sqlalchemy

This is a guest post by Bommakanti Krishna Chaitanya @chaitan94

Introduction

This post introduces mobilitydb-sqlalchemy, a tool I’m developing to make it easier for developers to use movement data in web applications. Many web developers use Object Relational Mappers such as SQLAlchemy to read/write Python objects from/to a database.

Mobilitydb-sqlalchemy integrates the moving objects database MobilityDB into SQLAlchemy and Flask. This is an important step towards dealing with trajectory data using appropriate spatiotemporal data structures rather than plain spatial points or polylines.

To make it even better, mobilitydb-sqlalchemy also supports MovingPandas. This makes it possible to write MovingPandas trajectory objects directly to MobilityDB.

For this post, I have made a demo application which you can find live at https://mobilitydb-sqlalchemy-demo.adonmo.com/. The code for this demo app is open source and available on GitHub. Feel free to explore both the demo app and code!

In the following sections, I will explain the most important parts of this demo app, to show how to use mobilitydb-sqlalchemy in your own webapp. If you want to reproduce this demo, you can clone the demo repository and do a “docker-compose up –build” as it automatically sets up this docker image for you along with running the backend and frontend. Just follow the instructions in README.md for more details.

Declaring your models

For the demo, we used a very simple table – with just two columns – an id and a tgeompoint column for the trip data. Using mobilitydb-sqlalchemy this is as simple as defining any regular table:

from flask_sqlalchemy import SQLAlchemy
from mobilitydb_sqlalchemy import TGeomPoint

db = SQLAlchemy()

class Trips(db.Model):
   __tablename__ = "trips"
   trip_id = db.Column(db.Integer, primary_key=True)
   trip = db.Column(TGeomPoint)

Note: The library also allows you to use the Trajectory class from MovingPandas as well. More about this is explained later in this tutorial.

Populating data

When adding data to the table, mobilitydb-sqlalchemy expects data in the tgeompoint column to be a time indexed pandas dataframe, with two columns – one for the spatial data  called “geometry” with Shapely Point objects and one for the temporal data “t” as regular python datetime objects.

from datetime import datetime
from shapely.geometry import Point

# Prepare and insert the data
# Typically it won’t be hardcoded like this, but it might be coming from 
# other data sources like a different database or maybe csv files
df = pd.DataFrame(
   [
       {"geometry": Point(0, 0), "t": datetime(2012, 1, 1, 8, 0, 0),},
       {"geometry": Point(2, 0), "t": datetime(2012, 1, 1, 8, 10, 0),},
       {"geometry": Point(2, -1.9), "t": datetime(2012, 1, 1, 8, 15, 0),},
   ]
).set_index("t")

trip = Trips(trip_id=1, trip=df)
db.session.add(trip)
db.session.commit()

Writing queries

In the demo, you see two modes. Both modes were designed specifically to explain how functions defined within MobilityDB can be leveraged by our webapp.

1. All trips mode – In this mode, we extract all trip data, along with distance travelled within each trip, and the average speed in that trip, both computed by MobilityDB itself using the ‘length’, ‘speed’ and ‘twAvg’ functions. This example also shows that MobilityDB functions can be chained to form more complicated queries.

mobilitydb-sqlalchemy-demo-1

trips = db.session.query(
   Trips.trip_id,
   Trips.trip,
   func.length(Trips.trip),
   func.twAvg(func.speed(Trips.trip))
).all()

2. Spatial query mode – In this mode, we extract only selective trip data, filtered by a user-selected region of interest. We then make a query to MobilityDB to extract only the trips which pass through the specified region. We use MobilityDB’s ‘intersects’ function to achieve this filtering at the database level itself.

mobilitydb-sqlalchemy-demo-2

trips = db.session.query(
   Trips.trip_id,
   Trips.trip,
   func.length(Trips.trip),
   func.twAvg(func.speed(Trips.trip))
).filter(
   func.intersects(Point(lat, lng).buffer(0.01).wkb, Trips.trip),
).all()

Using MovingPandas Trajectory objects

Mobilitydb-sqlalchemy also provides first-class support for MovingPandas Trajectory objects, which can be installed as an optional dependency of this library. Using this Trajectory class instead of plain DataFrames allows us to make use of much richer functionality over trajectory data like analysis speed, interpolation, splitting and simplification of trajectory points, calculating bounding boxes, etc. To make use of this feature, you have set the use_movingpandas flag to True while declaring your model, as shown in the below code snippet.

class TripsWithMovingPandas(db.Model):
   __tablename__ = "trips"
   trip_id = db.Column(db.Integer, primary_key=True)
   trip = db.Column(TGeomPoint(use_movingpandas=True))

Now when you query over this table, you automatically get the data parsed into Trajectory objects without having to do anything else. This also works during insertion of data – you can directly assign your movingpandas Trajectory objects to the trip column. In the below code snippet we show how inserting and querying works with movingpandas mode.

from datetime import datetime
from shapely.geometry import Point

# Prepare and insert the data
# Typically it won’t be hardcoded like this, but it might be coming from 
# other data sources like a different database or maybe csv files
df = pd.DataFrame(
   [
       {"geometry": Point(0, 0), "t": datetime(2012, 1, 1, 8, 0, 0),},
       {"geometry": Point(2, 0), "t": datetime(2012, 1, 1, 8, 10, 0),},
       {"geometry": Point(2, -1.9), "t": datetime(2012, 1, 1, 8, 15, 0),},
   ]
).set_index("t")

geo_df = GeoDataFrame(df)
traj = mpd.Trajectory(geo_df, 1)

trip = Trips(trip_id=1, trip=traj)
db.session.add(trip)
db.session.commit()

# Querying over this table would automatically map the resulting tgeompoint 
# column to movingpandas’ Trajectory class
result = db.session.query(TripsWithMovingPandas).filter(
   TripsWithMovingPandas.trip_id == 1
).first()

print(result.trip.__class__)
# <class 'movingpandas.trajectory.Trajectory'>

Bonus: trajectory data serialization

Along with mobilitydb-sqlalchemy, recently I have also released trajectory data serialization/compression libraries based on Google’s Encoded Polyline Format Algorithm, for python and javascript called trajectory and trajectory.js respectively. These libraries let you send trajectory data in a compressed format, resulting in smaller payloads if sending your data through human-readable serialization formats like JSON. In some of the internal APIs we use at Adonmo, we have seen this reduce our response sizes by more than half (>50%) sometimes upto 90%.

Want to learn more about mobilitydb-sqlalchemy? Check out the quick start & documentation.


This post is part of a series. Read more about movement data in GIS.

  • Page 1 of 116 ( 2315 posts )
  • >>

Back to Top

Sponsors