The goal of the new API is twofold:

1. provide a unified way to store and retrieve data provider connections in the QGIS settings
2. provide an abstract set of methods to perform most common operation on DB data sources (e.g. browse tables, drop/create a table/schema, run arbitrary SQL commands etc.)

 

The new API is documented in https://qgis.org/api/classQgsAbstractProviderConnection.html and it provides a few specializations for DB connections (https://qgis.org/api/classQgsAbstractDatabaseProviderConnection.html) and an initial PR implementation for web service-based connections (https://github.com/qgis/QGIS/pull/33045).

 

While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:

• postgres
• geopackage (OGR)
• spatialite

 

The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).

 

For developers interested in working with the new API, a set Python tests are available to show how to use the methods:  https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsproviderconnection_ogr_gpkg.py (see also the postgres and spatialite companion tests).

 

There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:

• port all remaining data providers to the new API
• refactor and eliminate the remaining DB-manager connectors to make use of the abstract API
• eliminate duplicate and untested code inside the Processing framework for working with Postgres databases and port the code to the new, stable, well-tested API
• refactor and eliminate the remaining QGIS browser data items to make use of the abstract API 

 

For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.

Background/motivation

• DB-Manager is an important part of the QGIS interface, which allows browsing/previews of different DB-based data sources, complex queries, management of layers, import-export etc., DB creation and deletion etc.
• After the QGIS 3.0 release, improvements within the core browser widgets implemented in C++ have resulted in a (constantly growing) degree of overlapping functionality between the browser and db manager.
• After QGIS 3 API improvements concerning layer import and export functionality, there are many duplicated implementations between browser and db manager – some functionality is better in browser, some functionality is better in db manager. Users are forced to choose between two competing semi-complete alternatives, instead of having one, complete, well integrated solution.
• There are no unit tests for DB-Manager and this leads to frequent regressions, which (aside from being frustrating for users) consume a substantial part of our development time and budget during the bugfixing programs. Furthermore the nature of large Python codebases like db manager makes it very easy to accidentally break functionality with no warning or errors during development.

 

Proposed solution

We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.

The clear advantages are:

• no duplicate functionality, so it’s easier for users to understand and use
• more usage of well-tested and well-maintained core C++ API
• testability and immediate feedback on API breaks (an advantage of C++ is that the application won’t even build if an API is changed or accidentally misused)
• better performance
• the ability to expose database management functionality via stable PyQGIS API, allowing other plugins and scripts to utilise this functionality. In future, Processing algorithms may also be developed which would take advantage of these functions (e.g. “create schema”, “drop table”, “vacuum table” algorithms)
• DB management functionality would be available within the main QGIS window (from the Browser panel), instead of as a separate dialog.

 

Grant proposal package

The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.

• create an interface for databases that expose the required functions to a coherent API
• add missing tests and documentation for the a.m. API
• porting some basic functions from db manager to the new api:
  • create table (with native field types support)
  • create schema
  • delete table
  • Rename table

The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.

The post QGIS Abstract Connections API first appeared on Open Web Solutions, GIS & Python Development.

What is OpenCL?

From https://en.wikipedia.org/wiki/OpenCL:

OpenCL (Open Computing Language) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) and other processors or hardware accelerators. OpenCL specifies programming languages (based on C99 and C++11) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. OpenCL provides a standard interface for parallel computing using task- and data-based parallelism.

Basically, you write a program and you execute it on a GPU (or, less frequently, on a CPU or on a DSP) taking advantage of the huge parallel programming capabilities of the modern graphic cards.

Depending on many different factors, the speed gain can vary to a great extent, but it is typically around one order of magnitude.

How QGIS benefits from OpenCL?

The work I’ve done consisted in integrating OpenCL support into QGIS and writing all the utilities to load, build and run OpenCL programs.

For now, I’ve ported the following QGIS core algorithms, all of them are availabe in processing:

• slope
• aspect
• hillshade
• ruggedness

Since the framework to support OpenCL is now in place, I think that more algorithms will be ported over the time.

During this development, even if was not in scope, the hillshade renderer has been optimized for speed and it can also benefit of OpenCL acceleration.

How to activate OpenCL support

OpenCL support is optional and opt-in, to use it, you need to activate it into the QGIS options dialog like shown in the screenshot below:

How much performance gain can I expect?

Well, YMMV, but here are some figures for a big DEM raster, low values mean faster execution.

GDAL means CPU execution using the GDAL processing algorithm.

How to install the OpenCL drivers?

Of course it depends on your specific hardware and on your O.S., AMD, NVidia and Intel have different distributions channels, in general the driver for your graphic card will also provide the OpenCL driver, if your GPU is compatible, if OpenCL is not available on your current machine, try to Google for OpenCL, your O.S. and graphic card.

If there is no OpenCL support for your graphic card, you might try to install a driver for your GPU (Intel for example provides them) and you will probably have a decent acceleration even if not as much as you can get on a real graphic card.

This fact worths some more explanation: you might ask your self why running and algorithm directly on the CPU and running it on the same CPU but using OpenCL would make any difference and the reason why it is generally faster by using OpenCL is that OpenCL will run the algorithm in parallel on all cores of your CPU, while a normal application (and QGIS does not make an exception here) will use a single core.

How to build QGIS with OpenCL support on Ubuntu

Just a quick note: you’ll need to install the OpenCL headers and the ICD library:

sudo apt-get install opencl-headers ocl-icd-opencl-dev

 

Credits

I started this work as a proof of concept in my spare time (that it is not much, lately) and when I realized that it was promising, I submitted a QGIS grant proposal in order to allocate some working time to port more algorithms, write tests and polish the implementation.

This work would not be possible without all the generous sponsors and donors that feed the QGIS grant program year after year, many thanks to the QGIS community for this amazing support!

Jürgen Fischer was as usual very helpful and took care of the windows builds, now available in OSGeo4W packages.

Nyall Dawson helped with the code review and with testing the implementation on different cards and machines.

Matthias Kuhn reviewed the code.

Even Rouault pointed me to some highly efficient GDAL algorithm optimizations that I’ve been able to integrate in QGIS.

 

 

The post OpenCL acceleration now available in QGIS first appeared on Open Web Solutions, GIS & Python Development.

 

Why python data providers?

My main reasons for having Python data provider were:

• quick prototyping
• web services
• why not?

 

This topic has been floating in my head for a while since I decided to give it a second look and I finally implemented it and merged for the next 3.2 release.

 

How it’s been done

To make this possible I had to:

• create a public API for registering the providers
• create the Python bindings (the hard part)
• create a sample Python vector data provider (the boring part)
• make all the tests pass

 

First, let me say that it wasn’t like a walk in the park: the Python bindings part is always like diving into woodoo and black magic recipes before I can get it to work properly.

For the Python provider sample implementation I decided to re-implement the memory (aka: scratch layers) provider because that’s one of the simplest providers and it does not depend on any external storage or backend.

 

How to and examples

For now, the main source of information is the API and the tests:

• • https://github.com/qgis/QGIS/blob/master/src/core/qgsproviderregistry.h#L188
  • https://github.com/qgis/QGIS/blob/master/tests/src/python/provider_python.py
  • https://github.com/qgis/QGIS/blob/master/tests/src/python/test_provider_python.py

To register your own provider (PyProvider in the snippet below) these are the basic steps:

metadata = QgsProviderMetadata(PyProvider.providerKey(), PyProvider.description(), PyProvider.createProvider)
QgsProviderRegistry.instance().registerProvider(metadata)

To create your own provider you will need at least the following components:

• the provider class itself (subclass of QgsVectorDataProvider)
• a feature source (subclass of QgsAbstractFeatureSource)
• a feature iterator (subclass of QgsAbstractFeatureIterator)

Be aware that the implementation of a data provider is not easy and you will need to write a lot of code, but at least you could get some inspiration from the existing example.

 

Enjoy wirting data providers in Python and please let me know if you’ve fond this implementation useful!

The post Create a QGIS vector data provider in Python is now possible first appeared on Open Web Solutions, GIS & Python Development.

Recently I was invited by the colleagues from OpenGIS.ch to lend a hand in a training session about QGIS server.

This was a good opportunity to update my presentation for QGIS3, to fix a few bugs and to explore the powerful capabilities of QGIS server and Python.

As a result, I published the full recipe of a Vagrant VM on github: https://github.com/elpaso/qgis3-server-vagrant

The presentation is online here: http://www.itopen.it/bulk/qgis3-server/

What’s worth mentioning is the sample plugins (I’ll eventually package and upload them to the official plugin site):

• XYZ: add simple XYZ tile server, ready to use within QGIS XYZ connections: https://github.com/elpaso/qgis3-server-vagrant/tree/master/resources/web/plugins/xyz
• custom service, does nothing but shows how to create a Python custom service by exploiting the new service capabilities available in QGIS3: https://github.com/elpaso/qgis3-server-vagrant/tree/master/resources/web/plugins/customservice
• HTTP Basic auth (how to add your auth scheme to QGIS server): https://github.com/elpaso/qgis3-server-vagrant/tree/master/resources/web/plugins/httpbasic
• more …

 

The VM uses 4 different (although similar) deployment strategies:

• good old Apache + mod_fcgi and plain CGI
• Nginx + Fast CGI
• Nginx + standalone HTTP Python wrapped server
• Nginx + standalone WSGI Python wrapped server

Have fun with QGIS server: it was completely refactored in QGIS 3 and it’s now better than ever!

 

The post QGIS 3 Server deployment showcase with Python superpowers first appeared on Open Web Solutions, GIS & Python Development.

Do you want to share your GPS data from your phone to QGIS? Here is how:   QGIS comes with a core plugin named GPS Tools that can be enabled in the Plugin installer dialog:   There are several ways to forward data from your phone and most of them are very well described in the QGIS manual page: https://docs.qgis.org/testing/en/docs/user_manual/working_with_gps/plugins_gps.html What I’m going to describe here is mostly useful when your phone and your host machine running QGIS are on the same network (for example they are connected to the same WiFi access point) and it is based on the simple application GPS 2 NET   Once the application is installed and started on your phone, you need to know the IP address of the phone, on a linux box you can simply run a port scanner and it will find all devices connected to the port 6000 (the default port used by GPS 2 NET):  

# Assuming your subnet is 192.168.9

nmap -p 6000 192.168.1.*

Nmap scan report for android-8899989888d02271.homenet.telecomitalia.it (192.168.99.50)
Host is up (0.0093s latency).
PORT STATE SERVICE
6000/tcp open X11

  Now, in QGIS you can open the plugin dialog through Vector -> GPS -> GPS Tools and enter the IP address and port of your GPS device:   Click on Connect button on the top right corner (mouse over the gray square for GPS status information)   Start digitizing!

The post Use your android phone’s GPS in QGIS first appeared on Open Web Solutions, GIS & Python Development.

Last week I’ve been in Madeira at the hackfest, like all the past events this has been an amazing happening, for those of you who have never been there, a QGIS hackfest is typically an event where QGIS developers and other pasionate contributors like documentation writers, translators etc. gather together to discuss the future of their beloved QGIS software. QGIS hackfest are informal events where meetings are scheduled freely and any topic relevant to the project can be discussed. This time we have brought to the table some interesting topics like:

• the future of processing providers: should they be part of QGIS code or handled independently as plugins?
• the road forward to a better bug reporting system and CI platform: move to gitlab?
• the certification program for QGIS training courses: how (and how much) training companies should give back to the project?
• SWOT analysis of current QGIS project: very interesting discussion about the status of the project.
• QGIS Qt Quick modules for mobile QGIS app

Tehre were also some mentoring sessions where I presented:

• How to set up a development environment and make your first pull request
• How to write tests for QGIS (in both python and C++)

  At this link you can find all the video recordings of the sessions: https://github.com/qgis/QGIS/wiki/DeveloperMeetingMadeira2018   Here is a link to the Vagrant QGIS developer VM I’ve prepared for the session: https://github.com/elpaso/qgis-dev-vagrant/   I’ve got a good feedback from other devs about my sessions and I’m really happy that somebody found them useful, one of the main goals of a QGIS hackfest should really be to help other developers to ramp up quicly into the project. Other than that, I’ve also find the time to update to QGIS 3.0 some of my old plugins like GeoCoding and QuickWKT.   Thanks to Giovanni Manghi and to Madeira Government for the organizazion and thanks to all QGIS sponsors and donors!   About me: I started as a QGIS plugin author, continued as the developer of the plugin official repository at https://plugins.qgis.org and now I’m one of the top 5 QGIS core contributors. After almost 10 years that I’m in the QGIS project I’m now not only a proud member of the QGIS community but also an advocate for the open source GIS software movement.

The post Welcome QGIS 3 and bye bye Madeira first appeared on Open Web Solutions, GIS & Python Development.

Building QGIS from sources is not hard at all on a recent linux box, but what about if you wanted to be able to step-debug into Qt core or if you wanted to build QGIS agains the latest Qt release? Here things become tricky. This short post is about my experiments to build Qt and and other Qt-based dependencies for QGIS in order to get a complete debugger-friendly build of QGIS.   Start with downloading the latest Qt installer from Qt official website: https://www.qt.io/download-qt-for-application-development choose the Open Source version.   Now install the Qt version you want to build, make sure you check the Sources and the components you might need. Whe you are done with that, you’ll have your sources in a location like /home/user/Qt/5.9.3/Src/ To build the sources, you can change into that directory and issue the following command – I assume that you have already installed all the dependencies normally needed to build C++ Qt programs – I’m using clang here but feel free to choose gcc, we are going to install the new Qt build into /opt/qt593.

./configure -prefix /opt/qt593 -debug -opensource -confirm-license -ccache -platform linux-clang

When done, you can build it with

make -j9
sudo make install

  To build QGIS you also need three additional Qt packages   QtWebKit from https://github.com/qt/qtwebkit (you can just download the zip): Extract it somewhere and build it with

/opt/qt593/bin/qmake WebKit.pro
make -j9
sudo make install

  Same with QScintila2 from https://www.riverbankcomputing.com/software/qscintilla

/opt/qt593/bin/qmake qscintilla.pro
make -j9
sudo make install

  QWT is also needed and it can be downloaded from https://sourceforge.net/projects/qwt/files/qwt/6.1.3/ but it requires a small edit in qwtconfig.pri before you can build it: set QWT_INSTALL_PREFIX = /opt/qt593_libs/qwt-6.1.3 to install it in a different folder than the default one (that would possibly overwrite a system install of QWT). The build it with:

/opt/qt593/bin/qmake qwt.pro
make -j9
sudo make install

  If everything went fine, you can now configure Qt Creator to use this new debug build of Qt: start with creating a new kit (you can probably clone a working Qt5 kit if you have one). What you need to change is the Qt version (the path to cmake) to point to your brand new Qt build,: Pick up a name and choose the Qt version, but before doing that you need to click on Manage… to create a new one: Now you should be able to build QGIS using your new Qt build, just make sure you disable the bindings in the CMake configuration: unfortunately you’d also need to build PyQt in order to create the bindings.   Whe QGIS is built using this debug-enabled Qt, you will be able to step-debug into Qt core libraries! Happy debugging!  

The post Building QGIS master with Qt 5.9.3 debug build first appeared on Open Web Solutions, GIS & Python Development.

Here is a little first experiment for a wsgi wrapper to QGIS 3 Server, not much tested, but basically working:  

#!/usr/bin/env python

# Simple QGIS 3 Server wsgi test

import signal
import sys
from cgi import escape, parse_qs
from urllib.parse import quote
# Python's bundled WSGI server
from wsgiref.simple_server import make_server

from qgis.core import QgsApplication
from qgis.server import *

# Init QGIS
qgs_app = QgsApplication([], False)
# Init server
qgs_server = QgsServer()


def reconstruct_url(environ):
    """Standard algorithm to retrieve the full URL from wsgi request
    From: https://www.python.org/dev/peps/pep-0333/#url-reconstruction
    """
    url = environ['wsgi.url_scheme']+'://'

    if environ.get('HTTP_HOST'):
        url += environ['HTTP_HOST']
    else:
        url += environ['SERVER_NAME']

        if environ['wsgi.url_scheme'] == 'https':
            if environ['SERVER_PORT'] != '443':
                url += ':' + environ['SERVER_PORT']
        else:
            if environ['SERVER_PORT'] != '80':
                url += ':' + environ['SERVER_PORT']

    url += quote(environ.get('SCRIPT_NAME', ''))
    url += quote(environ.get('PATH_INFO', ''))
    if environ.get('QUERY_STRING'):
        url += '?' + environ['QUERY_STRING']
    return url

def application (environ, start_response):

    headers = {} # Parse headers from environ here if needed

    # the environment variable CONTENT_LENGTH may be empty or missing
    # When the method is POST the variable will be sent
    # in the HTTP request body which is passed by the WSGI server
    # in the file like wsgi.input environment variable.
    try:
        request_body_size = int(environ.get('CONTENT_LENGTH', 0))
        request_body = environ['wsgi.input'].read(request_body_size)
    except (ValueError):
        request_body_size = 0
        request_body = None

    request = QgsBufferServerRequest(reconstruct_url(environ), (QgsServerRequest.PostMethod 
        if environ['REQUEST_METHOD'] == 'POST' else QgsServerRequest.GetMethod), {}, request_body)
    response = QgsBufferServerResponse()
    qgs_server.handleRequest(request, response)
    headers_dict = response.headers()
    try:
        status = headers_dict['Status']
    except KeyError:
        status = '200 OK'
    start_response(status, [(k, v) for k, v in headers_dict.items()])
    return [bytes(response.body())]

# Instantiate the server
httpd = make_server (
    'localhost', # The host name
    8051, # A port number where to wait for the request
    application # The application object name, in this case a function
)

print("Listening to http://localhost:8051 press CTRL+C to quit")

def signal_handler(signal, frame):
    """Exit QGIS cleanly"""
    global qgs_app
    print("\nExiting QGIS...")
    qgs_app.exitQgis()
    sys.exit(0)

signal.signal(signal.SIGINT, signal_handler)

httpd.serve_forever()

 

The post A little QGIS3 Server wsgi experiment first appeared on Open Web Solutions, GIS & Python Development.

Another great QGIS hackfest is gone, and it’s time for a quick report. The location was the Linux Hotel, one of the best places where open source developers could meet, friendly, geek-oriented and when the weather is good, like this time, villa Vogelsang is a wonderful place to have a beer in the garden while talking about software development or life in general. This is a short list of what kept me busy during the hackfest:

• fixed some bugs and feature requests on the official QGIS plugin repo that I’m maintaining since the very beginning
• make the QGIS official plugin repository website mobile-friendly
• QGIS Server Python Plugin API refactoring, I’ve completed the work on the new API, thanks to the ongoing server refactoring it’s now much cleaner than it was in the first version
• attribute table bugs: I started to address some nasty bugs in the attribute table, some of those were fixed during the week right after the hackfest
• unified add layer button, we had a productive meeting where we decided the path forward to implement this feature, thanks to Boundless that is funding the development, this feature is what’s I’m currently working on these days

Thanks to all QGIS donors and funders that made yet another great hackfest possible and in particular to Boundless Spatial Inc. for funding my personal expenses.    

The post Essen 2017 QGIS Hackfest first appeared on Open Web Solutions, GIS & Python Development.

QGIS Developer Sprint in Lyon   QGIS Server 3.0 is going to be better than ever! Last week I attended to the mini code-sprint organized by the french QGIS developers in Lyon.   The code sprint was focused on QGIS Server refactoring to reach the following goals:

• increase maintainability through modularity and clean code responsibilities
• increase performances
• better multi-project handling and caching
• scalability
• multi threaded rendering

By working for different companies on such a big Open Source project like QGIS, coordination between developers is fundamentally achieved through those kind of events. We were a small group of engaged QGIS Server developers and I think that the alternance between brainstorming and coding has proven to be very productive: after two days we were able to set common milestones and commitments that will ensure a bright future to QGIS Server. A huge and warm thank to the french QGIS developers that organized this meeting!   Photo: courtesy of Règis Haubourg    

The post QGIS Developer Sprint in Lyon first appeared on Open Web Solutions, GIS & Python Development.

Today I’m releasing the first version of QGIS Server Simple Browser Plugin, a simple Server plugin that generates a browsable table of contents of the project’s layers and a link to an OpenLayers map.  

How it works

The plugin adds an XSL stylesheet to GetProjectsettings XML response, the generated HTML looks like this:  

The openlayers format

The map preview is generated by adding a new application/openlayers FORMAT option to GetMap requests, the generated map automatically fits to the layer’s extent and has basic GetFeatureInfo capabilities.

Limitations

The current version only supports EPSG:4326 that must be available (enabled) on the server.

Source code and download

The plugin is available on the official repository: ServerSimpleBrowser The code is on GitHub.

Sometimes is hard to debug segfaults appearing in QGIS Server when running in CGI mode. The classic approach is attaching a gdb to the running process. The problem is that there is not enough time to do it! A simple plugin filter, can provide you the time you need to attach the debugger:

from qgis.server import *
from qgis.core import *
import os
     
class DelayFilter(QgsServerFilter):
     
    def __init__(self, serverIface):
        super(DelayFilter, self).__init__(serverIface)    
     
    def responseComplete(self):        
        request = self.serverInterface().requestHandler()
        params = request.parameterMap()
        if params.get('DELAY', ''):
            QgsMessageLog.logMessage("PID: %s" % os.getpid())     
            import time
            time.sleep(30)

Calling the server with DELAY=1 will wait for 30 seconds and print the current PID in the server logs. This will give you enough time to fire gdb and attach it to the process.

   

Description

This is a core-enhancement QGIS plugin that makes the implementation of complex dynamic filters in QGIS attribute forms an easy task. For example, this widget can be used to implement drill-down forms, where the values available in one field depend on the values of other fields.

Download

The plugin is available on the official QGIS Python Plugin Repository and the source code is on GitHub QGIS Form Value Relation plugin repository

Implementation

The new “Form Value Relation” widget is essentially a clone of the core “Value Relation” widget with some important differences: When the widget is created:

• the whole unfiltered features of the related layer are loaded and cached
• the form values of all the attributes are added to the context (see below)
• the filtering against the expression happens every time the widget is refreshed
• a signal is bound to the form changes and if the changed field is present in the filter expression, the features are filtered against the expression and the widget is refreshed

Using form values in the expression

A new expression function is available (in the “Custom” section):

CurrentFormValue('FIELD_NAME')

This function returns the current value of a field in the editor form.

Note

1. This function can only be used inside forms and it’s particularly useful when used together with the custom widget `Form Value Relation`
2. If the field does not exists the function returns an empty string.

Visual guide

  Download the example project.   This is the new widget in action: changing the field FK_PROV, the ISTAT values are filtered according to the filter expression.

First I wish to thank Pablo & friends for the amazing organization, unfortunately I couldn’t spend more than two full days there, but those two days have been memorable! Here is a picture of one of the most interesting discussions (photo: courtesy of Pablo). An hack fest is an event for writing good code but what it’s really good at is to establish and cultivate relations with other coders, to exchange opinions and ideas and last but not least to have some fun and make new friends.   This time, we have had many interesting presentations and a couple of meetings where we spoke about technical aspects of the project management and infrastructure and about some important challenges, both in terms of code size and economical implications for who relies on it, that a growing project must face.   The latter was something I’ve also been considering for a while: now that pull requests (PR) for new features are coming down the pipeline, we must find a better way to manage their queue by giving a clear and transparent approval path and deadline. This management and approval process cannot rely entirely on volunteer work, the main reason being that most of the times the PR proposers have been paid for that PR and it’s not fair (nor reliable) that the (sometimes hard) job of doing a code review is not rewarded. On the other end, an investor cannot waste its time and money on a project without having a reasonable good chance to see its work eventually land into the core of QGIS.   Hugo (thanks for that!) organized a meeting to discuss this topic, that crosses personal business interests, ethical considerations and personal beliefs to a point that it’s not really easy to discuss in a calm and objective way, despite the premises, the discussion was very interesting and constructive and a QEP that tries to address at least some of this problems is open for discussion right now: https://github.com/qgis/QGIS-Enhancement-Proposals/issues/52   Another topic we’ve been discussing was how to manage python plugin dependencies, we’ve decided to start by adding a new metadata tag called external_deps that’s supposed to contain the PIP install string for the required packages, since PIP will be a builtin in python 3.4, that will probably solve most of our problems when we’ll integrate that into the plugin manager. At the moment the metadata is not documented nor required, but it’s there to allow for experiments.   We didn’t miss the occasion to talk about the ugly bug that affects fTools, not something I’m going to dig into in this post though.   Of course an hack fest is still a good opportunity for squashing bugs and implement new cool features, I’ve been busy mainly on the following topics:

1. HiDPI screen support for web view widgets (help and plugin manager/installer)
2. Form relations editing longstanding bugs
3. New feature to optionally enter, edit and store Python form init code into the project (and DB), see the picture below
4. Plugins website maintenance (added new metadata and fixed a few bugs, added an RPC call to export author email for admins)

  Thanks to all participants, to the organizers and to all QGIS sponsors and donors that made this possible!  

With QGIS 2.12 the new Python bindings for QGIS server are now available and the server can be invoked directly from a python scripts with just a few lines of code:

 

from qgis.server import QgsServer
my_query_string = "map=/projects/my_project.qgs&SERVICE=WMS&request=GetCapabilities"
headers, body =  QgsServer().handleRequest(my_query_string)

 

Embedding QGIS in a Python web application

Embedding QGIS Server in a web application is now not only possible but really very easy, for example, a Django view:

# QGIS server view

from django.http import HttpResponse
from django.views.generic import View
from qgis.server import *


class OGC(View):
    """Pass a GET request to QGIS Server and return the response"""

    def __init__(self):
        self.server = QgsServer()        

    def get(self, request, *args, **kwargs):
        """Pass a GET request to QGIS Server and return the response"""
        headers, body = self.server.handleRequest(request.GET.urlencode())
        response = HttpResponse(body)
        # Parse headers
        for header in headers.split('\n'):
            if header:
                k, v = header.split(': ', 1)
                response[k] = v
        return response

 

Using server plugins

 

Of course Python server plugins can be plugged in easily, see the example below:

 

# QGIS server view

from django.http import HttpResponse
from django.views.generic import View
from qgis.server import *
from qgis.core import *



class OGC(View):
    """Pass a GET request to QGIS Server and return the response"""

    def __init__(self):
        self.server = QgsServer()
        # Call init to create serverInterface
        self.server.init()
        serverIface = self.server.serverInterface()

        class Filter1(QgsServerFilter):
            def responseComplete(self):
                QgsMessageLog.logMessage("Filter1.responseComplete", "Server", QgsMessageLog.INFO )
                request = self.serverInterface().requestHandler()
                if request.parameter('REQUEST') == 'HELLO':
                    request.clearHeaders()
                    request.setHeader('Content-type', 'text/plain')
                    request.clearBody()
                    request.appendBody('Hello from SimpleServer!')

            def requestReady(self):
                QgsMessageLog.logMessage("Filter1.requestReady")

        self.filter = Filter1(serverIface)
        serverIface.registerFilter(self.filter)


    def get(self, request, *args, **kwargs):
        """Pass a GET request to QGIS Server and return the response"""
        headers, body = self.server.handleRequest(request.GET.urlencode())
        response = HttpResponse(body)
        # Parse headers
        for header in headers.split('\n'):
            if header:
                k, v = header.split(': ', 1)
                response[k] = v
        return response

 

Enjoy QGIS Server with Python!

A few days ago the new WebGIS of the City of Asti, a 76000 inhabitants city in Piedmont, was launched.  The new WebGIS uses QGIS Server and QGIS Web Client to serve maps and provide street and cadastrial search and location services.

The new WebGIS was developed by ItOpen and is online at: http://sit.comune.asti.it/site/?map=PRGAsti

Some plugin core functions can now be called from a Python console:

g = QgsGeometry.fromWkt('POINT (9.9 43)')
iface.show_geometry(g)
iface.show_geometry(g.buffer(0.2, 2))
iface.show_wkt('POINT (9 45)')
iface.show_wkb(r'0103...') # cut

All functions accept a layer title as optional argument, if None is given, they are automatically added to a Quick WKT GeometryType (memory) layer, such as Quick WKT Polygon for polygons.

During the hackfest I’ve been working on the refactoring of the server component, aimed to wrap the server into a class and create python bindings for the new classes. This work is now in the PR queue and brings a first working python test for the server itself.

The server can now be invoked directly from python, like in the example below:

 

#!/usr/bin/env python
"""
Super simple QgsServer.
"""

from qgis.server import *
from BaseHTTPServer import *

class handler (BaseHTTPRequestHandler):

    server = QgsServer()

    def _doHeaders(self, response):
        l = response.pop(0)
        while l:
            h = l.split(':')
            self.send_header(h[0], ':'.join(h[1:]))
            self.log_message( "send_header %s - %s" % (h[0], ':'.join(h[1:])))
            l = response.pop(0)
        self.end_headers()

    def do_HEAD(self):
        self.send_response(200)
        response = str(handler.server.handleRequestGetHeaders(self.path[2:])).split('\n')
        self._doHeaders(response)

    def do_GET(self):
        response = str(handler.server.handleRequest(self.path[2:])).split('\n')
        i = 0
        self.send_response(200)
        self._doHeaders(response)
        self.wfile.write(('\n'.join(response[i:])).strip())

    def do_OPTIONS(s):
        handler.do_GET(s)

httpd = HTTPServer( ('', 8000), handler)

while True:
    httpd.handle_request()

The python bindings capture the server output instead of printing it on FCGI stdout and allow to pass the request parameters QUERY_STRING directly to the request handler as a string, this makes writing python tests very easy.

Prerequisites

I assume that you are working on a fresh install with Apache and FCGI module installed with:

$ sudo apt-get install apache2 libapache2-mod-fcgid
$ # Enable FCGI daemon apache module
$ sudo a2enmod fcgid

Package installation

First step is to add debian gis repository, add the following repository:

$ cat /etc/apt/sources.list.d/debian-gis.list
deb http://qgis.org/debian trusty main
deb-src http://qgis.org/debian trusty main

$ # Add keys
$ sudo gpg --recv-key DD45F6C3
$ sudo gpg --export --armor DD45F6C3 | sudo apt-key add -

$ # Update package list
$ sudo apt-get update && sudo apt-get upgrade

Now install qgis server:

$ sudo apt-get install qgis-server python-qgis

Install the HelloWorld example plugin

This is an example plugin and should not be used in production!
Create a directory to hold server plugins, you can choose whatever path you want, it will be specified in the virtual host configuration and passed on to the server through an environment variable:

$ sudo mkdir -p /opt/qgis-server/plugins
$ cd /opt/qgis-server/plugins
$ sudo wget https://github.com/elpaso/qgis-helloserver/archive/master.zip
$ # In case unzip was not installed before:
$ sudo apt-get install unzip
$ sudo unzip master.zip 
$ sudo mv qgis-helloserver-master HelloServer

Apache virtual host configuration

We are installing the server in a separate virtual host listening on port 81.
Rewrite module can be optionally enabled to pass HTTP BASIC auth headers (only needed by the HelloServer example plugin).

$ sudo a2enmod rewrite

Let Apache listen to port 81:

$ cat /etc/apache2/conf-available/qgis-server-port.conf
Listen 81
$ sudo a2enconf qgis-server-port

The virtual host configuration, stored in /etc/apache2/sites-available/001-qgis-server.conf:

<VirtualHost *:81>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html

    ErrorLog ${APACHE_LOG_DIR}/qgis-server-error.log
    CustomLog ${APACHE_LOG_DIR}/qgis-server-access.log combined

    # Longer timeout for WPS... default = 40
    FcgidIOTimeout 120 
    FcgidInitialEnv LC_ALL "en_US.UTF-8"
    FcgidInitialEnv PYTHONIOENCODING UTF-8
    FcgidInitialEnv LANG "en_US.UTF-8"
    FcgidInitialEnv QGIS_DEBUG 1
    FcgidInitialEnv QGIS_SERVER_LOG_FILE /tmp/qgis-000.log
    FcgidInitialEnv QGIS_SERVER_LOG_LEVEL 0
    FcgidInitialEnv QGIS_PLUGINPATH "/opt/qgis-server/plugins"

    # ABP: needed for QGIS HelloServer plugin HTTP BASIC auth
    <IfModule mod_fcgid.c>
        RewriteEngine on
        RewriteCond %{HTTP:Authorization} .
        RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
    </IfModule>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride All
        Options +ExecCGI -MultiViews +FollowSymLinks
        Require all granted
        #Allow from all
  </Directory>
</VirtualHost>

Enable the virtual host and restart Apache:

$ sudo a2ensite 001-qgis-server
$ sudo service apache2 restart

Test:

$ wget -q -O - "http://localhost:81/cgi-bin/qgis_mapserv.fcgi?SERVICE=HELLO"
HelloServer!

See all QGIS Server related posts

There have been some requests in the past about custom CSS for html GetFeatureInfo responses from QGIS Server.

Currently, the HTML response template is hardcoded and there is no way to customize it, the Python plugin support introduced with the latest version of QGIS Server provides an easy way to add some custom CSS rules or even provide custom templates.

To get you started, I’ve added a new filter to my example  HelloServer plugin:

import os

from qgis.server import *
from qgis.core import *
from PyQt4.QtCore import *
from PyQt4.QtGui import *


class GetFeatureInfoCSSFilter(QgsServerFilter):

    def __init__(self, serverIface):
        super(GetFeatureInfoCSSFilter, self).__init__(serverIface)

    def requestReady(self):
        """Nothing to do here, but it would be the ideal point
        to alter request **before** it gets processed, for example
        you could set INFO_FORMAT to text/xml to get XML instead of
        HTML in responseComplete"""
        pass

    def responseComplete(self):
        request = self.serverInterface().requestHandler()
        params = request.parameterMap( )
        if (params.get('SERVICE').upper() == 'WMS' \
                and params.get('REQUEST', '').upper() == 'GETFEATUREINFO' \
                and params.get('INFO_FORMAT', '').upper() == 'TEXT/HTML' \
                and not request.exceptionRaised() ):
            body = request.body()
            body.replace('<BODY>', """<BODY><STYLE type="text/css">* {font-family: arial, sans-serif; color: blue;}</STYLE>""")
            # Set the body
            request.clearBody()
            request.appendBody(body)

This filter is pretty simple, if the request is a WMS GetFeatureInfo with HTML format, it injects a STYLE tag into the HTML HEAD.

Here is the output with blue color and arial fonts applied:

As an exercise left to the reader, you can also intercept the call in requestReady(self), change the INFO_FORMAT to text/xml and then do some real templating, for example by using XSLT or by parsing the XML and injecting the values into a custom template.