Related Plugins and Tags

QGIS Planet

Automatically restarting services after upgrades on Debian and Ubuntu

There are various tools to automatically keep a Debian/Ubuntu system security wise up to date, among others the unattended-upgrades package.

Also, there’s the checkrestart script from the debian-goodies package, that scans all the open files on a system and tries to determine to what service they belong and how that service might be restarted.

The last piece that’d tie all those scripts together and would automatically restart all services that are using stale libraries or files was missing.

With the help of Michal Fiala there however is now the restart-services script, that does just that.

The script has not seen much real world usage and as such should be regarded as experimental (f.ex. by restarting /etc/init.d/screen it will as of the time of writing terminate existing screen sessions).

The script currently lives on Github. If you encounter any problem with the script then we’ll very much wellcome a patch that fixes it…

Tomáš Pospíšek

Update 18.6.2012: The most recent checkrestart (from debian-goodies 0.61) now excludes screen from beeing listed among the services to be restarted.

Automatically restarting services after upgrades on Debian and Ubuntu

There are various tools to automatically keep a Debian/Ubuntu system security wise up to date, among others the unattended-upgrades package.

Also, there’s the checkrestart script from the debian-goodies package, that scans all the open files on a system and tries to determine to what service they belong and how that service might be restarted.

The last piece that’d tie all those scripts together and would automatically restart all services that are using stale libraries or files was missing.

With the help of Michal Fiala there however is now the restart-services script, that does just that.

The script has not seen much real world usage and as such should be regarded as experimental (f.ex. by restarting /etc/init.d/screen it will as of the time of writing terminate existing screen sessions).

The script currently lives on Github. If you encounter any problem with the script then we’ll very much wellcome a patch that fixes it…

Tomáš Pospíšek

Update 18.6.2012: The most recent checkrestart (from debian-goodies 0.61) now excludes screen from beeing listed among the services to be restarted.

annotating third party web pages

Problem: I have a web site/page that I visit regularily which I want to annotate with my notes.

More specifically, I was regularly searching through the Homegate real estate hub looking for a new home. It goes without saying that I was again and again forgetting which objects I had already looked at, which objects were really interesting and I should check out more closely.

Therefore the need to annotate search results.

The here presented approach should be applicable for annotation of other web pages as well. It’s based on the observation, that restful web applications need to operate with asset IDs. These asset IDs can be reused to enrich the asset locally with additional data, such as notes. Thus we’re looking for those IDs in specific elements of the page and add a bit of HTML markup to those places.

Of course that aproach only works as long as the web site doesn’t heavily change its markup and doesn’t rendomly change asset IDs.

Here’s a screenshot of regular Homegate search results:

And here’s the same page after scripting it with Greasemonkey:

You’ll notice the input field with the comment in it.

The idea is simple: add an input box to each search result, where you write your comment. When the focus leaves the input box, the comment is stored to localStorage.

Starting with Greasemonkey is quite easy. There are howtos and templates to start from, such as this one.

However, regular JavaScripting and Greasemonkey JavaScripting do not work in exactly the same way:

One of the differences is that Greasemonkey creates a separate JavaScript environment, in which the Greasemonkey scripts are executed. That is calling Javascript contained in the page from Greasemonkey and the inverse calling Greasemonkey scripts from the page is not possible by default. This is on purpose, so that the web page can not detect and not interfere with the Greasemonkey scripts, because you want your Greasemonkey scripts to work allways on some page, whether or not that page likes it or not. Therefore no interference is possible.

Greasemonkey however provides a standard way to access the web page’s scripts, and that’s through the “unsafeWindow” object, which is a reference to the web page’s environment.

I had two mechanisms I had to make accessible using the “unsafeWindow” handle:

  • the first was accessing JQuery, which is included by default by the Homegate page. Since I needed to use JQuery functions in my Greasemonkey script, I got a reference to it via the standard Greasemonkey precedure:
    var jQuery = unsafeWindow['jQuery'];
  • the second mechanism that needed to cross the boundaries between the web page and Greasemonkey was callbacks from the web page to my Greasemonkey script. This is necessary, because I’m attaching “input” elements to each search result, which contain a note and which, “onblur”, need to call a function that saves the content of the input box. Here’s part that constructs the input element:
    jQuery("<input onclick='event.cancelBubble = true;'" +
                 " onblur='saveComment(this, immoID);'>").insertAfter(immoElement);

And this is the function that gets called back by “onblur”:

    unsafeWindow.saveComment = function(element, immoID) {
      unsafeWindow.localStorage.setItem(immoID, element.value);
    };

The next interesting thing you’ll note is usage of ‘locaStorage’. Support for the latter in browsers does not seem mature yet. One problem I’ve encountered when developing under Firefox 3.6 was that saving to ‘localStorage’ was not possible when cookies were disabled (see this report). Thus you’ll need to permanently enable cookies for Homegate in order for the script to be able to save its data.

Finally, while developing, a major problem was, that Firefox did not show me errors in the Greasemonkey scripts. Thus either the script would work or not work and fail completely silently. That made debugging a bit painful.

So now, here’s the script.

Tomáš Pospíšek

PS: This script also lives at userscripts.org

annotating third party web pages

Problem: I have a web site/page that I visit regularily which I want to annotate with my notes.

More specifically, I was regularly searching through the Homegate real estate hub looking for a new home. It goes without saying that I was again and again forgetting which objects I had already looked at, which objects were really interesting and I should check out more closely.

Therefore the need to annotate search results.

The here presented approach should be applicable for annotation of other web pages as well. It’s based on the observation, that restful web applications need to operate with asset IDs. These asset IDs can be reused to enrich the asset locally with additional data, such as notes. Thus we’re looking for those IDs in specific elements of the page and add a bit of HTML markup to those places.

Of course that aproach only works as long as the web site doesn’t heavily change its markup and doesn’t rendomly change asset IDs.

Here’s a screenshot of regular Homegate search results:

And here’s the same page after scripting it with Greasemonkey:

You’ll notice the input field with the comment in it.

The idea is simple: add an input box to each search result, where you write your comment. When the focus leaves the input box, the comment is stored to localStorage.

Starting with Greasemonkey is quite easy. There are howtos and templates to start from, such as this one.

However, regular JavaScripting and Greasemonkey JavaScripting do not work in exactly the same way:

One of the differences is that Greasemonkey creates a separate JavaScript environment, in which the Greasemonkey scripts are executed. That is calling Javascript contained in the page from Greasemonkey and the inverse calling Greasemonkey scripts from the page is not possible by default. This is on purpose, so that the web page can not detect and not interfere with the Greasemonkey scripts, because you want your Greasemonkey scripts to work allways on some page, whether or not that page likes it or not. Therefore no interference is possible.

Greasemonkey however provides a standard way to access the web page’s scripts, and that’s through the “unsafeWindow” object, which is a reference to the web page’s environment.

I had two mechanisms I had to make accessible using the “unsafeWindow” handle:

  • the first was accessing JQuery, which is included by default by the Homegate page. Since I needed to use JQuery functions in my Greasemonkey script, I got a reference to it via the standard Greasemonkey precedure:
    var jQuery = unsafeWindow['jQuery'];
  • the second mechanism that needed to cross the boundaries between the web page and Greasemonkey was callbacks from the web page to my Greasemonkey script. This is necessary, because I’m attaching “input” elements to each search result, which contain a note and which, “onblur”, need to call a function that saves the content of the input box. Here’s part that constructs the input element:
    jQuery("<input onclick='event.cancelBubble = true;'" +
                 " onblur='saveComment(this, immoID);'>").insertAfter(immoElement);

And this is the function that gets called back by “onblur”:

    unsafeWindow.saveComment = function(element, immoID) {
      unsafeWindow.localStorage.setItem(immoID, element.value);
    };

The next interesting thing you’ll note is usage of ‘locaStorage’. Support for the latter in browsers does not seem mature yet. One problem I’ve encountered when developing under Firefox 3.6 was that saving to ‘localStorage’ was not possible when cookies were disabled (see this report). Thus you’ll need to permanently enable cookies for Homegate in order for the script to be able to save its data.

Finally, while developing, a major problem was, that Firefox did not show me errors in the Greasemonkey scripts. Thus either the script would work or not work and fail completely silently. That made debugging a bit painful.

So now, here’s the script.

Tomáš Pospíšek

PS: This script also lives at userscripts.org

annotating third party web pages

Problem: I have a web site/page that I visit regularily which I want to annotate with my notes.

More specifically, I was regularly searching through the Homegate real estate hub looking for a new home. It goes without saying that I was again and again forgetting which objects I had already looked at, which objects were really interesting and I should check out more closely.

Therefore the need to annotate search results.

The here presented approach should be applicable for annotation of other web pages as well. It’s based on the observation, that restful web applications need to operate with asset IDs. These asset IDs can be reused to enrich the asset locally with additional data, such as notes. Thus we’re looking for those IDs in specific elements of the page and add a bit of HTML markup to those places.

Of course that aproach only works as long as the web site doesn’t heavily change its markup and doesn’t rendomly change asset IDs.

Here’s a screenshot of regular Homegate search results:

And here’s the same page after scripting it with Greasemonkey:

You’ll notice the input field with the comment in it.

The idea is simple: add an input box to each search result, where you write your comment. When the focus leaves the input box, the comment is stored to localStorage.

Starting with Greasemonkey is quite easy. There are howtos and templates to start from, such as this one.

However, regular JavaScripting and Greasemonkey JavaScripting do not work in exactly the same way:

One of the differences is that Greasemonkey creates a separate JavaScript environment, in which the Greasemonkey scripts are executed. That is calling Javascript contained in the page from Greasemonkey and the inverse calling Greasemonkey scripts from the page is not possible by default. This is on purpose, so that the web page can not detect and not interfere with the Greasemonkey scripts, because you want your Greasemonkey scripts to work allways on some page, whether or not that page likes it or not. Therefore no interference is possible.

Greasemonkey however provides a standard way to access the web page’s scripts, and that’s through the “unsafeWindow” object, which is a reference to the web page’s environment.

I had two mechanisms I had to make accessible using the “unsafeWindow” handle:

  • the first was accessing JQuery, which is included by default by the Homegate page. Since I needed to use JQuery functions in my Greasemonkey script, I got a reference to it via the standard Greasemonkey precedure:
    var jQuery = unsafeWindow['jQuery'];
  • the second mechanism that needed to cross the boundaries between the web page and Greasemonkey was callbacks from the web page to my Greasemonkey script. This is necessary, because I’m attaching “input” elements to each search result, which contain a note and which, “onblur”, need to call a function that saves the content of the input box. Here’s part that constructs the input element:
    jQuery("<input onclick='event.cancelBubble = true;'" +
                 " onblur='saveComment(this, immoID);'>").insertAfter(immoElement);

And this is the function that gets called back by “onblur”:

    unsafeWindow.saveComment = function(element, immoID) {
      unsafeWindow.localStorage.setItem(immoID, element.value);
    };

The next interesting thing you’ll note is usage of ‘locaStorage’. Support for the latter in browsers does not seem mature yet. One problem I’ve encountered when developing under Firefox 3.6 was that saving to ‘localStorage’ was not possible when cookies were disabled (see this report). Thus you’ll need to permanently enable cookies for Homegate in order for the script to be able to save its data.

Finally, while developing, a major problem was, that Firefox did not show me errors in the Greasemonkey scripts. Thus either the script would work or not work and fail completely silently. That made debugging a bit painful.

So now, here’s the script.

Tomáš Pospíšek

PS: This script also lives at userscripts.org

Web based printing with QGIS server

QGIS server is already known as a full featured, WMS 1.3 compliant map server (see e.g. ETHZ, Linfiniti or 3LIZ).

For the city of Uster, Switzerland, Sourcepole recently extended QGIS server with the possibility to use the print composer via WMS in order to offer printing functionality for web maps. A very nice GeoExt based client can be found at http://gis.uster.ch/webgis/. Andreas Neumann used and extended the GeoExt PrintProvider and PrintExtent classes which allows the user to intuitively select a layout, extent, scale, rotation and resolution for printing. The widget then sends a GetPrint request to QGIS server and a printable PDF is returned.

Now to the technical details: in order to use the print capability of QGIS server, the published project needs to contain one or more print compositions. QGIS server includes information about the available composer templates in the GetCapabilities response:

<WMS_Capabilities>
...
<ComposerTemplates xsi:type="wms:_ExtendedCapabilities">
<ComposerTemplate width="297" height="210" name="Composer 1">
<ComposerMap width="102" height="95" name="map0"/>
<ComposerMap width="133" height="79.87912087912088" name="map1"/>
<ComposerLabel name="hello_world"/>
</ComposerTemplate>
</ComposerTemplates>
...

With a 'GetPrint' request, the client may now request printable output in png or pdf format. The client needs to select an available composer template and needs to pass the selected extents for the composer maps. Additionally, there is the possibility to replace the text of composer labels. An example for a 'GetPrint' request is (note the new 'TEMPLATE' and 'map0:extent' Parameters):

http://localhost/fcgi-bin/qgis_mapserver/qgis_mapserv.fcgi?MAP=/home/marco/geodaten/projekte/composertest.qgs&SERVICE;=WMS&VERSION;=1.3.0&REQUEST;=GetPrint&TEMPLATE;=Composer 1&map0;:extent=693457.466131,227122.338236,700476.845177,230609.807051&BBOX;=693457.466131,227122.338236,700476.845177,230609.807051&CRS;=EPSG:21781&WIDTH;=1467&HEIGHT;=729&LAYERS;=layer0,layer1&STYLES;=,&FORMAT;=pdf&DPI;=300&TRANSPARENT;=true

In detail, the following parameters can be used to set properties for composer maps:
  • <mapname>:EXTENT=<xmin,ymin,xmax, ymax> //mandatory
  • <mapname>:ROTATION=<double> //optional, defaults to 0
  • <mapname>:SCALE=<double> //optional. Forces scale denominator as server and client may have different scale calculations
  • <mapname>:LAYERS=<comma separated list with layer names> //optional. Defaults to all layer in the WMS request
  • <mapname>:STYLES=<comma separated list with style names> //optional
  • <mapname>:GRID_INTERVAL_X=<double> //set the grid interval in x-direction for composer grids
  • <mapname>:GRID_INTERVAL_Y=<double> //set the grid interval in x-direction for composer grids

To replace text in composer labels, the label needs to have a text id (can be assigned in the print composer user interface of the composer label). The text to insert into the label can be passed in the GetPrint request with an additional parameter =text (of course, the label id should be different to the standard WMS parameters...).

Besides printing, an additional nice feature in the most recent developer version is the possibility to also use relative pathes in published project files. Like this, it's possible to conveniently export a project file recursively together with the corresponding data.

Finally, I'd like to thank the city of Uster and Andreas Neumann for the generous support and the creative ideas during the implementation of the printing functionality.

Working with big files/calling external scripts in JMeter

(This article is part of the JMeter Series)

JMeter’s “HTTP Request Sampler” will read the whole input (the downloaded page, pdf, movie etc.) into its main memory. That means that working with larger (f.ex. movies) files will bring JMeter, the JVM and/or many machines to their memory limits.

If you do not not need to do more than making sure that something gets downloaded, then you can “offload” the download work to an outside process, f.ex. to curl or wget:

A few things to note:

  • I used the “BeanShell Sampler” module, since both the “BeanShell Pre-” and the “PostProcessors” are only executed when there’s something to process after another sampler produced some data.

  • As you can see, we are passing a JMeter variable to the script

  • Aparently JMeter or the BeanShell do not care much about the output of the exec’ed script, thus you won’t see anything inside the “Response data” in the “View Results Tree Sampler”. I don’t know whether that behaveour is a missing feature, a bug or for some unknown reason intended as such.

You can download this JMeter test from here

Tomáš Pospíšek, 30.12.2010

Offline editing plugin for QGIS

For data collection, it is a common situation to work with a laptop or a phone offline in the field. Upon returning to the network, the changes need to be synchronized with the master data source, e.g. a PostGIS database. If several persons are working simultaneously on the same datasets, it is difficult to merge the edits by hand, even if people don’t change the same features.

Therefore, Mathias Walker implemented an offline plugin for QGIS. This plugin automates the synchronisation by copying the content of a datasource (usually PostGIS or WFS-T) to a spatialite database and storing the offline edits to dedicated tables. After being connected to the network again, it is possible to apply the offline edits to the master dataset.

To give the plugin a try, unpack the sources, apply the patch ‘qgissvn.diff’ to a current svn version of QGIS. Then copy the offlineediting folder to $PREFIX/src/plugins and recompile QGIS.

The usage of the plugin is straightforward:

  • Open some vector layers, e.g. from a PostGIS or WFS-T datasource
  • Save the project
  • Press the ‘Convert to offline project’ button and select the layers to save. The content of the layers is saved to spatialite tables.
  • Edit the layers offline
  • After being connected again, upload the changes with the ‘Synchronize’ button

Screenshot

Presumably, the offline editing plugin will be part of the next QGIS version (1.6)

New label tools in QGIS

In cartography, it is a frequent operation to set labels to fixed positions, together with the position of the fix point (left/middle/right, Top, Half, Bottom) that is kept constant in case of font change, rotation or zoom. Therefore, three new editing tools to manipulate text labels are now in the QGIS developer version:

  1. the move label tool drags text labels to a new position

  1. the rotate label tool is for interactive rotation of labels
  2. the label property tools opens a dialog that lets the user manipulate the data defined properties of a label (and also the text of the label attribute)

All three tools work on the new labeling engine and data defined labeling needs to be enabled for the layer (e.g. x coordinate attribute / y coordinate attribute for the move tool, rotation for the rotate tool). Additionally, the layer needs to be in edit mode. The new tools are well suited to mix fixed label positions and automated label opsitioning in the same or among several layers. If the x- or y attribute value is NULL, the position is set automatically by the pal library (http://pal.heig-vd.ch/). As soon as a position is manipulated by the move label tool, the position is written into the attribute field and the label position for this feature is fixed. So if a layer does not yet have attribute fields for x, y, you could create two new fields of type double (using the buttons in the vector properties dialog or in the attribute table). Initially, all values will be ‘NULL’ and all the label positions set automatically.

There are further plans to improve the user interface. It could be handy to have the properties dialog always open (non-modal), which would allow faster edits of a large number of labels. And a live text rotation preview is planned too. And yes, if someone likes making real icons, this would be highly appreciated (my graphical skills are somewhat limited…).

Finally I’d like to thank the city of Thun (Switzerland) for funding these tools and sharing it with the rest of the FOSSGIS world.

Testing UMN Mapfiles with QGIS

The Sunday night session of the QGIS hackfest resulted in a new release of the Mapfile Tools plugin.

This QGIS plugin allows you to display an UMN Mapserver mapfile in QGIS without running a Mapserver instance. It depends only on Mapscript (apt-get install python-mapscript on Debian/Ubuntu) and allows you to zoom and pan on the mapfile layer.

In release 0.6, an output window has been added, which shows error messages and detailed layer information. This makes it a convenient tool to test your mapfiles.

Back to Top

Sustaining Members