Related Plugins and Tags

QGIS Planet

USB Recovery Script

What do you do when you are managing a remote server and you need to make some critical changes (like to the networking configs) and you feel uncomfortable about the possibility of losing access to the server and never getting it back? This was the situation we were in today. The server is a little esoteric - its a headless box and even in the server center the engineers don't have any way to log in interactively at the server itself. Luckily the server is running Debian linux and has a usb port so help is at hand via bash!

I wrote this little script which is designed to be run from a cron job, for example every minute.


# This script is to rescue the system from usb while
# testing migration to the new vpn.

# It will mount the last partition of any inserted usb,
# cd to the mount point and try to run a script
# called ''
# After the script is run it will be renamed to
# rescue.ok
# You should set this script to run as a cron job
# at minute intervals.
# e.g. # m h  dom mon dow   command
#      * * * * * /root/
# Note we ignore partitions on devices sda - sdd as they are internal disks
LASTPARTITION=$(cat /proc/partitions  | awk '{print $4}' | grep -v 'sd[a-d]' \
| grep -v name | grep -v '^$' |sort | tail -1)
if [ $LASTPARTITION != "" ]
  if [ ! -b /dev/$LASTPARTITION ]
    echo "Error /dev/$LASTPARTITION is not a block device"
    echo "OK /dev/$LASTPARTITION is a block device"
  echo "Device found creating mount point"
  if [ ! -d "$MOUNTPOINT" ]
    mkdir $MOUNTPOINT
  echo "Mounting...."
  echo "Checking if rescue script exists"
  # Test the rescue script exists(-e) and is not 0 length (-s)
  if [ -e $SCRIPTPATH -a -s $SCRIPTPATH ]
    echo "Making $SCRIPTPATH executable"
    chmod +x $SCRIPTPATH
    echo "Running script"
    echo "Disabling script"
    echo "No Rescue script found"
  echo "Unmounting.."
  cd /
  umount $MOUNTPOINT
  echo "No rescue device found"
echo "done"

If you place the script in /root/ and add a cron job as outlined in the comments, it will poll for devices regularly, mount the last partition available.

If it finds a script on that partition labelled, it will run it then rename the script to rescue.ok and write any stderror and stdout logs to rescue.log on the partition. The script could perhaps be improved by adding a lock file so that it does not get run again if it is already running (if it takes longer than a minute to run for example), but it's a good starting point for a system rescue if things go wrong. Now the engineer on site can simply pop in his usb stick and any recovery commands will be run from it.

Creating and applying patches with Git

During the recent QGIS hackfest in Poland, we spent some time discussing the use of GIT and I spent a bit of time to learn the basics of using GIT. One thing about GIT that is different for users like myself coming from an SVN background is the way that creating and applying patches is done. Typically under svn I would do something like this to create a patch:

svn diff > foo.diff

And then to apply that same patch I would do:

patch -p0 < foo.diff

Which is all quite simple. Git's process is a little different. Here is my workflow for creating a patch:

  • Create a branch (GIT seems to encourage you to work in branches and to branch often)
  • Checkout your branch
  • Work and change things in your branch
  • Commit your work to your branch
  • Generate a patch as a diff between your branch and master
  • Submit your patch
  • Apply your patch

So here is a simple session that does the above:

git branch patch-testing
git checkout patch-testing
( do some work in that branch now)
git commit -m "Important changes" -a
git format-patch master

After you are done with that,  Git will have created a nice little diff for you. To apply the patch to another checkout do:

git apply foo.path

Then commit and push your changes to the origin repository.

A workflow for creating beautiful relief shaded dems using GDAL

Sometimes I create hillshades using the QGIS hillshade plugin and then overlay the original DEM over it. I set the DEM to have a false colour pallette and set it to be semi-transparent to produce something like this:

Typical usage of a hillshade with false colour overlay

That is all well and good but a bit impractical. It would be much nice to have the colour pallette permanetly assigned to the hillshade. Also I want to be able to clip and mask the resulting raster to the boundary specified in a shapefile. Fortunately, GDAL provides all the tools you need to make this happen. There are already some nice articles (like this one) that describe parts of this workflow but I am writing this because I wanted to note the additional steps that I took to make this work well for me.

Before you begin

Before you begin you should have:

  1. a raster containing digital elevation data (DEM) - in my case its called 'alt.tif'
  2. a vector layer (typically a shapefile) containing the area of interest for your final product - in my case its called 'tanzania.shp'

Create the hillshade image

The first thing we need to do is generate a hillshade. There is a nice python plugin for doing this in QGIS, you can do it in GRASS using the QGIS-GRASS plugin. But in this article I'm going for an all-GDAL approach so we will be using the handy **gdaldem** command line application.  I won't be explaining the parameters I have used here as they are well explained in the gdaldem manual pages.

So to create our hillshade we do something like this:

gdaldem hillshade alt.tif shade.tif -z 5 -s 111120 -az 90

Which will produce something like this: colorbrewer would be a good place to start if you want to learn more.  Another favourite site of mine is and for this tutorial I decided to use a pallette from there to see how it would turn out.

Once you have selected a suitable colour pallette (around 5 or 6 colour classes should do it), the next thing you need to do is get some information about the range of values contained in your DEM. Once again you can easily point and click your way to this in QGIS, but here is the way to get it in gdal from the command line:

gdalinfo -mm alt.tif

The resulting output includes the computed min/max for Band 1 - which is what we are after:

Band 1 Block=1334x3 Type=UInt16, ColorInterp=Gray
Computed Min/Max=1.000,5768.000 NoData Value=65535

Ok so our minimum height is 1m and maximum is 5768m - Tanzania is the home of Kilimanjaro after all! So lets split that range up into 5 classes to match the 'Landcarpet Europe' colourlover pallette I selected. I set nearly white as an additional colour for the highest altitude range.

65535 255 255 255 5800 254 254 254 3000 121 117 10 1500 151 106 47 800 127 166 122 500 213 213 149 1 218 179 122

The value in the first column is the base of the scale range. The subsequent values are RGB triplets for that range. I saved this as a text file called 'ramp.txt' in the same directory as my 'alt.tiff' dataset. You will notice that I made the value ranges closer together at lower altitudes and wider appart at higher altitudes. You may want to experiment a little to get pleasing results - especially if you have a relatively small number of high lying terrain pixels and the rest confined to lower lying areas.

Also note that I assigned the highest terrain 'nearly white' so that I could reserve white (RGB: 255 255 255) for the nodata value (65535) in this dataset. We will use the fact that white is only used for nodata to our advantage when we do a clip further on in these instructions.

Ok now we can use gdaldem to generate the terrain map:

gdaldem color-relief alt.tif ramp.txt relief.tif

This is what my relief map looked like:

The terrain colour map I produced

Don't worry about the fact that it does not resemble the colour pallette you have chosen - it will do in the next step!

Merging the two rasters

The next step is to combine the two products. I used Frank Warmerdam's handy script for this purpose.

./ relief.tif shade.tif colour_shade.tif

Which produced this:

The result of merging my hillshade and my terrain colour map

You may have noticed that it is only at this point that the colours of our product resemble the original pallette we used.

One little gotcha with the script is that it throws away our nodata values, so what was sea before (and nodata in my original alt.tif dataset) is now white (RGB: 255 255 255).

Clipping and masking

You may have everything you need from the above steps. Alternatively you can also clip and mask the dataset using a shapefile.

gdalwarp -co compress=deflate -dstnodata 255 -cutline Tanzania.shp
colour_shade.tif colour_shade_clipped.tif

My final image is now a compressed tif with nodata outside of the country of Tanzania and looks like this:

Final result: A false coloured elevation map for Tanzania

A final note

One of the things I love about the command line is the repeatable and reusable workflows it allows for. I can distill everything in this blog post into a sequence of commands and replay them any time. If you are still stuck doing things in a GUI only way, give BASH a try - you can really do powerful geospatial data processing with it!

Ubuntu Jaunty Post-Installation for FOSSGIS

It's quick (overlooking slow internet access) and easy to set up Ubuntu Jaunty 64 bit for use with the FOSSGIS stack. Here are the packages I install once the base system is set up (including a few non-fossgis ones that are nice to have).

sudo apt-get install build-essential vim grass libqt4-core \
libqt4-debug libqt4-dev libqt4-gui libqt4-qt3support libqt4-sql \
lsb-qt4 qt4-designer   qt4-dev-tools qt4-doc qt4-qtconfig uim-qt gcc \
libapt-pkg-perl resolvconf   gdal-bin libgdal1-dev libgeos-dev proj \
libgdal-doc libhdf4g-dev libhdf4g-run python-dev  python-gdal libgsl0-dev \
g++ libjasper-dev libtiff4-dev subversion libsqlite3-dev sqlite3 ccache \
make libpq-dev flex bison cmake txt2tags python-qt4 python-qt4-dev \
python-sip4 sip4 python-sip4-dev postgresql-8.3-postgis vim ctags \
vim-scripts python-django python-psycopg2 vim apache2 \
sun-java6-fonts python-geoip imagemagick grass-dev ia32-libs \
graphviz txt2tags texlive-fonts-recommended \
texlive-fonts-recommended-doc texlive-latex-base texlive-latex-base-doc \
tipa tex-common texlive-base texlive-base-bin texlive-base-bin-doc \
texlive-common texlive-doc-base gv latex2html texlive-latex-extra \
libqwt5-qt4-dev pyqt4-dev-tools drivel lightning-extension thunderbird-gnome-support \
gwibber python-django-evolution python-django-dmigrations python-django-djblets \
python-django-debug-toolbar python-django-registration python-django-lint \
python-django-tagging awn-applets-c-extras awn-applets-python-extras\
gnome-do tomcat6-admin tomcat6 enigmail

You also need to get rid of this since it doesnt play nice with QGIS:

sudo apt-get remove uim-qt3

Note: the ia32-libs is needed to run google earth under ubuntu 64

Back to Top

Sustaining Members