Sean Gillies has managed to get me addicted to using generators, itertools, map, filter, and a generally more declarative style of programming.

@madmanwoo \o/

— Sean Gillies (@sgillies) July 17, 2013

What has this got to do with anything? Well let me explain. A little project I have recently started working on in my free time is a mini query engine for pyqgis to make fetching features easier. The goals are simple.

• Use a query style syntax to say what I need, not how to get it. The basics of declarative programming.
• Use generators to save on memory unless the user asks.

Before we go any further, just note that this is just a hobby project of mine. Not feature complete nor bug free but it's evolving.

The code can be found on github or its homepage.

What good is a mini query engine? Lets try with some examples.

Get all the features that match the condition `postcode > 6164 OR postcode < 6167'. We will use QgsExpression because that provides a nice where clause type base for us to use.

exp = QgsExpression("postcode > 6164 OR postcode < 6167")
fields = layer.pendingFields()
exp.prepare(fields)
for feature in layer.getFeatures():
    if exp.evaluate(feature):
        print "Pass"

at best you can reduce it to this:

exp = QgsExpression("postcode > 6164 OR postcode < 6167")
fields = layer.pendingFields()
exp.prepare(fields)
features = filter(exp.evaluate, layer.getFeatures())

Not too bad. But still. it's a bit of a pain because you have to care about QgsExpression and looping all the features to check.

How about something like this?

q = query(layer).where("postcode > 6164 OR postcode < 6167")
for feature in q():
    print feature

Define a query on layer and return only the features that match the .where() condition. Sounds pretty descriptive to me.

Nothing is executed until you call the query object itself q() and the return result is a generator so it will only serve up records as we ask. The return type is a Python dict because they are simple, fast, and work well for a return type.

>>> from query import query
>>> q = query(layer).where("postcode > 6164 OR postcode < 6167")
>>> type(q)
<class 'query.Query'>
>>> results = q()
>>> type(results)
<generator object <genexpr> at 0x1A8DDD78>
>>> results.next()
{
u'old': u'http://www.seabreeze.com.au|mylink', 
u'pin_string': u'286633', 
'qgs_geometry': <qgis.core.QgsGeometry object at 0x1A8DF588>, 
u'bool': u'F', 
u'location': None, 
u'lot': u'LOT 79', 
'qgs_feature': <qgis.core.QgsFeature object at 0x1A8DF0C0>
}

Select support

What is the point of a query if you can't ask it to give you just what you need?

>>> q = (query(layer).where("postcode > 6164 OR postcode < 6167")
                     .select('assessment','address', 'lot',
                              geom = lambda f: f.geometry().buffer(20,3),
                              mylot = lambda f: int(f['house_numb']) * 100))
>>> q().next()
{
'geom': <qgis.core.QgsGeometry object at 0x1A8E64B0>, 
'assessment': u'4309719', 
'mylot': 7900, 
'lot': u'LOT 79', 
'address': u'BIRCHLEY RD'
}

You can use "colname" or any Python callable in the select statement. Using keyword arguments is the same as "Col" AS MyName in SQL.

On the TODO list

I would really like to have some kind of index support in the future to make queries run faster. I had basic index support working using a dict however it was nothing smart and was hard coded. QgsExpression does provide support to walk the generated tree see what the expression includes. In "theory" it would involve walking the expression tree and calculating what should be used for index lookup.

Join support would also be nice. Attribute and spatial.

Helpers welcome

Like I said at the start, this is just a hobby project and it's still only in early days however I am always happy for help if you have ideas, or know how to improve something. If you find it handy that would be cool to know too.

Installing

• Just download the code from here.
• Add the files to your Python project
• from query import query

Sean Gillies has managed to get me addicted to using generators, itertools, map, filter, and a generally more declarative style of programming.

@madmanwoo \o/

— Sean Gillies (@sgillies) July 17, 2013

What has this got to do with anything? Well let me explain. A little project I have recently started working on in my free time is a mini query engine for pyqgis to make fetching features easier. The goals are simple.

• Use a query style syntax to say what I need, not how to get it. The basics of declarative programming.
• Use generators to save on memory unless the user asks.

Before we go any further, just note that this is just a hobby project of mine. Not feature complete nor bug free but it's evolving.

The code can be found on github or its homepage.

What good is a mini query engine? Lets try with some examples.

Get all the features that match the condition `postcode > 6164 OR postcode < 6167'. We will use QgsExpression because that provides a nice where clause type base for us to use.

exp = QgsExpression("postcode > 6164 OR postcode < 6167")
fields = layer.pendingFields()
exp.prepare(fields)
for feature in layer.getFeatures():
    if exp.evaluate(feature):
        print "Pass"

at best you can reduce it to this:

exp = QgsExpression("postcode > 6164 OR postcode < 6167")
fields = layer.pendingFields()
exp.prepare(fields)
features = filter(exp.evaluate, layer.getFeatures())

Not too bad. But still. it's a bit of a pain because you have to care about QgsExpression and looping all the features to check.

How about something like this?

q = query(layer).where("postcode > 6164 OR postcode < 6167")
for feature in q():
    print feature

Define a query on layer and return only the features that match the .where() condition. Sounds pretty descriptive to me.

Nothing is executed until you call the query object itself q() and the return result is a generator so it will only serve up records as we ask. The return type is a Python dict because they are simple, fast, and work well for a return type.

>>> from query import query
>>> q = query(layer).where("postcode > 6164 OR postcode < 6167")
>>> type(q)
<class 'query.Query'>
>>> results = q()
>>> type(results)
<generator object <genexpr> at 0x1A8DDD78>
>>> results.next()
{
u'old': u'http://www.seabreeze.com.au|mylink', 
u'pin_string': u'286633', 
'qgs_geometry': <qgis.core.QgsGeometry object at 0x1A8DF588>, 
u'bool': u'F', 
u'location': None, 
u'lot': u'LOT 79', 
'qgs_feature': <qgis.core.QgsFeature object at 0x1A8DF0C0>
}

Select support

What is the point of a query if you can't ask it to give you just what you need?

>>> q = (query(layer).where("postcode > 6164 OR postcode < 6167")
                     .select('assessment','address', 'lot',
                              geom = lambda f: f.geometry().buffer(20,3),
                              mylot = lambda f: int(f['house_numb']) * 100))
>>> q().next()
{
'geom': <qgis.core.QgsGeometry object at 0x1A8E64B0>, 
'assessment': u'4309719', 
'mylot': 7900, 
'lot': u'LOT 79', 
'address': u'BIRCHLEY RD'
}

You can use "colname" or any Python callable in the select statement. Using keyword arguments is the same as "Col" AS MyName in SQL.

On the TODO list

I would really like to have some kind of index support in the future to make queries run faster. I had basic index support working using a dict however it was nothing smart and was hard coded. QgsExpression does provide support to walk the generated tree see what the expression includes. In "theory" it would involve walking the expression tree and calculating what should be used for index lookup.

Join support would also be nice. Attribute and spatial.

Helpers welcome

Like I said at the start, this is just a hobby project and it's still only in early days however I am always happy for help if you have ideas, or know how to improve something. If you find it handy that would be cool to know too.

Installing

• Just download the code from here.
• Add the files to your Python project
• from query import query

This post was inspired by a post written by Xaquín G.V, some extra credit must also go to Mathieu for helping me with the expression.

One of the new great features in the upcoming QGIS 2.0 is the ability to use data defined symbology and labelling. It might look like nothing on the surface but it can really open up the possibility of some great looking maps.

Lets start with the example Xaquín has used by classifying by the unemployment rate. I have even created a pre-joined dataset for you if you want to follow along, grab it here.

So first lets make a custom colour ramp just for this:

call it "usa" and classify the data

Ok not bad. But you will always have those hard edges on the colours. What if we could get the exact colour ramp value for a given value? Well you can.

We need to make a new column for the colour value for each feature. Open up the field calculator and add a new column with the following expression

ramp_color('usa', scale_linear( "unemployed_by_county_xgv_Rate",0,18,0,1 ))

What are the ramp_colur and scale_linear functions? ramp_colur is a new function that allows you to get a RGBA value for a position inside the ramp. So if you want the RGBA value at the 75% mark you can use ramp_color('usa', 0.75).

Cool but to make it dynamic we need to bring in the value from the "Rate" column, however that goes up higher then 1. How do we get in a range from 0..1? Well we can scale it back down using scale_linear.

scale_linear(<column>,<min in value>,<max in value>,<min out value>,<max out value>) 

Having said all that here is our result:

To use this new colour column we need to use data defined symbols

Note: I have selected Single Symbol

And the result

This isn't going to suit every map but when something is scale based this method works well.

Bumping it up a notch

Higher unemployment rates plus high lobor force really should get more weight. One way to handle this is by using Alpha-By-Value like Xaquín said in his post. Well to do that in QGIS we need to change the alpha value of what is returned from color_ramp. Time for a bit of regex (don't worry in 2.1 I will add a colour_ramp_rgb function with no aplha part)

First lets get just the aplha value part by scaling the lobor force coloum. Add a new column using the following expression

toint(scale_linear("unemployed_by_county_xgv_Labor_Force", 0, 100000, 0, 255))

Now make a new column that replaces the alpha part of the colour column

regexp_replace( "colour", ',[^,]*$' , format(',%1',"alpha" ))

Then we can use data defined symbols on that new column

Nifty!

Now you will have to play with the min and max values for the scaling functions and also the colours in the ramp but you get the idea.

Bonus

You can also use the same colour column on the labels to match the polygon and even increase the label size based on the rate column

So that is your Thursday lesson on how to create a Alpha by value map in the upcoming QGIS 2.0.

Extra stuff

If you want to avoid creating all those extra columns or even do it on the fly you can use this single expression

regexp_replace( ramp_color('usa', scale_linear( "unemployed_by_county_xgv_Rate",0,15,0,1)),',[^,]*$',','|| toint(scale_linear("unemployed_by_county_xgv_Labor_Force",0,100000,0,255)))

Using the single expression block can be handy if you want to get your upper and lower limits right.

This post was inspired by a post written by Xaquín G.V, some extra credit must also go to Mathieu for helping me with the expression.

One of the new great features in the upcoming QGIS 2.0 is the ability to use data defined symbology and labelling. It might look like nothing on the surface but it can really open up the possibility of some great looking maps.

Lets start with the example Xaquín has used by classifying by the unemployment rate. I have even created a pre-joined dataset for you if you want to follow along, grab it here.

So first lets make a custom colour ramp just for this:

call it "usa" and classify the data

Ok not bad. But you will always have those hard edges on the colours. What if we could get the exact colour ramp value for a given value? Well you can.

We need to make a new column for the colour value for each feature. Open up the field calculator and add a new column with the following expression

ramp_color('usa', scale_linear( "unemployed_by_county_xgv_Rate",0,18,0,1 ))

What are the ramp_colur and scale_linear functions? ramp_colur is a new function that allows you to get a RGBA value for a position inside the ramp. So if you want the RGBA value at the 75% mark you can use ramp_color('usa', 0.75).

Cool but to make it dynamic we need to bring in the value from the "Rate" column, however that goes up higher then 1. How do we get in a range from 0..1? Well we can scale it back down using scale_linear.

scale_linear(<column>,<min in value>,<max in value>,<min out value>,<max out value>) 

Having said all that here is our result:

To use this new colour column we need to use data defined symbols

Note: I have selected Single Symbol

And the result

This isn't going to suit every map but when something is scale based this method works well.

Bumping it up a notch

Higher unemployment rates plus high lobor force really should get more weight. One way to handle this is by using Alpha-By-Value like Xaquín said in his post. Well to do that in QGIS we need to change the alpha value of what is returned from color_ramp. Time for a bit of regex (don't worry in 2.1 I will add a colour_ramp_rgb function with no aplha part)

First lets get just the aplha value part by scaling the lobor force coloum. Add a new column using the following expression

toint(scale_linear("unemployed_by_county_xgv_Labor_Force", 0, 100000, 0, 255))

Now make a new column that replaces the alpha part of the colour column

regexp_replace( "colour", ',[^,]*$' , format(',%1',"alpha" ))

Then we can use data defined symbols on that new column

Nifty!

Now you will have to play with the min and max values for the scaling functions and also the colours in the ramp but you get the idea.

Bonus

You can also use the same colour column on the labels to match the polygon and even increase the label size based on the rate column

So that is your Thursday lesson on how to create a Alpha by value map in the upcoming QGIS 2.0.

Extra stuff

If you want to avoid creating all those extra columns or even do it on the fly you can use this single expression

regexp_replace( ramp_color('usa', scale_linear( "unemployed_by_county_xgv_Rate",0,15,0,1)),',[^,]*$',','|| toint(scale_linear("unemployed_by_county_xgv_Labor_Force",0,100000,0,255)))

Using the single expression block can be handy if you want to get your upper and lower limits right.

This post was inspired by a post written by Xaquín G.V, some extra credit must also go to Mathieu for helping me with the expression.

One of the new great features in the upcoming QGIS 2.0 is the ability to use data defined symbology and labelling. It might look like nothing on the surface but it can really open up the possibility of some great looking maps.

Lets start with the example Xaquín has used by classifying by the unemployment rate. I have even created a pre-joined dataset for you if you want to follow along, grab it here.

So first lets make a custom colour ramp just for this:

call it "usa" and classify the data

Ok not bad. But you will always have those hard edges on the colours. What if we could get the exact colour ramp value for a given value? Well you can.

We need to make a new column for the colour value for each feature. Open up the field calculator and add a new column with the following expression

ramp_color('usa', scale_linear( "unemployed_by_county_xgv_Rate",0,18,0,1 ))

What are the ramp_colur and scale_linear functions? ramp_colur is a new function that allows you to get a RGBA value for a position inside the ramp. So if you want the RGBA value at the 75% mark you can use ramp_color('usa', 0.75).

Cool but to make it dynamic we need to bring in the value from the "Rate" column, however that goes up higher then 1. How do we get in a range from 0..1? Well we can scale it back down using scale_linear.

scale_linear(<column>,<min in value>,<max in value>,<min out value>,<max out value>)

Having said all that here is our result:

To use this new colour column we need to use data defined symbols

Note: I have selected Single Symbol

And the result

This isn't going to suit every map but when something is scale based this method works well.

Bumping it up a notch

Higher unemployment rates plus high lobor force really should get more weight. One way to handle this is by using Alpha-By-Value like Xaquín said in his post. Well to do that in QGIS we need to change the alpha value of what is returned from color_ramp. Time for a bit of regex (don't worry in 2.1 I will add a colourramprgb function with no aplha part)

First lets get just the aplha value part by scaling the lobor force coloum. Add a new column using the following expression

toint(scale_linear("unemployed_by_county_xgv_Labor_Force", 0, 100000, 0, 255))

Now make a new column that replaces the alpha part of the colour column

regexp_replace( "colour", ',[^,]*$' , format(',%1',"alpha" ))

Then we can use data defined symbols on that new column

Nifty!

Now you will have to play with the min and max values for the scaling functions and also the colours in the ramp but you get the idea.

Bonus

You can also use the same colour column on the labels to match the polygon and even increase the label size based on the rate column

So that is your Thursday lesson on how to create a Alpha by value map in the upcoming QGIS 2.0.

Extra stuff

If you want to avoid creating all those extra columns or even do it on the fly you can use this single expression

regexp_replace( ramp_color('usa', scale_linear( "unemployed_by_county_xgv_Rate",0,15,0,1)),',[^,]*$',','|| toint(scale_linear("unemployed_by_county_xgv_Labor_Force",0,100000,0,255)))

Using the single expression block can be handy if you want to get your upper and lower limits right.

What is __geo_interface__ anyway? If you have never seen __geo_interface__ it was defined by Sean Gillies at https://gist.github.com/sgillies/2217756. A cool lightweight interface that describes a single spatial object using a GeoJSON like structure, in fact parts of it are just normal GeoJSON.

Since QGIS in feature freeze at the moment I can't add this to the main code base just yet. Awwww sad face :(

but wait!

Thanks to good ol' Python we can just monkey patch it right on.

def mapping_feature(feature):
    geom = feature.geometry()
    properties = {}
    fields = [field.name() for field in feature.fields()]
    properties = dict(zip(fields, feature.attributes()))
    return { 'type' : 'Feature',
             'properties' : properties,
             'geometry' : geom.__geo_interface__}

def mapping_geometry(geometry):
    geo = geometry.exportToGeoJSON()
    # We have to use eval because exportToGeoJSON() gives us
    # back a string that looks like a dictionary. 
    return eval(geo)

QgsFeature.__geo_interface__ = property(lambda self: mapping_feature(self))
QgsGeometry.__geo_interface__ = property(lambda self: mapping_geometry(self))

And that's that. Easy hey. Really got to love dynamic languages.

The __geo_interface__ property now exists on any instance of QgsGeometry or QgsFeature. Lets test that theory.

>>> import pprint
>>> layer = iface.activeLayer()
>>> feature = layer.selectedFeatures()[0]
>>> feature.__geo_interface__
>>> pprint.pprint(f.__geo_interface__)
{'geometry': {'coordinates': [[[385039.90567724, 6449154.61878853],
                               [385059.01135993, 6449154.80874077],
                               [385059.41538719, 6449114.58680192],
                               [385040.30145863, 6449114.38685169],
                               [385040.16953133, 6449127.46423076],
                               [385039.90567724, 6449154.61878853]]],
              'type': 'Polygon'},
 'properties': {u'address': u'HYNES WY',
                u'assessment': u'2204315',
                u'bool': u'F',
                u'dola_pin': 283678,
                u'field18': 2920,
                u'field19': 0.0,
                u'house_numb': u'3',
                u'location': None,
                u'lot': u'LOT 107',
                u'new': None,
                u'old': u'http://www.seabreeze.com.au|mylink',
                u'paid_in_fu': u'F',
                u'pin_string': u'283678',
                u'postcode': u'6163',
                u'reserve': None,
                u'sample_dat': u'2003-05-15',
                u'subdivided': None,
                u'suburb': u'HAMILTON HILL',
                u'suburb_wit': u'HAMILTON HILL',
                u'type': u'H',
                u'v_auto_dat': None,
                u'v_auto_use': None,
                u'v_boolean': None,
                u'v_datetime': None,
                u'v_decimal': None,
                u'v_int': None,
                u'v_numeric': None,
                u'v_varcha_1': None,
                u'v_varchar': None,
                u'v_varchar_': None},
 'type': 'Feature'}
>>> feature.geometry().__geo_interface__
{'type': 'Point', 'coordinates': [388197.74503284, 6450504.16670842]}

Excellent!

What is __geo_interface__ anyway? If you have never seen __geo_interface__ it was defined by Sean Gillies at https://gist.github.com/sgillies/2217756. A cool lightweight interface that describes a single spatial object using a GeoJSON like structure, in fact parts of it are just normal GeoJSON.

Since QGIS in feature freeze at the moment I can't add this to the main code base just yet. Awwww sad face :(

but wait!

Thanks to good ol' Python we can just monkey patch it right on.

def mapping_feature(feature):
    geom = feature.geometry()
    properties = {}
    fields = [field.name() for field in feature.fields()]
    properties = dict(zip(fields, feature.attributes()))
    return { 'type' : 'Feature',
             'properties' : properties,
             'geometry' : geom.__geo_interface__}

def mapping_geometry(geometry):
    geo = geometry.exportToGeoJSON()
    # We have to use eval because exportToGeoJSON() gives us
    # back a string that looks like a dictionary. 
    return eval(geo)

QgsFeature.__geo_interface__ = property(lambda self: mapping_feature(self))
QgsGeometry.__geo_interface__ = property(lambda self: mapping_geometry(self))

And that's that. Easy hey. Really got to love dynamic languages.

The __geo_interface__ property now exists on any instance of QgsGeometry or QgsFeature. Lets test that theory.

>>> import pprint
>>> layer = iface.activeLayer()
>>> feature = layer.selectedFeatures()[0]
>>> feature.__geo_interface__
>>> pprint.pprint(f.__geo_interface__)
{'geometry': {'coordinates': [[[385039.90567724, 6449154.61878853],
                               [385059.01135993, 6449154.80874077],
                               [385059.41538719, 6449114.58680192],
                               [385040.30145863, 6449114.38685169],
                               [385040.16953133, 6449127.46423076],
                               [385039.90567724, 6449154.61878853]]],
              'type': 'Polygon'},
 'properties': {u'address': u'HYNES WY',
                u'assessment': u'2204315',
                u'bool': u'F',
                u'dola_pin': 283678,
                u'field18': 2920,
                u'field19': 0.0,
                u'house_numb': u'3',
                u'location': None,
                u'lot': u'LOT 107',
                u'new': None,
                u'old': u'http://www.seabreeze.com.au|mylink',
                u'paid_in_fu': u'F',
                u'pin_string': u'283678',
                u'postcode': u'6163',
                u'reserve': None,
                u'sample_dat': u'2003-05-15',
                u'subdivided': None,
                u'suburb': u'HAMILTON HILL',
                u'suburb_wit': u'HAMILTON HILL',
                u'type': u'H',
                u'v_auto_dat': None,
                u'v_auto_use': None,
                u'v_boolean': None,
                u'v_datetime': None,
                u'v_decimal': None,
                u'v_int': None,
                u'v_numeric': None,
                u'v_varcha_1': None,
                u'v_varchar': None,
                u'v_varchar_': None},
 'type': 'Feature'}
>>> feature.geometry().__geo_interface__
{'type': 'Point', 'coordinates': [388197.74503284, 6450504.16670842]}

Excellent!

What is __geo_interface__ anyway? If you have never seen __geo_interface__ it was defined by Sean Gillies at https://gist.github.com/sgillies/2217756. A cool lightweight interface that describes a single spatial object using a GeoJSON like structure, in fact parts of it are just normal GeoJSON.

Since QGIS in feature freeze at the moment I can't add this to the main code base just yet. Awwww sad face :(

but wait!

Thanks to good ol' Python we can just monkey patch it right on.

def mapping_feature(feature):
    geom = feature.geometry()
    properties = {}
    fields = [field.name() for field in feature.fields()]
    properties = dict(zip(fields, feature.attributes()))
    return { 'type' : 'Feature',
             'properties' : properties,
             'geometry' : geom.__geo_interface__}

def mapping_geometry(geometry):
    geo = geometry.exportToGeoJSON()
    # We have to use eval because exportToGeoJSON() gives us
    # back a string that looks like a dictionary. 
    return eval(geo)

QgsFeature.__geo_interface__ = property(lambda self: mapping_feature(self))
QgsGeometry.__geo_interface__ = property(lambda self: mapping_geometry(self))

And that's that. Easy hey. Really got to love dynamic languages.

The __geo_interface__ property now exists on any instance of QgsGeometry or QgsFeature. Lets test that theory.

>>> import pprint
>>> layer = iface.activeLayer()
>>> feature = layer.selectedFeatures()[0]
>>> feature.__geo_interface__
>>> pprint.pprint(f.__geo_interface__)
{'geometry': {'coordinates': [[[385039.90567724, 6449154.61878853],
                               [385059.01135993, 6449154.80874077],
                               [385059.41538719, 6449114.58680192],
                               [385040.30145863, 6449114.38685169],
                               [385040.16953133, 6449127.46423076],
                               [385039.90567724, 6449154.61878853]]],
              'type': 'Polygon'},
 'properties': {u'address': u'HYNES WY',
                u'assessment': u'2204315',
                u'bool': u'F',
                u'dola_pin': 283678,
                u'field18': 2920,
                u'field19': 0.0,
                u'house_numb': u'3',
                u'location': None,
                u'lot': u'LOT 107',
                u'new': None,
                u'old': u'http://www.seabreeze.com.au|mylink',
                u'paid_in_fu': u'F',
                u'pin_string': u'283678',
                u'postcode': u'6163',
                u'reserve': None,
                u'sample_dat': u'2003-05-15',
                u'subdivided': None,
                u'suburb': u'HAMILTON HILL',
                u'suburb_wit': u'HAMILTON HILL',
                u'type': u'H',
                u'v_auto_dat': None,
                u'v_auto_use': None,
                u'v_boolean': None,
                u'v_datetime': None,
                u'v_decimal': None,
                u'v_int': None,
                u'v_numeric': None,
                u'v_varcha_1': None,
                u'v_varchar': None,
                u'v_varchar_': None},
 'type': 'Feature'}
>>> feature.geometry().__geo_interface__
{'type': 'Point', 'coordinates': [388197.74503284, 6450504.16670842]}

Excellent!

QGIS 2.0 has under gone some radical changes for its upcoming release. Some of these major changes relate to the API, both C++ and Python. Something that always comes up when trying to improve an API is the chance that you might have to remove the old way of doing something so that things are cleaner and improved for the future. These changes will always break existing working code and can be a burden for a little while until everything is adjusted. A little pain for a lot of gain is the way I like to think about it. I will also add that breaking API so radically isn't something that we, as developers, like to do or take lightly, but sometimes it has to be done.

This post is a summary of some of the new APIs, what has changed, and what it means to you as a user.

New Vector API (Iterator pattern)

We would like to introduce multithreaded rendering in the future and the old API for getting a set of features from the provider wasn't going to cut it in a multithreaded environment. In order to allow something that will be more threadable then the old select model we changed to a iterator pattern.

The old used to look like this:

layer.select()
for feature in layer:
    print feature

select() wouldn't return anything rather it would tell the provider of the layer to get ready to return some features when asked e.g the for loop. Want to thread that? Nope. Not going to happen.

The solution:

Create a method that returns an iterator that each thread can loop over on their own without changing the result of the other.

features = layer.getFeatures()
for feature in features:
    print feature

if you look at layer.getFeatures() you can see it returns a QgsFeatureIterator

>>> layer.getFeatures()
<qgis.core.QgsFeatureIterator object at 0x1A05F810>

QgsFeatureIterator has a next() method, just like a normal Python iterator, which you can call to get the next feature, it will just serve them up as you need them.

>>> features = layer.getFeatures()
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062CD8>
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062C48>
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062C00>
>>> features.close()

So in the future (you can't do this yet due to some other restrictions):

>>> features = layer.getFeatures()
>>> features2 = layer.getFeatures()
>>> features.next().id()
1L
>>> features.next().id()
2L
>>> features2.next().id()
1L

Nifty!

New Vector API (Fields)

Something else that changed was how attributes are accessed.

This is the old style:

>>> index = layer.fieldNameIndex("yourcolumn")
>>> feature.attributeMap()[index].toString()
"Hello World"

That is a big pain in the butt just to get a value from a field.

The new method now uses a list of attributes on the feature itself.

>>> feature[5]
"Hello World"
>>> feature["yourcolumn"]
"Hello World"

You can even access them like attributes.

>>> feature.yourcolumn
"Hello World"

Be careful with accessing fields via the attribute magic. If you have a id field feature.id will return the feature id method not the field named id. In fact any method with the same name as any QgsFeature methods will return the method rather then the field value. I like this magic but use it wisely.

SIP API v2

This is the most recent change that has caused all of the 1.8 plugins to no longer work in 2.0. A bit of background: PyQt4 and pyqgis use sip to define their Python API. There are two versions of SIP V1 and V2. Using SIP V1 PyQt, and therefor pyqgis, will take and return QVariant and QString objects in Python.

So:

>>> feature["yourcolumn"]
<PyQt4.QtCore.QVariant object at 0x026AD5E0>

Well that is fine but to get a string value you had to do this:

>>> feature["yourcolumn"].toString()
"Hello World"

but that isn't a normal Python string. It's a QString. QStrings can't be used in normal Python methods so you have to do str(feature["yourcolumn"].toString()). Gross!

With the change to SIP V2 QVariant and QString are now auto converted to native Python types without any extra work.

>>> type(f["Name"])
<type 'unicode'>

Good stuff.

If you are looking for a conversion guide check out: http://hub.qgis.org/wiki/quantum-gis/Python_plugin_API_changes_from_18_to_20

The QGIS plugin repository can house both 1.8 and 2.0 plugins under the same plugin name, and the plugin installer will only download the version compatible with the QGIS version you are using.

The SIP version update was a hard one to make. On one hand we could just leave it the way it was with backwards compatibility and all the messy str(variant.toString()) stuff; OR we can make the switch now while everyone is migrating to the new vector API and have a cleaner API for the future. I picked the later. Once we switch to Python 3 at some stage in the future SIP v2 is the default so it would have broken all the plugins at that point anyway.

I can tell you that I did lose some sleep over if I should push the SIP update or not, wondering what impact it could have on plugin authors and the project in general. In the end I stand by our decision to make the update in QGIS 2.0.

Hopefully these API changes won't cause to much grief for plugin authors and the API is now better to work with.

Fine. But why I do care as a user?

All this means that when QGIS 2.0 comes out you may be missing some plugins that you used to have in 1.8 until all the plugins are migrated. If you have the skills to help updating a plugin please let the plugin author know so you can make their life easier. If you have the ability to fund a plugins update that would be excellent.

QGIS 2.0 has under gone some radical changes for its upcoming release. Some of these major changes relate to the API, both C++ and Python. Something that always comes up when trying to improve an API is the chance that you might have to remove the old way of doing something so that things are cleaner and improved for the future. These changes will always break existing working code and can be a burden for a little while until everything is adjusted. A little pain for a lot of gain is the way I like to think about it. I will also add that breaking API so radically isn't something that we, as developers, like to do or take lightly, but sometimes it has to be done.

This post is a summary of some of the new APIs, what has changed, and what it means to you as a user.

New Vector API (Iterator pattern)

We would like to introduce multithreaded rendering in the future and the old API for getting a set of features from the provider wasn't going to cut it in a multithreaded environment. In order to allow something that will be more threadable then the old select model we changed to a iterator pattern.

The old used to look like this:

layer.select()
for feature in layer:
    print feature

select() wouldn't return anything rather it would tell the provider of the layer to get ready to return some features when asked e.g the for loop. Want to thread that? Nope. Not going to happen.

The solution:

Create a method that returns an iterator that each thread can loop over on their own without changing the result of the other.

features = layer.getFeatures()
for feature in features:
    print feature

if you look at layer.getFeatures() you can see it returns a QgsFeatureIterator

>>> layer.getFeatures()
<qgis.core.QgsFeatureIterator object at 0x1A05F810>

QgsFeatureIterator has a next() method, just like a normal Python iterator, which you can call to get the next feature, it will just serve them up as you need them.

>>> features = layer.getFeatures()
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062CD8>
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062C48>
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062C00>
>>> features.close()

So in the future (you can't do this yet due to some other restrictions):

>>> features = layer.getFeatures()
>>> features2 = layer.getFeatures()
>>> features.next().id()
1L
>>> features.next().id()
2L
>>> features2.next().id()
1L

Nifty!

New Vector API (Fields)

Something else that changed was how attributes are accessed.

This is the old style:

>>> index = layer.fieldNameIndex("yourcolumn")
>>> feature.attributeMap()[index].toString()
"Hello World"

That is a big pain in the butt just to get a value from a field.

The new method now uses a list of attributes on the feature itself.

>>> feature[5]
"Hello World"
>>> feature["yourcolumn"]
"Hello World"

>>> feature.yourcolumn
"Hello World"

SIP API v2

This is the most recent change that has caused all of the 1.8 plugins to no longer work in 2.0. A bit of background: PyQt4 and pyqgis use sip to define their Python API. There are two versions of SIP V1 and V2. Using SIP V1 PyQt, and therefor pyqgis, will take and return QVariant and QString objects in Python.

So:

>>> feature["yourcolumn"]
<PyQt4.QtCore.QVariant object at 0x026AD5E0>

Well that is fine but to get a string value you had to do this:

>>> feature["yourcolumn"].toString()
"Hello World"

but that isn't a normal Python string. It's a QString. QStrings can't be used in normal Python methods so you have to do str(feature["yourcolumn"].toString()). Gross!

With the change to SIP V2 QVariant and QString are now auto converted to native Python types without any extra work.

>>> type(f["Name"])
<type 'unicode'>

Good stuff.

If you are looking for a conversion guide check out: http://hub.qgis.org/wiki/quantum-gis/PythonpluginAPIchangesfrom18to_20

The QGIS plugin repository can house both 1.8 and 2.0 plugins under the same plugin name, and the plugin installer will only download the version compatible with the QGIS version you are using.

The SIP version update was a hard one to make. On one hand we could just leave it the way it was with backwards compatibility and all the messy str(variant.toString()) stuff; OR we can make the switch now while everyone is migrating to the new vector API and have a cleaner API for the future. I picked the later. Once we switch to Python 3 at some stage in the future SIP v2 is the default so it would have broken all the plugins at that point anyway.

I can tell you that I did lose some sleep over if I should push the SIP update or not, wondering what impact it could have on plugin authors and the project in general. In the end I stand by our decision to make the update in QGIS 2.0.

Hopefully these API changes won't cause to much grief for plugin authors and the API is now better to work with.

Fine. But why I do care as a user?

All this means that when QGIS 2.0 comes out you may be missing some plugins that you used to have in 1.8 until all the plugins are migrated. If you have the skills to help updating a plugin please let the plugin author know so you can make their life easier. If you have the ability to fund a plugins update that would be excellent.

QGIS 2.0 has under gone some radical changes for its upcoming release. Some of these major changes relate to the API, both C++ and Python. Something that always comes up when trying to improve an API is the chance that you might have to remove the old way of doing something so that things are cleaner and improved for the future. These changes will always break existing working code and can be a burden for a little while until everything is adjusted. A little pain for a lot of gain is the way I like to think about it. I will also add that breaking API so radically isn't something that we, as developers, like to do or take lightly, but sometimes it has to be done.

This post is a summary of some of the new APIs, what has changed, and what it means to you as a user.

New Vector API (Iterator pattern)

We would like to introduce multithreaded rendering in the future and the old API for getting a set of features from the provider wasn't going to cut it in a multithreaded environment. In order to allow something that will be more threadable then the old select model we changed to a iterator pattern.

The old used to look like this:

layer.select()
for feature in layer:
    print feature

select() wouldn't return anything rather it would tell the provider of the layer to get ready to return some features when asked e.g the for loop. Want to thread that? Nope. Not going to happen.

The solution:

Create a method that returns an iterator that each thread can loop over on their own without changing the result of the other.

features = layer.getFeatures()
for feature in features:
    print feature

if you look at layer.getFeatures() you can see it returns a QgsFeatureIterator

>>> layer.getFeatures()
<qgis.core.QgsFeatureIterator object at 0x1A05F810>

QgsFeatureIterator has a next() method, just like a normal Python iterator, which you can call to get the next feature, it will just serve them up as you need them.

>>> features = layer.getFeatures()
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062CD8>
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062C48>
>>> features.next()
<qgis.core.QgsFeature object at 0x1A062C00>
>>> features.close()

So in the future (you can't do this yet due to some other restrictions):

>>> features = layer.getFeatures()
>>> features2 = layer.getFeatures()
>>> features.next().id()
1L
>>> features.next().id()
2L
>>> features2.next().id()
1L

Nifty!

New Vector API (Fields)

Something else that changed was how attributes are accessed.

This is the old style:

>>> index = layer.fieldNameIndex("yourcolumn")
>>> feature.attributeMap()[index].toString()
"Hello World"

That is a big pain in the butt just to get a value from a field.

The new method now uses a list of attributes on the feature itself.

>>> feature[5]
"Hello World"
>>> feature["yourcolumn"]
"Hello World"

You can even access them like attributes.

>>> feature.yourcolumn
"Hello World"

Be careful with accessing fields via the attribute magic. If you have a id field feature.id will return the feature id method not the field named id. In fact any method with the same name as any QgsFeature methods will return the method rather then the field value. I like this magic but use it wisely.

SIP API v2

This is the most recent change that has caused all of the 1.8 plugins to no longer work in 2.0. A bit of background: PyQt4 and pyqgis use sip to define their Python API. There are two versions of SIP V1 and V2. Using SIP V1 PyQt, and therefor pyqgis, will take and return QVariant and QString objects in Python.

So:

>>> feature["yourcolumn"]
<PyQt4.QtCore.QVariant object at 0x026AD5E0>

Well that is fine but to get a string value you had to do this:

>>> feature["yourcolumn"].toString()
"Hello World"

but that isn't a normal Python string. It's a QString. QStrings can't be used in normal Python methods so you have to do str(feature["yourcolumn"].toString()). Gross!

With the change to SIP V2 QVariant and QString are now auto converted to native Python types without any extra work.

>>> type(f["Name"])
<type 'unicode'>

Good stuff.

If you are looking for a conversion guide check out: http://hub.qgis.org/wiki/quantum-gis/Python_plugin_API_changes_from_18_to_20

The QGIS plugin repository can house both 1.8 and 2.0 plugins under the same plugin name, and the plugin installer will only download the version compatible with the QGIS version you are using.

The SIP version update was a hard one to make. On one hand we could just leave it the way it was with backwards compatibility and all the messy str(variant.toString()) stuff; OR we can make the switch now while everyone is migrating to the new vector API and have a cleaner API for the future. I picked the later. Once we switch to Python 3 at some stage in the future SIP v2 is the default so it would have broken all the plugins at that point anyway.

I can tell you that I did lose some sleep over if I should push the SIP update or not, wondering what impact it could have on plugin authors and the project in general. In the end I stand by our decision to make the update in QGIS 2.0.

Hopefully these API changes won't cause to much grief for plugin authors and the API is now better to work with.

Fine. But why I do care as a user?

All this means that when QGIS 2.0 comes out you may be missing some plugins that you used to have in 1.8 until all the plugins are migrated. If you have the skills to help updating a plugin please let the plugin author know so you can make their life easier. If you have the ability to fund a plugins update that would be excellent.

Here is a preview of a new cool feature coming in QGIS 2.0. Alpha channel in colour ramps.

Fancy!

The best part is it even works on raster layers.

How bloody awesome is that!

Mathieu Pellerin made a cool made showing of this new feature on our flickr map group

Here is a preview of a new cool feature coming in QGIS 2.0. Alpha channel in colour ramps.

Fancy!

The best part is it even works on raster layers.

How bloody awesome is that!

Mathieu Pellerin made a cool made showing of this new feature on our flickr map group

Here is a preview of a new cool feature coming in QGIS 2.0. Alpha channel in colour ramps.

Fancy!

The best part is it even works on raster layers.

How bloody awesome is that!

Mathieu Pellerin made a cool made showing of this new feature on our flickr map group

Joining an open source project has been one of the best things I did for my career. To better myself in the process of improving QGIS. To grow as a GIS professional. To learn to be part of team and respect each others ideas even if don't agree. To be open to ideas that others have spent a lot of time on. To not just be single tool pusher and learn a wider range of toolkits. To work with people who you have never seen in person, or even talked to.

How did it feel to join an open source project? Scary but awesome I would say. Scary in that that everyone would read my code - my very amateur code. Scary in that I had never really joined an open source project before and wasn't sure what everyone would think of my work, or my ideas. Scary because there are so many people better at this than me and they might think I'm crap.

However all of this fear is outweighed by the awesome feeling of knowing that people benefit from the work you, and the rest of the team, put in. Sometimes even the small things can make a huge difference to what people can do with your software - and yes I did use "your software" on purpose. Contributing to open source means you, personally, are part of the software. Being a contributor, for me at least, gives you a deeper relationship with the project. The kind of relationship that sees you staying up at night when the rest of the family is in bed trying out a new idea, fixing some annoying bug, or writing a blog post about being an open source contributor. All for free! - well mostly. Some call it obsession I prefer passion.

It's not pretty roses all the time. Passion can lead to burnout. Trying to be involved in most of the major parts of the project can result in stretching yourself thin. A project that never sleeps makes this even harder. The wave of ideas can sometimes be a distraction from just getting stuff done. Rejection of your work can be hard to handle the first time, you just have to remember it's never personal. Most of these are just personal demons that need to be managed but they do sneak up if you enjoy what you do. Having a family - and an xbox - helps to ground you and make sure you don't spend all your free time on the computer hacking away.

Like I said, and despite personal demons, joining an open source project has been one of the best things I did. There is an emotional kick working on something and seeing it used by other people. I didn't expect my expression based labeling addition would get such good remarks but it did and that helped push me further into becoming a QGIS contributor.

Joining an open source project has been one of the best things I did for my career. To better myself in the process of improving QGIS. To grow as a GIS professional. To learn to be part of team and respect each others ideas even if don't agree. To be open to ideas that others have spent a lot of time on. To not just be single tool pusher and learn a wider range of toolkits. To work with people who you have never seen in person, or even talked to.

How did it feel to join an open source project? Scary but awesome I would say. Scary in that that everyone would read my code - my very amateur code. Scary in that I had never really joined an open source project before and wasn't sure what everyone would think of my work, or my ideas. Scary because there are so many people better at this than me and they might think I'm crap.

However all of this fear is outweighed by the awesome feeling of knowing that people benefit from the work you, and the rest of the team, put in. Sometimes even the small things can make a huge difference to what people can do with your software - and yes I did use "your software" on purpose. Contributing to open source means you, personally, are part of the software. Being a contributor, for me at least, gives you a deeper relationship with the project. The kind of relationship that sees you staying up at night when the rest of the family is in bed trying out a new idea, fixing some annoying bug, or writing a blog post about being an open source contributor. All for free! - well mostly. Some call it obsession I prefer passion.

It's not pretty roses all the time. Passion can lead to burnout. Trying to be involved in most of the major parts of the project can result in stretching yourself thin. A project that never sleeps makes this even harder. The wave of ideas can sometimes be a distraction from just getting stuff done. Rejection of your work can be hard to handle the first time, you just have to remember it's never personal. Most of these are just personal demons that need to be managed but they do sneak up if you enjoy what you do. Having a family - and an xbox - helps to ground you and make sure you don't spend all your free time on the computer hacking away.

Like I said, and despite personal demons, joining an open source project has been one of the best things I did. There is an emotional kick working on something and seeing it used by other people. I didn't expect my expression based labeling addition would get such good remarks but it did and that helped push me further into becoming a QGIS contributor.

Joining an open source project has been one of the best things I did for my career. To better myself in the process of improving QGIS. To grow as a GIS professional. To learn to be part of team and respect each others ideas even if don't agree. To be open to ideas that others have spent a lot of time on. To not just be single tool pusher and learn a wider range of toolkits. To work with people who you have never seen in person, or even talked to.

How did it feel to join an open source project? Scary but awesome I would say. Scary in that that everyone would read my code - my very amateur code. Scary in that I had never really joined an open source project before and wasn't sure what everyone would think of my work, or my ideas. Scary because there are so many people better at this than me and they might think I'm crap.

However all of this fear is outweighed by the awesome feeling of knowing that people benefit from the work you, and the rest of the team, put in. Sometimes even the small things can make a huge difference to what people can do with your software - and yes I did use "your software" on purpose. Contributing to open source means you, personally, are part of the software. Being a contributor, for me at least, gives you a deeper relationship with the project. The kind of relationship that sees you staying up at night when the rest of the family is in bed trying out a new idea, fixing some annoying bug, or writing a blog post about being an open source contributor. All for free! - well mostly. Some call it obsession I prefer passion.

It's not pretty roses all the time. Passion can lead to burnout. Trying to be involved in most of the major parts of the project can result in stretching yourself thin. A project that never sleeps makes this even harder. The wave of ideas can sometimes be a distraction from just getting stuff done. Rejection of your work can be hard to handle the first time, you just have to remember it's never personal. Most of these are just personal demons that need to be managed but they do sneak up if you enjoy what you do. Having a family - and an xbox - helps to ground you and make sure you don't spend all your free time on the computer hacking away.

Like I said, and despite personal demons, joining an open source project has been one of the best things I did. There is an emotional kick working on something and seeing it used by other people. I didn't expect my expression based labeling addition would get such good remarks but it did and that helped push me further into becoming a QGIS contributor.

Want to show of some the cool maps you have made using QGIS 1.9/2.0?

Mathieu has now setup a new Flickr group to collect and show of cool maps that have been made using the new features in the upcoming QGIS 2.0.

The group can be found at http://www.flickr.com/groups/qgis/pool

Here is one I uploaded a few days ago

and a cool one from Anita

Anyone can join and submit to the group.

There are just a few rules:

No Screen shots of the application Only output from the composer or Save Image As.. List data sources used List any new QGIS features used e.g blending, label buffers, etc Only post maps you are proud of or you would show your mother A QGIS Map Boo k might be on the cards in the future so keep that in mind when uploading a map to the group.

Now go forth and make awesome maps!

Want to show of some the cool maps you have made using QGIS 1.9/2.0?

Mathieu has now setup a new Flickr group to collect and show of cool maps that have been made using the new features in the upcoming QGIS 2.0.

The group can be found at http://www.flickr.com/groups/qgis/pool

Here is one I uploaded a few days ago

and a cool one from Anita

Anyone can join and submit to the group.

There are just a few rules:

No Screen shots of the application Only output from the composer or Save Image As.. List data sources used List any new QGIS features used e.g blending, label buffers, etc Only post maps you are proud of or you would show your mother A QGIS Map Boo k might be on the cards in the future so keep that in mind when uploading a map to the group.

Now go forth and make awesome maps!

Want to show of some the cool maps you have made using QGIS 1.9/2.0?

Mathieu has now setup a new Flickr group to collect and show of cool maps that have been made using the new features in the upcoming QGIS 2.0.

The group can be found at http://www.flickr.com/groups/qgis/pool

Here is one I uploaded a few days ago

and a cool one from Anita

Anyone can join and submit to the group.

There are just a few rules:

No Screen shots of the application Only output from the composer or Save Image As.. List data sources used List any new QGIS features used e.g blending, label buffers, etc Only post maps you are proud of or you would show your mother A QGIS Map Boo k might be on the cards in the future so keep that in mind when uploading a map to the group.

Now go forth and make awesome maps!