GRASS GIS is an open source geoinformation system which is developed by a globally distributed team of developers. Besides the source code developers also message translators, people who write documentation, those who report bugs and wishes and more are involved.

1. Early days… from pre-Internet to CVS and SVN

While GRASS GIS is under development since 1982 (no typo!) it has been put into a centralized source code management system in December 1999. Why so late? Because the World Wide Web (WWW) became available in the 1990s along with tools like browsers and such, followed by the development of distributed source code management tools. We moved on 29th Dec 1999 (think Y2K bug) the entire code into our instance of CVS (Concurrent Versioning System). With OSGeo being founded in 2006, we migrated the CVS repository to SVN (Subversion for the source code management) and trac (bug and wish tracker) on 8 Dec 2007. See here for historic details on our various bug trackers.

2. Time to move on: git

Now, after more than 10 years using SVN/trac time had come to move on and join the large group of projects managing their source code in git (see also our related Wiki page on migration). Git comes with numerous advantages, yet we needed to decide which hosting platform to use. Options where github.com, gitlab.com, gitlab or gitea on OSGeo infrastructure, or other platforms. Through a survey we found out that the preference among contributors is GitHub. While not being open source itself it offers several advantages: it is widely known (good to get new developers interested and involved), numerous OSGeo projects are hosted there under the GitHub “OSGeo organization“.

If all fails (say, one day GitHub no longer being a reasonable choice) the import of our project from GitHub to GitLab is always possible. Indeed, we meanwhile mirror our code on OSGeo’s gitea server.

Relevant script code and migration ticket:

• svn -> git test migration ongoing, see #3722
• svn2git code written by Martin Landa
  • https://trac.osgeo.org/grass/browser/grass-addons/tools/svn2git/
• Martin Landa also forked an existing “trac2github” PHP convertion script which includes GRASS related changes; it is available at https://github.com/landam/trac2github/tree/grass

Relevant steps:

• migrated SVN trunk -> git master
• migrated and tagged release branches (milestones)
• deleted “develbranch6” (we compared it to “releasebranch_6_4” and didn’t discover relevant differences)
• Fix commit messages (yes, we really wanted to be brave, updating decades of commit messages!):
  • references to old RT tracker tickets (used Dec 2000 – Dec 2006)
  • references to old GForge tracker tickets (used Jan 2007 – Dec 2008)
  • references to other trac tickets (#x -> https://trac.osgeo.org/…)
    • see also https://grasswiki.osgeo.org/wiki/Bug_tracking

3. Source code migration: the new git repositories

• github repository “grass” (repo)
  
  • Source code from 1999 to present day (SVN-trunk -> git-master)
  • all 7.x release branches
• github repository “grass-legacy” (repo)
  
  • separate repository for older GRASS GIS releases (3.2, 4.x, 5.x, 6.x), hence source code now available in git since 1987!
• github repository “grass-addons” (repo)
  
  • repository for addons
• github repository “grass-promo” (repo)
  • repository for promotional material
• github repository “grass-website” (repo)
  • repository for upcoming new Website

4. Remarks on the “grass-legacy” repository

What special about it:

• the source code goes back to 1987!
• file timestamps (which I tried to preserve for decades :-) have been used to reconstruct the source code history (e.g., releasebranch_3_2)
• junk files removed (plenty of leftover old binary files, files consisting of a special char only etc)
• having this grass-legacy repo available in parallel to the main grass repo which contains the  recent source code we have a continuous source code coverage from 1987 to today in git.
• size is about 250MB

What’s missing

• the 4.3 source code doesn’t have distinct timestamps. Someone must have once packaged without mtime preservation… a pity. Perhaps a volunteer may fix that by carrying over the timestamps from GRASS GIS 4.2 in case the md5sum of a file is identical (or so).

5. Trac issue migration

A series of links had to be updated. Martin Landa invested days and days on that (thanks!!). He used the related GDAL efforts as a basis (Even Rouault: thanks!). As the date for the trac migration we selected 2007-12-09 (r25479) as it was the first SVN commit (after the years in CVS). The migration of trac bugs to github (i.e. transfer of trac ticket content) required several steps:

Link updates in the ticket texts:

• links to other tickets (now to be pointed to full trac URL). Note that there were many styles of referring in the commit log message which had to be parsed accordingly
• links to trac wiki (now to be pointed to full trac URL)
• links source code in SVN (now to be pointed to full trac URL)
• images and attachments (now to be pointed to full trac URL)

Transferring:

• “operating system” trac label into the github issue text itself (following the new issue reporting template)
• converting milestones/tickets/comments/labels
• converting trac usernames to Github usernames
• setting assignees if possible, set new “grass-svn2git” an assignee otherwise
• slowing down transfer to match the 60 requests per second API limit rate at github

6. Fun with user name mapping

Given GRASS GIS’ history of 35+ years we had to invest major effort in identifying and mapping user names throughout the decades (see also bug tracker history). The following circumstances could be identified:

• user present in CVS but not in SVN
• user present in SVN but not in CVS
• user present in both with identical name
• user present in both with different name (well, in our initial CVS days in 1999 we often naivly picked our surnames like “martin”, “helena”, “markus”, “michael” … cute yet no scaling very much over the years!) as some were changed in the CVS to SVN migration in 2007, leading to
  • colliding user names
• some users already having a github account (with mostly different name again)

We came up with several lookup tables, aiming at catching all variants. Just a “few” hours to dig in old source code files and in emails for finding all the missing email addresses…

7. Labels for issues

We cleaned up the trac component of the bug reports, coming up with the following categories which have to be visually grouped by color since the label list is just sorted alphabetically in github/gitlab:

• Issue category:
  • bug
  • enhancement
• Issue solution (other than fixing and closing it normally):
  • duplicate
  • invalid
  • wontfix
  • worksforme
• Priority:
  • blocker
  • critical
  • feedback needed
• Components:
  • docs
  • GUI
  • libs
  • modules
  • packaging
  • python
  • translations
  • unittests
  • Windows specific

Note that the complete issue migration is still to be done (as of Nov. 2019). Hopefully addressed at the GRASS GIS Community Sprint Prague 2019.

8. Setting up the github repository

In order to avoid users being flooded by emails due to the parsing of user contributions which normally triggers an email from github) we reached out to GitHub support in order to temporarily disable these notifications until all source code and selected issues were migrated.

The issue conversion rate was 4 min per trac bug to be converted and uploaded to github. Fairly slow but likely due to the API rate limit imposed and the fact that the migration script above generates a lot of API requests rather than combined ones..
Note to future projects to be migrated: use the new gihub import API (unfortunately we got to know about its existence too late in our migration process).

Here out timings which occurred during the GRASS GIS project migration from SVN to github:

• grass repo: XX hours (all GRASS GIS 7.x code)
• grass-legacy repo: XX hours (all GRASS GIS 3.x-6.x code)
• NNN issues: XX hours – forthcoming.

9. New issue reporting template

In order to guide the user when reporting new issues, we will develop a small template – forthcoming.

10. Email notifications: issues to grass-dev and commits to grass-commit

We changed the settings from SVN post-hook to Github commit notifications and they flow in smoothly into the grass-commit mailing list. Join it to follow the development.

Overall, after now several months of using our new workflow we can state that things work fine.

The post Remarks on SVN-trac to GitHub migration appeared first on GFOSS Blog | GRASS GIS and OSGeo News.

I often do this – I work away on my master branch and suddenly realise ‘damn I should have branched before I started this’. Usually this happens because what I think will be a trivial change snowballs into something much more sweeping. I thought I would record here how to deal with situations like this... Read more »

I often do this - I work away on my master branch and suddenly realise 'damn I should have branched before I started this'. Usually this happens because what I think will be a trivial change snowballs into something much more sweeping. I thought I would record here how to deal with situations like this by retroactively creating a branch and rolling master back to a previous state. Note: backup your data first and this is only ideal if you haven't pushed changes to a remote repo yet.

So here was the situation. My last 'known good' state was 8399227db and I wanted to move all subsequent work out of master and into a branch. So first lets create that branch starting from the last known good state:

git branch refactor_branch 8399227db

Next I check out the branch and merge all the changes from master into it:

git checkout refactor_branch
git merge master

Now I switch back to master and chuck away all changes made since the branch point (did I mention you should backup your work first?):

git checkout master
git reset --hard 8399227db

So I rebased the master branch to an older commit. Now I push my branch up to the remote repository and breath a sigh of relief that my work is backed up in the cloud and master remains usable for others working on it:

git push origin refactor_branch

As a last couple of steps, I remove my local branch and checkout the remote branch as a tracking branch:

git branch -D riab_to_inasafe
git branch --track riab_to_inasafe origin/riab_to_inasafe
git checkout riab_to_inasafe

Git being git, there are probably a thousand other ways to accomplish the same thing, but the above process works for me!

This post is for those of you that build QGIS on a regular basis and want to keep up with everything going on in the current release branches (1.7.2 and 1.8) as well as the master branch that will eventually become version 2.0.

While you can do all your work in one clone, this method has a couple of advantages, at the expense of a bit of disk space:

1. Quicker compiles compared to branch switching, especially if you are using ccache
2. Less likelihood of making a merge mess when switching branches

The basic steps are:

1. Login to github and clone the QGIS repository (https://github.com/qgis/Quantum-GIS)
2. Create a working copy of your github clone on your machine
3. Add a reference to the upstream repository
4. Fetch and merge (if required) from the upstream repository
5. Create a new clone for the branch by cloning the working copy created in step 2
6. Change to the branch clone directory
7. Add the upstream remote
8. Fetch from upstream
9. Create the tracking branch
10. Checkout the branch

It’s simpler than it sounds—the following steps should work on any platform and assume you have already created your own clone of the QGIS repository on github.

First make a directory for development, change to it, and fetch a copy of your clone of QGIS from github (this will take a while):

gsherman@dork:~$ mkdir qgis_dev
gsherman@dork:~$ cd qgis_dev
gsherman@dork:~/qgis_dev$ git clone [email protected]:g-sherman/Quantum-GIS.git
Cloning into Quantum-GIS...
remote: Counting objects: 183812, done.
remote: Compressing objects: 100% (42255/42255), done.
remote: Total 183812 (delta 140627), reused 183281 (delta 140221)
Receiving objects: 100% (183812/183812), 240.80 MiB | 1.19 MiB/s, done.
Resolving deltas: 100% (140627/140627), done.


Now change to your new clone directory and add the upstream repository (the QGIS repo on github):

gsherman@dork:~/qgis_dev$ cd Quantum-GIS/
gsherman@dork:~/qgis_dev/Quantum-GIS$ git remote add upstream [email protected]:qgis/Quantum-GIS.git

We can then list our config to see that the remote was added:

gsherman@dork:~/qgis_dev/Quantum-GIS$ git config --list
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
[email protected]:g-sherman/Quantum-GIS.git
branch.master.remote=origin
branch.master.merge=refs/heads/master
[email protected]:qgis/Quantum-GIS.git
remote.upstream.fetch=+refs/heads/*:refs/remotes/upstream/*


Then we do a fetch from the QGIS repo to make sure things are up to date:

gsherman@dork:~/qgis_dev/Quantum-GIS$ git fetch upstream


If your clone of the QGIS repository on github is not brand new, you should do a merge before proceeding:

gsherman@dork:~/qgis_dev/Quantum-GIS$ git merge upstream/master


Now we can create a new clone for the 1.7.2 branch using our local master:

gsherman@dork:~/qgis_dev/Quantum-GIS$ cd ..
gsherman@dork:~/qgis_dev$ git clone ./Quantum-GIS Quantum-GIS-1_7_2
Cloning into Quantum-GIS-1_7_2...
done.


Now we need to add the remote for the main QGIS repository:

gsherman@dork:~/qgis_dev$ cd Quantum-GIS-1_7_2/
gsherman@dork:~/qgis_dev/Quantum-GIS-1_7_2$ git remote add upstream [email protected]:qgis/Quantum-GIS.git


The next step is to fetch from upstream to make sure we have references to the branches:

gsherman@dork:~/qgis_dev/Quantum-GIS-1_7_2$ git fetch upstream
* [new branch] dev-threading -> upstream/dev-threading
* [new branch] master -> upstream/master
* [new branch] release-0_0_11 -> upstream/release-0_0_11
* [new branch] release-0_0_12 -> upstream/release-0_0_12
...
* [new branch] release-1_7_1 -> upstream/release-1_7_1
* [new branch] release-1_7_2 -> upstream/release-1_7_2
* [new branch] release-1_8 -> upstream/release-1_8

Now we create the branch to track the release of interest—in this case 1.7.2:

gsherman@dork:~/qgis_dev/Quantum-GIS-1_7_2$ git branch --track release-1_7_2 upstream/release-1_7_2
Branch release-1_7_2 set up to track remote branch release-1_7_2 from upstream.

The last step is to check out the branch which should go pretty quick:

gsherman@dork:~/qgis_dev/Quantum-GIS-1_7_2$ git checkout release-1_7_2
Checking out files: 100% (1809/1809), done.
Switched to branch 'release-1_7_2'

You are now ready to build the 1.7.2 release branch. Repeat the process for any other branches you want to track.

Here is a quick video I did this morning using the very cool tool Gource. The video shows all the commits (1265) that were made between release 1.6 and 1.7. QGIS 1.7 was released on 19th June 2011 but there was still some clean up in the 1.7 branch paste that date into September.

It’s interesting to note the large burst of activity around November.

Tonight Tim Sutton officially made the release announcement for QGIS 1.7, so I’m guess I’m free to blog about the newest version now and its cool features.

What are you still doing here? Go and get it! http://www.qgis.org/wiki/Download

I am, as a heavy QGIS user and a guy-who-tries-to-write-features-and-patches-for-the-code, very happy with this release. I know a lot of people have put a lot of hard work and free time into working on features and bug fixes that keep making this free GIS system even better.

Some of the more notable new features in this release are…well there are just way to many for me to list here so go and check out the official list at http://qgis.org/component/content/article/127-qgis-1-7-release.html

The QGIS team has shifted their source control system to using GIT, which I am very happy about as a lot of the guys on the #qgis IRC channel will know :). The bug tracer has also been moved tohttp://hub.qgis.org/projects/quantum-gis.

Since the release of QGIS 1.6 there have been 1199 commits (using git to count: git log –pretty=oneline upstream/release-1_6_0..upstream/release-1_7_0 | wc -l). Not a bad effort if I may so myself.

If you are still reading this, I really hope it’s because you are waiting for QGIS 1.7 to install.

I forked the QGIS repo on GitHub a little while ago although one thing that bothered me was that it gave me copies of all the branches that the main QGIS git repo had.  This is understandable as it’s the way it works however I don’t really need all these branches in my forked copy of the repo as I don’t care about them.   I only care the ones I am working on, and I don’t want to see a big list of branches in my git fork that have nothing to do with me.

So the next question was how do I delete all the branches on the remote repo at GitHub. Well you would normally do:

git push origin :branch_name

although doing that by hand for each branch is, well, a pain in the butt! After a chat with a guy (strk) on the #qgis IRC channel who is more skilled at bash then me (I’m still a Linux noob) he came up with this:

git branch -r | grep origin/ | grep -v master | grep -v HEAD| cut -d/ -f2 | while read line; do git push origin :$line; done;

Very handy.

Writing a QGIS plugin is not overly complicated but represents a bit of work. Using
git in conjunction with your development efforts can make sure your investment in
coding time is preserved.

Development Tools

The QGIS project team has set up a central location for plugin development which includes pretty much everything you need to develop and support your plugins, including:

• Issue tracking
• Wiki
• Documents
• Repository

The repository feature allows you to create a central place to store your plugin code using git. Others can clone your repository and contribute through patches or pull requests.

Creating a Plugin

The chore of setting up the boilerplate for a plugin is made simple by using the Plugin Builder. Previously this was a web application but it has now been replaced by a QGIS plugin aptly named Plugin Builder. You can install Plugin Builder from within QGIS by selecting Fetch Python Plugins… from the Plugins menu.

Once Plugin Builder is installed you can quickly create a starter plugin that implements a simple dialog box with OK and Cancel buttons. The plugin created by Plugin Builder is fully functional—it will load in QGIS but not do anything useful until you customize it.

Setting up the Workflow

You have a couple of options for setting up your development directory and workflow:

1. Copy your plugin template to the QGIS plugin directory, initialize the git repository, and develop from there.
2. Work within your plugin template directory that was created by Plugin Builder (not within the QGIS plugin directory)

Option one is very convenient as long as you don’t test the uninstall feature of your plugin. This will delete your entire plugin directory and git repository—not what we really want.

Option two is safer, however to test your plugin you have to continually copy from your development area to your QGIS plugin directory. Alternatively you could make commits and pull from your development area to the copy (assuming it is a git repo) in the QGIS plugin directory. If you are on a unix based system you could also create a Makefile to deploy the plugin for you.

The best solution is to use option two and use the QGIS_PLUGINPATH environment variable to point to your development directory. When present, QGIS_PLUGINPATH tells QGIS to search additional directories for plugins. Going this route allows you to develop in the directory created by Plugin Builder and test your plugin without any copying or pulling. The upside to this is since your plugin wasn’t installed through the Plugin Installer it can’t be uninstalled accidentally. When you are ready to test the uninstall and unloading of your plugin you can copy it to the main QGIS plugin directory.

Note: If you are using Windows there was a problem specifying QGIS_PLUGINPATH with colons in the path. This issue is fixed in revision 15073 and will make it into the next release.

No matter how you implement your workflow I suggest creating a repository on the QGIS hub (http://hub.qgis.org) or github.com.

Summary

Here is a summary of the steps to get started. We’ll assume your new plugin is named zoomer:

1. Install Plugin Builder
2. Create a directory that will contain all your plugins, for example my_plugins
3. Create your plugin template using Plugin Builder. Be sure to select my_plugins when Plugin Builder asks where to create your plugin.
4. Change to your plugin directory (e.g. my_plugins/zoomer) created by Plugin Buidler and create a git repository using:

   git init

5. Set the QGIS_PLUGINPATH evironment variable to point to the directory containing your plugin directory (my_plugins). Be sure to use the full path to my_plugins
6. Start QGIS and use the Plugin Manager (Manage Plugins… from the Plugins menu) to enable your plugin. If it doesn’t show up in the list of plugins check to make sure you have set QGIS_PLUGINPATH correctly.
7. Develop away, testing as you go. Make sure to commit changes regularly and push them to your repository on hub.qgis.org.

If you are new to git, take a look at the Pro Git book at http://progit.org/.

For help on developing QGIS plugins with Python, see the PyQGIS Cookbook.

A couple of days I found a pretty cool open source project for visualizing the history of version controlled code.  The project is called Gource and can be found here: http://code.google.com/p/gource/ On the videos wiki page there are a few videos of other projects that have used gource to generate cool videos of their commit history so I thought I should make one for QGIS.

After downloading and building the latest source for Gource and fetching the current trunk of QGIS following these instructions: http://spatialgalaxy.net/2010/12/27/contributing-to-qgis-using-git/ I ran the following in my terminal on my Ubuntu machine, with my current directory being the download git repo from the above instructions:

  gource --title "Quantum GIS" --logo images\icons\qgis-icon.png \
  --hide filenames --date-format "%d, %B %Y" --seconds-per-day 0.15 \
  --highlight-all-users --auto-skip-seconds 0.5 --file-idle-time 0 --max-files 999999999 \
  --multi-sampling --stop-at-end --elasticity 0.1 -b 000000 \
  --disable-progress --user-friction .2 --output-ppm-stream - | \
  ffmpeg -an -threads 4 -y -b 3000K -vb 8000000 -r 60 -f image2pipe \
  -vcodec ppm -i - -vcodec libx264 -vpre libx264-medium qgis.mp4

Don’t worry I know it looks crazy but it’s really not that bad.  I’ll break it down.

• –title “Quantum GIS” Well, yeah, adds a title to the project.
• –logo images/icons/qgis-icon.png Adds a icon watermark
• –hide filenames This hides the filenames of the files being committed . I hide these because it makes it pretty hard to see.
• –date-format “%d, %B %Y” Formats the date at the top of the video.
• –seconds-per-day 0.15 How many seconds represent a day. The lower this is the fast a day goes by, meaning me commits in less time.
• –highlight-all-users Highlights all the users all the time.
• –auto-skip-seconds 0.5 If there are no commits for this time it will skip to the next commit.
• –file-idle-time 0 How long before the file disappears from the video, 0 means never good for seeing the full file tree.
• –stop-at-end Stops the video at the end of all the commits.
• -b 000000 The background colour, in this case black.
• –output-ppm-stream – Tells gource to output the result ppm stream to STDOUT, which is then piped ( | ) into the ffmpeg

That is a quick overview of some of the gource arguments, running grouce -H in your terminal will print out the full list.  I’m not going to go into the ffmpeg arguments because frankly I don’t understand them very well and video isn’t really my thing.  I’m sure there are ffmpeg experts that would be able to do it better then what I have.

After running the above commands in my terminal window and letting it do its thing, I had a resulting mp4 file which I then uploaded to YouTube.

Below is the video that I uploaded to YouTube, which took about 5 hours due to my very very slow (read 138kb/s) upload speed.  The video is about 8 years of QGIS development in just over 9 minutes.

As it’s a bit hard to see in the video due to the quality, each cluster of files is a directory and the branches show the folder hierarchy.

Direct link to video: http://www.youtube.com/watch?v=-NILKRiMtcU

Happy new year and happy coding

One of the challenges in any open source project is accepting contributions from people that don’t have, need, or want access to your centralized source code repository. Managing repository accounts for occasional or one-time contributors can be come a bit of an administrative issue. To date, the QGIS project has accepted one-time or occasional contributions through patches submitted via a help ticket.

To make it easier for you to contribute to QGIS, we have created a clone of the Subversion repository on GitHub. This allows you to “fork” the QGIS repository and have your own local copy of the source code. Through GitHub you can easily submit your enhancements and bug fixes for inclusion in the Subversion repository.

You will need to install git on your computer and create a GitHub account. Once you have done that, here is the process for creating your working copy and contributing to QGIS:

1. Login to GitHub
2. Fork the project at https://github.com/qgis/qgis by clicking “Fork”. This may take a while
3. Create a working copy, replacing g-sherman with your GitHub name:
   git clone [email protected]:g-sherman/qgis.git

   This will also take a while, depending on the speed of your network connection. My clone downloaded 162 Mb from GitHub.

4. Change to the directory containing your local clone and add a reference to the original QGIS repo:
   git remote add upstream git://github.com/qgis/qgis.git

   This gives you are reference named upstream that points to the repo you forked.

5. Now fetch from upstream:
   git fetch upstream

6. Now you are ready to work with QGIS. Make your changes, commit them, and then push back to your fork on GitHub:
   git push origin master

7. You can go to your repo on GitHub and you should see your commit message from the push
8. Now you want to tell the QGIS developer team there is something they should look at. You do this by issuing a pull request by clicking on the “Pull Request” button on your GitHub QGIS fork page.
9. Fill in a title and message for the request, review the commit(s) and file(s) involved and when satisfied, click “Send pull request”
   

10. Now sit back and wait for someone to review, comment on, and hopefully merge your request into the QGIS repo.

    To keep in sync with the QGIS repo, use
    git fetch upstream
git merge upstream/master


    Make sure to check out these resources for help and more information on working with repositories on GitHub:

    • GitHub Help: http://help/github.com
    • Forking: http://help.github.com/forking/
    • Pull Requests: http://help.github.com/pull-requests/
      
      by Gary Sherman at 4:43 PM under git , open source , qgis , quantum gis (Comments)

For source code management in the QGIS project we have been happily using subversion for some time now, having made a not-too-painful transition over from CVS several years ago. Lately we have bandied around the idea of moving over to GIT. This is mostly out of fear of being called bad things by Linus Torvalds for using SVN (ok I'm kidding). Actually there are a lot of technical reasons why a move over to GIT would work well for us - not least of which is the idea of easier merging of branches and the redunancy offered by multiple repos being 'out there'.

Adding a little impetus to my interest in learning GIT is the fact that more people are using it for managing their code so I need to update my skills to be able to collaborate with them effectively.

All this brings me to my real point: today I found a great resource which provides a guide for those familiar with SVN and wanting to migrate to GIT.

Using Git with Subversion makes adding new features easy. Here are the metrics for my latest QGIS hack:

• SVN revisions by others while working on my branch: 177
• Time to complete merge with latest SVN revision: 1 second
• Conflicts: None

Coincidence? Maybe not.