This is just an example of the advantage provided by the multithreading on the Toolbox. As each algorithm is running in a separate thread, this allows us to run more than one algorithm at the same time.

I’m polishing the implementation and trying to find some major problems in the current implementation.  It has some minor bugs that can be easily solved by adding a parameter to the  cancel signal (e.g. when you have two algorithms running and try to cancel one of them, the signal emitted will cancel two algorithms instead of just one)

What do I have completed this week?

• Solved the issue with the outputs.
• Did some tests with QGIS algorithms which seems to be working fine.
• Refactored all QGIS algorithms in order to remove the progress bar parameter and use the signal instead.
• I have changed the Buffer used by QGIS algorithms: the Buffer is now a subclass of QObject. This new implementation allows the buffer to have a parent (the algorithm) and get access to the algorithm’s signal.
• SagaUtils refactored in the same way I did with the Buffer.
• SAGA algorithms refactored in order to use the signals instead of the progress parameter (Not finished yet)

What am I going to achieve for the next week?

• Solve the issue with SAGA algorithms.
• Code refactoring in order to improve the previous implementation on the AlgorithmDialog.
• Refactor Grass/Grass7.
• Bug fixes.

Is there any blocking issue?

There is an issue with SAGA algorithms that makes the Toolbar crash.

What do I have completed this week?

• Added a dedicated threadPool for the Processing Toolbox to run the algorithms.

• Bugfix on the cancel option. Avoid connecting  the signals every time the algorithm runs. This was making the slots to be called multiple times when running and cancelling the algorithm several times.

Is there any blocking issue?

• I wasn’t able to fix the issue with the cancel option when you have more than one algorithm running. The issue turned out to be more complicated than I first expected.

What do I have completed this week?

• Solved the issue with SAGA algorithms that makes the Toolbox crash.
• Refactored Grass/Grass7 and OTB algorithms to work with signals.
• Bug fixes.

What am I going to achieve for the next week?

• Test Grass/Grass7 and OTB algorithms.
• Add an option to cancel the execution of the algorithm and kill the thread.
• Code refactoring.
• Start implementing the runalgIterating.

Is there any blocking issue?

• I had a very busy week due to my university deadlines.
• An issue with my SAGA installation made me think that something was wrong with my implementation and made me lose some time.
• The issue with the SAGA installation also prevented me from testing the algorithms.

What do I have completed this week?

• Processing refactored in order to create a non-blocking implementation of the python console. The algorithm is processed in another thread asynchronously and the result is printed to the python console when the execution is finished.
• AlgorithmDialog refactored to create a non-blocking dialog interface with a button to cancel the execution.
• R algorithms refactored to support the progress signal.

What am I going to achieve for the next week?

• Create a mechanism to hold the result of the algorithm taking into account that there can be several algorithms running at the same time.
• Figure out a way to stop the execution of the algorithm. In third-party algorithms it can not be possible to just interrupt the thread loop and quit the thread. The only available option is to discard the result because the algorithms don’t have any kind of check-point that allows to verify once in while if we want to stop the execution. Kill the thread during the execution can be dangerous and it is not a good option.
• Test the refactored algorithms.

Is there any blocking issue?

• It took more time than I expect to create a non-blocking interface.
• Quitting the thread execution turned out to be far more complicated due to the fact that the algorithms don’t provide any mechanism to allow quitting the thread loop during the execution.

What do I have completed this week?

• Created a mechanism to hold the result of the algorithm taking into account that there can be several algorithms running at the same time. The results are stored in a global variable called algResults. This variable is accessible via python console (e.g. “processing.algResults”).

• The signals are connected in a different way in order to close the dialog and quit the thread when the algorithm is finished.
• I have updated my fork and merged my work into the master branch.
• Added the runalgIterating method to the AlgorithmExecutor with some changes to support the signals to set the progress
• AlgorithmDialog refactored.
• I have been searching for a solution to stop the algorithm execution and I think that is possible to create a sort of checkpoints inside the algorithm to check if we want to stop the execution and raise an exception to stop the thread event loop.

What am I going to achieve for the next week?

• Allow the user to cancel the execution when running the runalgIterating.
• Solve the problem with QGIS dependencies on Ubuntu and test the refactored algorithms.
• Continue working on the algorithm iterating.

Is there any blocking issue?

• I had to postpone the tests due to some problems installing QGIS dependencies on Ubuntu.
• Didn’t had time to implement a solution to stop the execution of the algorithm.

What do I have completed this week?

• Solved an issue that makes QGIS crash when the main thread doesn’t wait for the worker thread to start.
• I spent most of the time looking up some tutorials about unit testing on QGIS and trying to configure python paths to run the processing tests.

What am I going to achieve for the next week?

• This week I’ll change back to the multithreading implementation in order to add the cancel option to stop the algorithm execution. I will be working on a mechanism to stop the execution on QGIS algorithm.

Is there any blocking issue?

• This week my mentor proposed to start working on a suite test and I changed the plan to start working on a unit test to ensure the correct behaviour of the multithreading implementation. At the end of the week we discussed about asking the community to test the new implementation and report the issues if there is any.

What do I have completed this week?

• During this week I worked on a mechanism to stop the algorithm execution on QGIS algorithms. The approach used to cancel the execution explores python exceptions and QT signals to stop the algorithm from the main thread without having to wait in order to stop the execution and terminate the worker thread. I have used signals to trigger an exception inside the algorithm and stop the execution from the main thread. Since there is a significant amount of algorithms, the mechanism had to be implemented in just two of the QGIS algorithms. This implementation can be further replicated  towards the remaining  QGIS algorithms.

What am I going to achieve for the next week?

• Start working on a test suite for the multithreading to ensure the correct behaviour of the multithreading implementation. The test suite must check if the worker thread is starting, check if signals are triggering the functions when it is supposed to and verify if the cancel option is working properly.

• Figure out how to stop the execution of a non-QGIS algorithm.

Is there any blocking issue?

• No blocking issues

What do I have completed this week?

• During this week I have changed the multithreading implementation to use the QThreadPool to allow thread recycling and avoid thread creation costs every time we want to run an algorithm. This new implementation adds some flexibility to the multithread support and avoid unexpected behaviours in different machines. The number of threads in the QThreadPool is initialised accordingly to the number of cores in the PC and may happen that the QT is not able to detect the number of cores. In this case we will only have one thread in the thread pool, which can be occupied with something else. I’m trying to detect when there is no available threads on the ThreadPool and increase the number of threads if necessary.

What am I going to achieve for the next week?

• Start working on a test suite for the multithreading to ensure the correct behaviour of the multithreading implementation. The test suite must check if the worker thread is starting, check if signals are triggering the functions when it is supposed to and verify if the cancel option is working properly.

• Figure out how to stop the execution of a non-QGIS algorithm.

Is there any blocking issue?

• I have spotted some issues when testing the code in a different machine (with Linux Mint 17) that may have to do with thread unavailability.  Therefore, I decided to postpone the current tasks and focus on changing the multithreading implementation in order to correct this kind of unexpected behaviours.

What do I have completed this week?

• I have finished the cancel option in order to stop the execution of non-QGIS algorithms. It’s important to remember that as well as the QGIS algorithms the cancel option do not cover all the non-QGIS algorithms and this implementation only covers GDAL and SAGA algorithms. However this option can be easily replicated to the othersthird party algorithms.

• I have changed the python console to use the QThreadPool.

• I have been testing and debugging the multithreading implementation looking for unexpected behaviours or incorrect results.

• Core refactoring.

What am I going to achieve for the next week?

• Write documentation for the multithreading implementation.

• Code cleaning and refactoring.

• Search for problems in the multithreading implementation and correct them.

Is there any blocking issue?

• No.

What do I have completed this week?

• Added a dedicated threadPool for the Processing Toolbox to run the algorithms.

• Bugfix on the cancel option. Avoid connecting  the signals every time the algorithm runs. This was making the slots to be called multiple times when running and cancelling the algorithm several times.

Is there any blocking issue?

• I wasn’t able to fix the issue with the cancel option when you have more than one algorithm running. The issue turned out to be more complicated than I first expected.

This is just an example of the advantage provided by the multithreading on the Toolbox. As each algorithm is running in a separate thread, this allows us to run more than one algorithm at the same time.

I’m polishing the implementation and trying to find some major problems in the current implementation.  It has some minor bugs that can be easily solved by adding a parameter to the  cancel signal (e.g. when you have two algorithms running and try to cancel one of them, the signal emitted will cancel two algorithms instead of just one)

What do I have completed this week?

• I have finished the cancel option in order to stop the execution of non-QGIS algorithms. It’s important to remember that as well as the QGIS algorithms the cancel option do not cover all the non-QGIS algorithms and this implementation only covers GDAL and SAGA algorithms. However this option can be easily replicated to the othersthird party algorithms.

• I have changed the python console to use the QThreadPool.

• I have been testing and debugging the multithreading implementation looking for unexpected behaviours or incorrect results.

• Core refactoring.

What am I going to achieve for the next week?

• Write documentation for the multithreading implementation.

• Code cleaning and refactoring.

• Search for problems in the multithreading implementation and correct them.

Is there any blocking issue?

• No.

What do I have completed this week?

• During this week I have changed the multithreading implementation to use the QThreadPool to allow thread recycling and avoid thread creation costs every time we want to run an algorithm. This new implementation adds some flexibility to the multithread support and avoid unexpected behaviours in different machines. The number of threads in the QThreadPool is initialised accordingly to the number of cores in the PC and may happen that the QT is not able to detect the number of cores. In this case we will only have one thread in the thread pool, which can be occupied with something else. I’m trying to detect when there is no available threads on the ThreadPool and increase the number of threads if necessary.

What am I going to achieve for the next week?

• Start working on a test suite for the multithreading to ensure the correct behaviour of the multithreading implementation. The test suite must check if the worker thread is starting, check if signals are triggering the functions when it is supposed to and verify if the cancel option is working properly.

• Figure out how to stop the execution of a non-QGIS algorithm.

Is there any blocking issue?

• I have spotted some issues when testing the code in a different machine (with Linux Mint 17) that may have to do with thread unavailability.  Therefore, I decided to postpone the current tasks and focus on changing the multithreading implementation in order to correct this kind of unexpected behaviours.

What do I have completed this week?

• Solved the issue with SAGA algorithms that makes the Toolbox crash.
• Refactored Grass/Grass7 and OTB algorithms to work with signals.
• Bug fixes.

What am I going to achieve for the next week?

• Test Grass/Grass7 and OTB algorithms.
• Add an option to cancel the execution of the algorithm and kill the thread.
• Code refactoring.
• Start implementing the runalgIterating.

Is there any blocking issue?

• I had a very busy week due to my university deadlines.
• An issue with my SAGA installation made me think that something was wrong with my implementation and made me lose some time.
• The issue with the SAGA installation also prevented me from testing the algorithms.

What do I have completed this week?

• Solved an issue that makes QGIS crash when the main thread doesn’t wait for the worker thread to start.
• I spent most of the time looking up some tutorials about unit testing on QGIS and trying to configure python paths to run the processing tests.

What am I going to achieve for the next week?

• This week I’ll change back to the multithreading implementation in order to add the cancel option to stop the algorithm execution. I will be working on a mechanism to stop the execution on QGIS algorithm.

Is there any blocking issue?

• This week my mentor proposed to start working on a suite test and I changed the plan to start working on a unit test to ensure the correct behaviour of the multithreading implementation. At the end of the week we discussed about asking the community to test the new implementation and report the issues if there is any.

What do I have completed this week?

• Created a mechanism to hold the result of the algorithm taking into account that there can be several algorithms running at the same time. The results are stored in a global variable called algResults. This variable is accessible via python console (e.g. “processing.algResults”).

• The signals are connected in a different way in order to close the dialog and quit the thread when the algorithm is finished.
• I have updated my fork and merged my work into the master branch.
• Added the runalgIterating method to the AlgorithmExecutor with some changes to support the signals to set the progress
• AlgorithmDialog refactored.
• I have been searching for a solution to stop the algorithm execution and I think that is possible to create a sort of checkpoints inside the algorithm to check if we want to stop the execution and raise an exception to stop the thread event loop.

What am I going to achieve for the next week?

• Allow the user to cancel the execution when running the runalgIterating.
• Solve the problem with QGIS dependencies on Ubuntu and test the refactored algorithms.
• Continue working on the algorithm iterating.

Is there any blocking issue?

• I had to postpone the tests due to some problems installing QGIS dependencies on Ubuntu.
• Didn’t had time to implement a solution to stop the execution of the algorithm.

What do I have completed this week?

• Processing refactored in order to create a non-blocking implementation of the python console. The algorithm is processed in another thread asynchronously and the result is printed to the python console when the execution is finished.
• AlgorithmDialog refactored to create a non-blocking dialog interface with a button to cancel the execution.
• R algorithms refactored to support the progress signal.

What am I going to achieve for the next week?

• Create a mechanism to hold the result of the algorithm taking into account that there can be several algorithms running at the same time.
• Figure out a way to stop the execution of the algorithm. In third-party algorithms it can not be possible to just interrupt the thread loop and quit the thread. The only available option is to discard the result because the algorithms don’t have any kind of check-point that allows to verify once in while if we want to stop the execution. Kill the thread during the execution can be dangerous and it is not a good option.
• Test the refactored algorithms.

Is there any blocking issue?

• It took more time than I expect to create a non-blocking interface.
• Quitting the thread execution turned out to be far more complicated due to the fact that the algorithms don’t provide any mechanism to allow quitting the thread loop during the execution.

What do I have completed this week?

• Solved the issue with the outputs.
• Did some tests with QGIS algorithms which seems to be working fine.
• Refactored all QGIS algorithms in order to remove the progress bar parameter and use the signal instead.
• I have changed the Buffer used by QGIS algorithms: the Buffer is now a subclass of QObject. This new implementation allows the buffer to have a parent (the algorithm) and get access to the algorithm’s signal.
• SagaUtils refactored in the same way I did with the Buffer.
• SAGA algorithms refactored in order to use the signals instead of the progress parameter (Not finished yet)

What am I going to achieve for the next week?

• Solve the issue with SAGA algorithms.
• Code refactoring in order to improve the previous implementation on the AlgorithmDialog.
• Refactor Grass/Grass7.
• Bug fixes.

Is there any blocking issue?

There is an issue with SAGA algorithms that makes the Toolbar crash.

What do I have completed this week?

• During this week I worked on a mechanism to stop the algorithm execution on QGIS algorithms. The approach used to cancel the execution explores python exceptions and QT signals to stop the algorithm from the main thread without having to wait in order to stop the execution and terminate the worker thread. I have used signals to trigger an exception inside the algorithm and stop the execution from the main thread. Since there is a significant amount of algorithms, the mechanism had to be implemented in just two of the QGIS algorithms. This implementation can be further replicated  towards the remaining  QGIS algorithms.

What am I going to achieve for the next week?

• Start working on a test suite for the multithreading to ensure the correct behaviour of the multithreading implementation. The test suite must check if the worker thread is starting, check if signals are triggering the functions when it is supposed to and verify if the cancel option is working properly.

• Figure out how to stop the execution of a non-QGIS algorithm.

Is there any blocking issue?

• No blocking issues