Parallel Process with Python via joblib and multiprocessing.

[Hemmingway]

I am looking to implement parallels processing in one of my python scripts via the joblib and multiprocessing python packages. In this NI Help Document it states For parallel processing you should use VBS scripts, because DIAdem does not currently support Python for user commands. I'm unsure what this means however. Below is an example of how I am trying to implement parallel processing. Any guidance would be appreciated.

from joblib import Parallel, delayed
import multiprocessing
# what are your inputs, and what operation do you want to
# perform on each input. For example...
inputs = range(10)
def processInput(i):
    return i * i
num_cores = multiprocessing.cpu_count()
results = Parallel(n_jobs=num_cores)(delayed(processInput)(i) for i in inputs

[markwkiehl]

DIAdem parallel processes / workers are a DIAdem capability that allows you to run multiple instances of the DIAdem application in parallel, with the 'master' monitoring and controlling the running worker (parallel) instances of DIAdem.  

You are limited to five (5) running parallel processes or workers.  But you can check to see how many are active, and launch a new parallel process with one finishes.  

You can find a VBScript template for running DIAdem parallel processes here:

http://www.savvydiademsolutions.com/script.php?topic=Parallel-Processing-Workers

It would be interesting to see if running a Python script from DIAdem using  joblib and multiprocessing python packages would be successful.  I would say that if the tasks executed by the Python packages are only Python code and do not reference DIAdem capabilities, then you might have a chance of success.  It is doubtful that you would be able to launch and control multiple instances of DIAdem from Python code using  joblib and multiprocessing python packages.  

Hope this helps.  Please post your results / conclusions.  

[gsklyr]

Any progress with this?  My attempts to implement parallel processing in python in DIAdem have not been sucessful so far.  The following simple script just opens a bunch of DIAdem instances until I kill it:

from multiprocessing import Pool
import time

def f(x):
    for i in range(10000):
        for j in range(10000):
            i * j
    return x * x

start = time.time()
with Pool(5) as p:
    print(p.map(f, [1, 2, 3,4,5,6,7,8]))
print("took", time.time()-start, "seconds")

I have attached the example to use workers.

Run 'supervisor.vbs' and look at the timing and active cores

Run 'NOT PARALLEL.vbs' and compare

When attempting to convert this example to python code (mostly by adding 'dd.' in front of stuff) it ran extremely slow and launched workers sequentially and not in parallel.

[Hemmingway]

@Gsklyr I have not made any further progress on this topic and have put the script "on the shelf" for now. I was getting similar behaviors to you with multiple instances of Diadem opening running very very slowly. I also found the following expert  in DIAdem documentation:

Parallel Processing > Objects > Object: ParallelProcessControl

The ParallelProcessControl object provides parallel processing in DIAdem. The ParallelProcessControl object starts more DIAdem applications which run in the background or in the foreground, or you can request the status of a Worker object. You cannot use the ParallelProcessControl object in Analysis Server scripts.

For parallel processing you should use VBS scripts, because DIAdem does not currently support Python for user commands. However, in the OnWorkerStateChange for ParallelProcessControl property, you should define a user command that DIAdem executes when the worker status changes.

I've concluded python parallel processing is not compatible currently with the way the python is "wrapped" in the DIAdem SCRIPT environment