diff --git a/docs/source/parallel/dag_dependencies.txt b/docs/source/parallel/dag_dependencies.txt index 779e2e2..4b6d6ce 100644 --- a/docs/source/parallel/dag_dependencies.txt +++ b/docs/source/parallel/dag_dependencies.txt @@ -10,7 +10,7 @@ for working with Graphs is NetworkX_. Here, we will walk through a demo mapping a nx DAG to task dependencies. The full script that runs this demo can be found in -:file:`docs/examples/newparallel/dagdeps.py`. +:file:`docs/examples/parallel/dagdeps.py`. Why are DAGs good for task dependencies? ---------------------------------------- @@ -30,7 +30,7 @@ A Sample DAG Here, we have a very simple 5-node DAG: -.. figure:: simpledag.* +.. figure:: figs/ simpledag.* With NetworkX, an arrow is just a fattened bit on the edge. Here, we can see that task 0 depends on nothing, and can run immediately. 1 and 2 depend on 0; 3 depends on @@ -80,7 +80,7 @@ The code to generate the simple DAG: For demonstration purposes, we have a function that generates a random DAG with a given number of nodes and edges. -.. literalinclude:: ../../examples/newparallel/dagdeps.py +.. literalinclude:: ../../examples/parallel/dagdeps.py :language: python :lines: 20-36 @@ -137,7 +137,7 @@ These objects store a variety of metadata about each task, including various tim We can validate that the dependencies were respected by checking that each task was started after all of its predecessors were completed: -.. literalinclude:: ../../examples/newparallel/dagdeps.py +.. literalinclude:: ../../examples/parallel/dagdeps.py :language: python :lines: 64-70 @@ -164,7 +164,7 @@ will be at the top, and quick, small tasks will be at the bottom. In [13]: nx.draw(G, pos, node_list=colors.keys(), node_color=colors.values(), ...: cmap=gist_rainbow) -.. figure:: dagdeps.* +.. figure:: figs/ dagdeps.* Time started on x, runtime on y, and color-coded by engine-id (in this case there were four engines). Edges denote dependencies. diff --git a/docs/source/parallel/asian_call.pdf b/docs/source/parallel/figs/asian_call.pdf similarity index 100% rename from docs/source/parallel/asian_call.pdf rename to docs/source/parallel/figs/asian_call.pdf Binary files a/docs/source/parallel/asian_call.pdf and b/docs/source/parallel/figs/asian_call.pdf differ diff --git a/docs/source/parallel/asian_call.png b/docs/source/parallel/figs/asian_call.png similarity index 100% rename from docs/source/parallel/asian_call.png rename to docs/source/parallel/figs/asian_call.png Binary files a/docs/source/parallel/asian_call.png and b/docs/source/parallel/figs/asian_call.png differ diff --git a/docs/source/parallel/asian_put.pdf b/docs/source/parallel/figs/asian_put.pdf similarity index 100% rename from docs/source/parallel/asian_put.pdf rename to docs/source/parallel/figs/asian_put.pdf Binary files a/docs/source/parallel/asian_put.pdf and b/docs/source/parallel/figs/asian_put.pdf differ diff --git a/docs/source/parallel/asian_put.png b/docs/source/parallel/figs/asian_put.png similarity index 100% rename from docs/source/parallel/asian_put.png rename to docs/source/parallel/figs/asian_put.png Binary files a/docs/source/parallel/asian_put.png and b/docs/source/parallel/figs/asian_put.png differ diff --git a/docs/source/parallel/dagdeps.pdf b/docs/source/parallel/figs/dagdeps.pdf similarity index 100% rename from docs/source/parallel/dagdeps.pdf rename to docs/source/parallel/figs/dagdeps.pdf Binary files a/docs/source/parallel/dagdeps.pdf and b/docs/source/parallel/figs/dagdeps.pdf differ diff --git a/docs/source/parallel/dagdeps.png b/docs/source/parallel/figs/dagdeps.png similarity index 100% rename from docs/source/parallel/dagdeps.png rename to docs/source/parallel/figs/dagdeps.png Binary files a/docs/source/parallel/dagdeps.png and b/docs/source/parallel/figs/dagdeps.png differ diff --git a/docs/source/parallel/hpc_job_manager.pdf b/docs/source/parallel/figs/hpc_job_manager.pdf similarity index 100% rename from docs/source/parallel/hpc_job_manager.pdf rename to docs/source/parallel/figs/hpc_job_manager.pdf Binary files a/docs/source/parallel/hpc_job_manager.pdf and b/docs/source/parallel/figs/hpc_job_manager.pdf differ diff --git a/docs/source/parallel/hpc_job_manager.png b/docs/source/parallel/figs/hpc_job_manager.png similarity index 100% rename from docs/source/parallel/hpc_job_manager.png rename to docs/source/parallel/figs/hpc_job_manager.png Binary files a/docs/source/parallel/hpc_job_manager.png and b/docs/source/parallel/figs/hpc_job_manager.png differ diff --git a/docs/source/parallel/ipcluster_create.pdf b/docs/source/parallel/figs/ipcluster_create.pdf similarity index 100% rename from docs/source/parallel/ipcluster_create.pdf rename to docs/source/parallel/figs/ipcluster_create.pdf Binary files a/docs/source/parallel/ipcluster_create.pdf and b/docs/source/parallel/figs/ipcluster_create.pdf differ diff --git a/docs/source/parallel/ipcluster_create.png b/docs/source/parallel/figs/ipcluster_create.png similarity index 100% rename from docs/source/parallel/ipcluster_create.png rename to docs/source/parallel/figs/ipcluster_create.png Binary files a/docs/source/parallel/ipcluster_create.png and b/docs/source/parallel/figs/ipcluster_create.png differ diff --git a/docs/source/parallel/ipcluster_start.pdf b/docs/source/parallel/figs/ipcluster_start.pdf similarity index 100% rename from docs/source/parallel/ipcluster_start.pdf rename to docs/source/parallel/figs/ipcluster_start.pdf Binary files a/docs/source/parallel/ipcluster_start.pdf and b/docs/source/parallel/figs/ipcluster_start.pdf differ diff --git a/docs/source/parallel/ipcluster_start.png b/docs/source/parallel/figs/ipcluster_start.png similarity index 100% rename from docs/source/parallel/ipcluster_start.png rename to docs/source/parallel/figs/ipcluster_start.png Binary files a/docs/source/parallel/ipcluster_start.png and b/docs/source/parallel/figs/ipcluster_start.png differ diff --git a/docs/source/parallel/ipython_shell.pdf b/docs/source/parallel/figs/ipython_shell.pdf similarity index 100% rename from docs/source/parallel/ipython_shell.pdf rename to docs/source/parallel/figs/ipython_shell.pdf Binary files a/docs/source/parallel/ipython_shell.pdf and b/docs/source/parallel/figs/ipython_shell.pdf differ diff --git a/docs/source/parallel/ipython_shell.png b/docs/source/parallel/figs/ipython_shell.png similarity index 100% rename from docs/source/parallel/ipython_shell.png rename to docs/source/parallel/figs/ipython_shell.png Binary files a/docs/source/parallel/ipython_shell.png and b/docs/source/parallel/figs/ipython_shell.png differ diff --git a/docs/source/parallel/mec_simple.pdf b/docs/source/parallel/figs/mec_simple.pdf similarity index 100% rename from docs/source/parallel/mec_simple.pdf rename to docs/source/parallel/figs/mec_simple.pdf Binary files a/docs/source/parallel/mec_simple.pdf and b/docs/source/parallel/figs/mec_simple.pdf differ diff --git a/docs/source/parallel/mec_simple.png b/docs/source/parallel/figs/mec_simple.png similarity index 100% rename from docs/source/parallel/mec_simple.png rename to docs/source/parallel/figs/mec_simple.png Binary files a/docs/source/parallel/mec_simple.png and b/docs/source/parallel/figs/mec_simple.png differ diff --git a/docs/source/parallel/parallel_pi.pdf b/docs/source/parallel/figs/parallel_pi.pdf similarity index 100% rename from docs/source/parallel/parallel_pi.pdf rename to docs/source/parallel/figs/parallel_pi.pdf Binary files a/docs/source/parallel/parallel_pi.pdf and b/docs/source/parallel/figs/parallel_pi.pdf differ diff --git a/docs/source/parallel/parallel_pi.png b/docs/source/parallel/figs/parallel_pi.png similarity index 100% rename from docs/source/parallel/parallel_pi.png rename to docs/source/parallel/figs/parallel_pi.png Binary files a/docs/source/parallel/parallel_pi.png and b/docs/source/parallel/figs/parallel_pi.png differ diff --git a/docs/source/parallel/simpledag.pdf b/docs/source/parallel/figs/simpledag.pdf similarity index 100% rename from docs/source/parallel/simpledag.pdf rename to docs/source/parallel/figs/simpledag.pdf Binary files a/docs/source/parallel/simpledag.pdf and b/docs/source/parallel/figs/simpledag.pdf differ diff --git a/docs/source/parallel/simpledag.png b/docs/source/parallel/figs/simpledag.png similarity index 100% rename from docs/source/parallel/simpledag.png rename to docs/source/parallel/figs/simpledag.png Binary files a/docs/source/parallel/simpledag.png and b/docs/source/parallel/figs/simpledag.png differ diff --git a/docs/source/parallel/single_digits.pdf b/docs/source/parallel/figs/single_digits.pdf similarity index 100% rename from docs/source/parallel/single_digits.pdf rename to docs/source/parallel/figs/single_digits.pdf Binary files a/docs/source/parallel/single_digits.pdf and b/docs/source/parallel/figs/single_digits.pdf differ diff --git a/docs/source/parallel/single_digits.png b/docs/source/parallel/figs/single_digits.png similarity index 100% rename from docs/source/parallel/single_digits.png rename to docs/source/parallel/figs/single_digits.png Binary files a/docs/source/parallel/single_digits.png and b/docs/source/parallel/figs/single_digits.png differ diff --git a/docs/source/parallel/two_digit_counts.pdf b/docs/source/parallel/figs/two_digit_counts.pdf similarity index 100% rename from docs/source/parallel/two_digit_counts.pdf rename to docs/source/parallel/figs/two_digit_counts.pdf Binary files a/docs/source/parallel/two_digit_counts.pdf and b/docs/source/parallel/figs/two_digit_counts.pdf differ diff --git a/docs/source/parallel/two_digit_counts.png b/docs/source/parallel/figs/two_digit_counts.png similarity index 100% rename from docs/source/parallel/two_digit_counts.png rename to docs/source/parallel/figs/two_digit_counts.png Binary files a/docs/source/parallel/two_digit_counts.png and b/docs/source/parallel/figs/two_digit_counts.png differ diff --git a/docs/source/parallel/parallel_demos.txt b/docs/source/parallel/parallel_demos.txt index 08b09a9..fd2edee 100644 --- a/docs/source/parallel/parallel_demos.txt +++ b/docs/source/parallel/parallel_demos.txt @@ -4,7 +4,7 @@ Parallel examples .. note:: - Performance numbers from ``IPython.kernel``, not newparallel. + Performance numbers from ``IPython.kernel``, not new ``IPython.parallel``. In this section we describe two more involved examples of using an IPython cluster to perform a parallel computation. In these examples, we will be using @@ -27,7 +27,7 @@ million digits. In both the serial and parallel calculation we will be using functions defined in the :file:`pidigits.py` file, which is available in the -:file:`docs/examples/newparallel` directory of the IPython source distribution. +:file:`docs/examples/parallel` directory of the IPython source distribution. These functions provide basic facilities for working with the digits of pi and can be loaded into IPython by putting :file:`pidigits.py` in your current working directory and then doing: @@ -75,7 +75,7 @@ The resulting plot of the single digit counts shows that each digit occurs approximately 1,000 times, but that with only 10,000 digits the statistical fluctuations are still rather large: -.. image:: single_digits.* +.. image:: figs/single_digits.* It is clear that to reduce the relative fluctuations in the counts, we need to look at many more digits of pi. That brings us to the parallel calculation. @@ -101,13 +101,13 @@ compute the two digit counts for the digits in a single file. Then in a final step the counts from each engine will be added up. To perform this calculation, we will need two top-level functions from :file:`pidigits.py`: -.. literalinclude:: ../../examples/newparallel/pidigits.py +.. literalinclude:: ../../examples/parallel/pi/pidigits.py :language: python :lines: 47-62 We will also use the :func:`plot_two_digit_freqs` function to plot the results. The code to run this calculation in parallel is contained in -:file:`docs/examples/newparallel/parallelpi.py`. This code can be run in parallel +:file:`docs/examples/parallel/parallelpi.py`. This code can be run in parallel using IPython by following these steps: 1. Use :command:`ipcluster` to start 15 engines. We used an 8 core (2 quad @@ -188,7 +188,7 @@ most likely and that "06" and "07" are least likely. Further analysis would show that the relative size of the statistical fluctuations have decreased compared to the 10,000 digit calculation. -.. image:: two_digit_counts.* +.. image:: figs/two_digit_counts.* Parallel options pricing @@ -209,12 +209,12 @@ simulation of the underlying asset price. In this example we use this approach to price both European and Asian (path dependent) options for various strike prices and volatilities. -The code for this example can be found in the :file:`docs/examples/newparallel` +The code for this example can be found in the :file:`docs/examples/parallel` directory of the IPython source. The function :func:`price_options` in :file:`mcpricer.py` implements the basic Monte Carlo pricing algorithm using the NumPy package and is shown here: -.. literalinclude:: ../../examples/newparallel/mcpricer.py +.. literalinclude:: ../../examples/parallel/options/mcpricer.py :language: python To run this code in parallel, we will use IPython's :class:`LoadBalancedView` class, @@ -222,21 +222,21 @@ which distributes work to the engines using dynamic load balancing. This view is a wrapper of the :class:`Client` class shown in the previous example. The parallel calculation using :class:`LoadBalancedView` can be found in the file :file:`mcpricer.py`. The code in this file creates a -:class:`TaskClient` instance and then submits a set of tasks using -:meth:`TaskClient.run` that calculate the option prices for different +:class:`LoadBalancedView` instance and then submits a set of tasks using +:meth:`LoadBalancedView.apply` that calculate the option prices for different volatilities and strike prices. The results are then plotted as a 2D contour plot using Matplotlib. -.. literalinclude:: ../../examples/newparallel/mcdriver.py +.. literalinclude:: ../../examples/parallel/options/mckernel.py :language: python To use this code, start an IPython cluster using :command:`ipcluster`, open -IPython in the pylab mode with the file :file:`mcdriver.py` in your current +IPython in the pylab mode with the file :file:`mckernel.py` in your current working directory and then type: .. sourcecode:: ipython - In [7]: run mcdriver.py + In [7]: run mckernel.py Submitted tasks: [0, 1, 2, ...] Once all the tasks have finished, the results can be plotted using the @@ -257,9 +257,9 @@ entire calculation (10 strike prices, 10 volatilities, 100,000 paths for each) took 30 seconds in parallel, giving a speedup of 7.7x, which is comparable to the speedup observed in our previous example. -.. image:: asian_call.* +.. image:: figs/asian_call.* -.. image:: asian_put.* +.. image:: figs/asian_put.* Conclusion ========== @@ -280,5 +280,5 @@ parallel architecture that have been demonstrated: .. note:: - The newparallel code has never been run on Windows HPC Server, so the last + The new parallel code has never been run on Windows HPC Server, so the last conclusion is untested. diff --git a/docs/source/parallel/parallel_task.txt b/docs/source/parallel/parallel_task.txt index 1e13c0b..d338a6f 100644 --- a/docs/source/parallel/parallel_task.txt +++ b/docs/source/parallel/parallel_task.txt @@ -87,12 +87,12 @@ To load-balance :meth:`map`,simply use a LoadBalancedView: In [62]: lview.block = True - In [63]: serial_result = map(lambda x:x**10, range(32)) + In [63]: serial_result = map(lambda x:x**10, range(32)) - In [64]: parallel_result = lview.map(lambda x:x**10, range(32)) + In [64]: parallel_result = lview.map(lambda x:x**10, range(32)) - In [65]: serial_result==parallel_result - Out[65]: True + In [65]: serial_result==parallel_result + Out[65]: True Parallel function decorator --------------------------- @@ -111,6 +111,15 @@ that turns any Python function into a parallel function: In [11]: f.map(range(32)) # this is done in parallel Out[11]: [0.0,10.0,160.0,...] +.. _parallel_taskmap: + +The AsyncMapResult +================== + +When you call ``lview.map_async(f, sequence)``, or just :meth:`map` with `block=True`, then +what you get in return will be an :class:`~AsyncMapResult` object. These are similar to +AsyncResult objects, but with one key difference + .. _parallel_dependencies: Dependencies @@ -291,8 +300,6 @@ you can skip using Dependency objects, and just pass msg_ids or AsyncResult obje onto task dependencies. - - Impossible Dependencies *********************** diff --git a/docs/source/parallel/parallel_winhpc.txt b/docs/source/parallel/parallel_winhpc.txt index 83dedf7..a19a9eb 100644 --- a/docs/source/parallel/parallel_winhpc.txt +++ b/docs/source/parallel/parallel_winhpc.txt @@ -120,7 +120,7 @@ opening a Windows Command Prompt and typing ``ipython``. This will start IPython's interactive shell and you should see something like the following screenshot: -.. image:: ipython_shell.* +.. image:: figs/ipython_shell.* Starting an IPython cluster =========================== @@ -168,7 +168,7 @@ You should see a number of messages printed to the screen, ending with "IPython cluster: started". The result should look something like the following screenshot: -.. image:: ipcluster_start.* +.. image:: figs/ipcluster_start.* At this point, the controller and two engines are running on your local host. This configuration is useful for testing and for situations where you want to @@ -210,7 +210,7 @@ The output of this command is shown in the screenshot below. Notice how :command:`ipcluster` prints out the location of the newly created cluster directory. -.. image:: ipcluster_create.* +.. image:: figs/ipcluster_create.* Configuring a cluster profile ----------------------------- @@ -279,7 +279,7 @@ must be run again to regenerate the XML job description files. The following screenshot shows what the HPC Job Manager interface looks like with a running IPython cluster. -.. image:: hpc_job_manager.* +.. image:: figs/hpc_job_manager.* Performing a simple interactive parallel computation ==================================================== @@ -330,5 +330,5 @@ The :meth:`map` method has the same signature as Python's builtin :func:`map` function, but runs the calculation in parallel. More involved examples of using :class:`MultiEngineClient` are provided in the examples that follow. -.. image:: mec_simple.* +.. image:: figs/mec_simple.*