upstream/ipython Files · docs/examples/kernel/pwordfreq_skel.py

The refactoring of the Task system is nearly complete. Now there are...

The refactoring of the Task system is nearly complete. Now there are multiple types of tasks including `StringTask` and `MapTask`. Each task type is responsible for running itself and processing its own result. This makes it much easier for people to create new task types. Also, the map and parallel function support has been completely refactored and improved. This includes a map and parallel function implementation for the task controller as well as a @parallel decorator.

Brian E Granger - - Load All Authors

File last commit:

r1338:72652d65


                r1395:1feaf0a3

Download file

             pwordfreq_skel.py
        
                    35 lines
            
             | 982 B
            
                | text/x-python
            
             |
                PythonLexer
            
             / docs / examples / kernel / pwordfreq_skel.py
          
                    History
                
                 |
                  Annotation
                 | Raw
                 |Copy content
                 |Copy permalink

      #!/usr/bin/env python

      """Parallel word frequency counter."""

      from itertools import repeat

      from wordfreq import print_wordfreq, wordfreq

      def pwordfreq(rc, text):

          """Parallel word frequency counter.

          rc - An IPython RemoteController

          text - The name of a string on the engines to do the freq count on.

          """

      if __name__ == '__main__':

          # Create a MultiEngineClient

          from IPython.kernel import client

          ipc = client.MultiEngineClient()

          # Run the wordfreq script on the engines.

          ipc.run('wordfreq.py')

          # Run the serial version

          print "Serial word frequency count:"

          text = open('davinci.txt').read()

          freqs = wordfreq(text)

          print_wordfreq(freqs, 10)

          # The parallel version

          print "\nParallel word frequency count:"

          files = ['davinci%i.txt' % i for i in range(4)]

          ipc.scatter('textfile', files)

          ipc.execute('text = open(textfile[0]).read()')

          pfreqs = pwordfreq(ipc,'text')

          print_wordfreq(freqs)

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages

				#!/usr/bin/env python
				"""Parallel word frequency counter."""


				from itertools import repeat
				from wordfreq import print_wordfreq, wordfreq

				def pwordfreq(rc, text):
				"""Parallel word frequency counter.

				rc - An IPython RemoteController
				text - The name of a string on the engines to do the freq count on.
				"""

				if __name__ == '__main__':
				# Create a MultiEngineClient
				from IPython.kernel import client
				ipc = client.MultiEngineClient()

				# Run the wordfreq script on the engines.
				ipc.run('wordfreq.py')

				# Run the serial version
				print "Serial word frequency count:"
				text = open('davinci.txt').read()
				freqs = wordfreq(text)
				print_wordfreq(freqs, 10)

				# The parallel version
				print "\nParallel word frequency count:"
				files = ['davinci%i.txt' % i for i in range(4)]
				ipc.scatter('textfile', files)
				ipc.execute('text = open(textfile[0]).read()')
				pfreqs = pwordfreq(ipc,'text')
				print_wordfreq(freqs)