Merging upstream changes from trunk->module-reorg->inputhook.

IPython/core/release.py

@@ -1,121 +1,121 b''
 # -*- coding: utf-8 -*-
 """Release data for the IPython project."""
 
 #*****************************************************************************
 #       Copyright (C) 2008-2009  The IPython Development Team
 #       Copyright (C) 2001-2008 Fernando Perez <[e-mail]>
 #       Copyright (c) 2001 Janko Hauser <[e-mail]> and Nathaniel Gray
 #       <[e-mail]>
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
 #*****************************************************************************
 
 # Name of the package for release purposes.  This is the name which labels
 # the tarballs and RPMs made by distutils, so it's best to lowercase it.
 name = 'ipython'
 
 # For versions with substrings (like 0.6.16.svn), use an extra . to separate
 # the new substring.  We have to avoid using either dashes or underscores,
 # because bdist_rpm does not accept dashes (an RPM) convention, and
 # bdist_deb does not accept underscores (a Debian convention).
 
-development = False    # change this to False to do a release
-version_base = '0.10'
+development = True    # change this to False to do a release
+version_base = '0.11'
 branch = 'ipython'
-revision = '1188'
+revision = '1205'
 
 if development:
     if branch == 'ipython':
         version = '%s.bzr.r%s' % (version_base, revision)
     else:
         version = '%s.bzr.r%s.%s' % (version_base, revision, branch)
 else:
     version = version_base
 
 
 description = "An interactive computing environment for Python"
 
 long_description = \
 """
 The goal of IPython is to create a comprehensive environment for
 interactive and exploratory computing.  To support this goal, IPython
 has two main components:
 
 * An enhanced interactive Python shell.
 
 * An architecture for interactive parallel computing.
 
 The enhanced interactive Python shell has the following main features:
 
 * Comprehensive object introspection.
 
 * Input history, persistent across sessions.
 
 * Caching of output results during a session with automatically generated
   references.
 
 * Readline based name completion.
 
 * Extensible system of 'magic' commands for controlling the environment and
   performing many tasks related either to IPython or the operating system.
 
 * Configuration system with easy switching between different setups (simpler
   than changing $PYTHONSTARTUP environment variables every time).
 
 * Session logging and reloading.
 
 * Extensible syntax processing for special purpose situations.
 
 * Access to the system shell with user-extensible alias system.
 
 * Easily embeddable in other Python programs and wxPython GUIs.
 
 * Integrated access to the pdb debugger and the Python profiler. 
 
 The parallel computing architecture has the following main features:
 
 * Quickly parallelize Python code from an interactive Python/IPython session.
 
 * A flexible and dynamic process model that be deployed on anything from 
   multicore workstations to supercomputers.
 
 * An architecture that supports many different styles of parallelism, from
   message passing to task farming.
 
 * Both blocking and fully asynchronous interfaces.
 
 * High level APIs that enable many things to be parallelized in a few lines
   of code.
 
 * Share live parallel jobs with other users securely.  
 
 * Dynamically load balanced task farming system.
 
 * Robust error handling in parallel code.
 
 The latest development version is always available from IPython's `Launchpad 
 site <http://launchpad.net/ipython>`_.
 """
 
 license = 'BSD'
 
 authors = {'Fernando' : ('Fernando Perez','[e-mail]'),
            'Janko'    : ('Janko Hauser','[e-mail]'),
            'Nathan'   : ('Nathaniel Gray','[e-mail]'),
            'Ville'    : ('Ville Vainio','[e-mail]'),
            'Brian'    : ('Brian E Granger', '[e-mail]'),
            'Min'      : ('Min Ragan-Kelley', '[e-mail]')
            }
 
 author = 'The IPython Development Team'
 
 author_email = '[e-mail]'
 
 url = 'http://ipython.scipy.org'
 
 download_url = 'http://ipython.scipy.org/dist'
 
 platforms = ['Linux','Mac OSX','Windows XP/2000/NT','Windows 95/98/ME']
 
 keywords = ['Interactive','Interpreter','Shell','Parallel','Distributed']

IPython/external/argparse.py

@@ -1,1867 +1,2216 b''
 # -*- coding: utf-8 -*-
 
-# Copyright � 2006 Steven J. Bethard <[e-mail]>.
+# Copyright © 2006-2009 Steven J. Bethard <[e-mail]>.
 #
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted under the terms of the 3-clause BSD
-# license. No warranty expressed or implied.
-# For details, see the accompanying file LICENSE.txt.
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy
+# of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
 
 """Command-line parsing library
 
 This module is an optparse-inspired command-line parsing library that:
 
-* handles both optional and positional arguments
-* produces highly informative usage messages
-* supports parsers that dispatch to sub-parsers
+    - handles both optional and positional arguments
+    - produces highly informative usage messages
+    - supports parsers that dispatch to sub-parsers
 
 The following is a simple usage example that sums integers from the
-command-line and writes the result to a file:
+command-line and writes the result to a file::
 
     parser = argparse.ArgumentParser(
         description='sum the integers at the command line')
     parser.add_argument(
         'integers', metavar='int', nargs='+', type=int,
         help='an integer to be summed')
     parser.add_argument(
         '--log', default=sys.stdout, type=argparse.FileType('w'),
         help='the file where the sum should be written')
     args = parser.parse_args()
     args.log.write('%s' % sum(args.integers))
     args.log.close()
 
 The module contains the following public classes:
 
-    ArgumentParser -- The main entry point for command-line parsing. As the
+    - ArgumentParser -- The main entry point for command-line parsing. As the
         example above shows, the add_argument() method is used to populate
         the parser with actions for optional and positional arguments. Then
         the parse_args() method is invoked to convert the args at the
         command-line into an object with attributes.
 
-    ArgumentError -- The exception raised by ArgumentParser objects when
+    - ArgumentError -- The exception raised by ArgumentParser objects when
         there are errors with the parser's actions. Errors raised while
         parsing the command-line are caught by ArgumentParser and emitted
         as command-line messages.
 
-    FileType -- A factory for defining types of files to be created. As the
+    - FileType -- A factory for defining types of files to be created. As the
         example above shows, instances of FileType are typically passed as
         the type= argument of add_argument() calls.
 
-    Action -- The base class for parser actions. Typically actions are
+    - Action -- The base class for parser actions. Typically actions are
         selected by passing strings like 'store_true' or 'append_const' to
         the action= argument of add_argument(). However, for greater
         customization of ArgumentParser actions, subclasses of Action may
         be defined and passed as the action= argument.
 
-    HelpFormatter, RawDescriptionHelpFormatter -- Formatter classes which
+    - HelpFormatter, RawDescriptionHelpFormatter, RawTextHelpFormatter,
+        ArgumentDefaultsHelpFormatter -- Formatter classes which
         may be passed as the formatter_class= argument to the
-        ArgumentParser constructor. HelpFormatter is the default, while
-        RawDescriptionHelpFormatter tells the parser not to perform any
-        line-wrapping on description text.
+        ArgumentParser constructor. HelpFormatter is the default,
+        RawDescriptionHelpFormatter and RawTextHelpFormatter tell the parser
+        not to change the formatting for help text, and
+        ArgumentDefaultsHelpFormatter adds information about argument defaults
+        to the help.
 
 All other classes in this module are considered implementation details.
 (Also note that HelpFormatter and RawDescriptionHelpFormatter are only
 considered public as object names -- the API of the formatter objects is
 still considered an implementation detail.)
 """
 
-__version__ = '0.8.0'
-
+__version__ = '1.0'
+__all__ = [
+    'ArgumentParser',
+    'ArgumentError',
+    'Namespace',
+    'Action',
+    'FileType',
+    'HelpFormatter',
+    'RawDescriptionHelpFormatter',
+    'RawTextHelpFormatter'
+    'ArgumentDefaultsHelpFormatter',
+]
+
+
+import copy as _copy
 import os as _os
 import re as _re
 import sys as _sys
 import textwrap as _textwrap
 
 from gettext import gettext as _
 
+try:
+    _set = set
+except NameError:
+    from sets import Set as _set
+
+try:
+    _basestring = basestring
+except NameError:
+    _basestring = str
+
+try:
+    _sorted = sorted
+except NameError:
+
+    def _sorted(iterable, reverse=False):
+        result = list(iterable)
+        result.sort()
+        if reverse:
+            result.reverse()
+        return result
+
+
 SUPPRESS = '==SUPPRESS=='
 
 OPTIONAL = '?'
 ZERO_OR_MORE = '*'
 ONE_OR_MORE = '+'
 PARSER = '==PARSER=='
 
 # =============================
 # Utility functions and classes
 # =============================
 
 class _AttributeHolder(object):
     """Abstract base class that provides __repr__.
 
-    The __repr__ method returns a string in the format:
+    The __repr__ method returns a string in the format::
         ClassName(attr=name, attr=name, ...)
     The attributes are determined either by a class-level attribute,
     '_kwarg_names', or by inspecting the instance __dict__.
     """
-    
+
     def __repr__(self):
         type_name = type(self).__name__
         arg_strings = []
         for arg in self._get_args():
             arg_strings.append(repr(arg))
         for name, value in self._get_kwargs():
             arg_strings.append('%s=%r' % (name, value))
         return '%s(%s)' % (type_name, ', '.join(arg_strings))
 
     def _get_kwargs(self):
-        return sorted(self.__dict__.items())
+        return _sorted(self.__dict__.items())
 
     def _get_args(self):
         return []
 
+
 def _ensure_value(namespace, name, value):
     if getattr(namespace, name, None) is None:
         setattr(namespace, name, value)
     return getattr(namespace, name)
-    
 
 
 # ===============
 # Formatting Help
 # ===============
 
 class HelpFormatter(object):
+    """Formatter for generating usage messages and argument help strings.
+
+    Only the name of this class is considered a public API. All the methods
+    provided by the class are considered an implementation detail.
+    """
 
     def __init__(self,
                  prog,
                  indent_increment=2,
                  max_help_position=24,
                  width=None):
 
         # default setting for width
         if width is None:
             try:
                 width = int(_os.environ['COLUMNS'])
             except (KeyError, ValueError):
                 width = 80
             width -= 2
-        
+
         self._prog = prog
         self._indent_increment = indent_increment
         self._max_help_position = max_help_position
         self._width = width
 
         self._current_indent = 0
         self._level = 0
         self._action_max_length = 0
 
         self._root_section = self._Section(self, None)
         self._current_section = self._root_section
 
         self._whitespace_matcher = _re.compile(r'\s+')
         self._long_break_matcher = _re.compile(r'\n\n\n+')
 
     # ===============================
     # Section and indentation methods
     # ===============================
-
     def _indent(self):
         self._current_indent += self._indent_increment
         self._level += 1
 
     def _dedent(self):
         self._current_indent -= self._indent_increment
         assert self._current_indent >= 0, 'Indent decreased below 0.'
         self._level -= 1
 
     class _Section(object):
+
         def __init__(self, formatter, parent, heading=None):
             self.formatter = formatter
             self.parent = parent
             self.heading = heading
             self.items = []
 
         def format_help(self):
             # format the indented section
             if self.parent is not None:
                 self.formatter._indent()
             join = self.formatter._join_parts
             for func, args in self.items:
                 func(*args)
-            item_help = join(func(*args) for func, args in self.items)
+            item_help = join([func(*args) for func, args in self.items])
             if self.parent is not None:
                 self.formatter._dedent()
 
-            # return nothing if the section was empty            
+            # return nothing if the section was empty
             if not item_help:
                 return ''
 
             # add the heading if the section was non-empty
             if self.heading is not SUPPRESS and self.heading is not None:
                 current_indent = self.formatter._current_indent
                 heading = '%*s%s:\n' % (current_indent, '', self.heading)
             else:
                 heading = ''
 
             # join the section-initial newline, the heading and the help
             return join(['\n', heading, item_help, '\n'])
 
     def _add_item(self, func, args):
         self._current_section.items.append((func, args))
 
     # ========================
     # Message building methods
     # ========================
-
     def start_section(self, heading):
         self._indent()
         section = self._Section(self, self._current_section, heading)
         self._add_item(section.format_help, [])
         self._current_section = section
 
     def end_section(self):
         self._current_section = self._current_section.parent
         self._dedent()
 
     def add_text(self, text):
         if text is not SUPPRESS and text is not None:
             self._add_item(self._format_text, [text])
 
-    def add_usage(self, usage, optionals, positionals, prefix=None):
+    def add_usage(self, usage, actions, groups, prefix=None):
         if usage is not SUPPRESS:
-            args = usage, optionals, positionals, prefix
+            args = usage, actions, groups, prefix
             self._add_item(self._format_usage, args)
 
     def add_argument(self, action):
         if action.help is not SUPPRESS:
 
-            # find all invocations            
+            # find all invocations
             get_invocation = self._format_action_invocation
             invocations = [get_invocation(action)]
             for subaction in self._iter_indented_subactions(action):
                 invocations.append(get_invocation(subaction))
-                    
+
             # update the maximum item length
-            invocation_length = max(len(s) for s in invocations)
+            invocation_length = max([len(s) for s in invocations])
             action_length = invocation_length + self._current_indent
             self._action_max_length = max(self._action_max_length,
                                           action_length)
 
-            # add the item to the list        
+            # add the item to the list
             self._add_item(self._format_action, [action])
 
     def add_arguments(self, actions):
         for action in actions:
             self.add_argument(action)
 
     # =======================
     # Help-formatting methods
     # =======================
-
     def format_help(self):
         help = self._root_section.format_help() % dict(prog=self._prog)
         if help:
             help = self._long_break_matcher.sub('\n\n', help)
             help = help.strip('\n') + '\n'
         return help
 
     def _join_parts(self, part_strings):
-        return ''.join(part
-                       for part in part_strings
-                       if part and part is not SUPPRESS)
+        return ''.join([part
+                        for part in part_strings
+                        if part and part is not SUPPRESS])
 
-    def _format_usage(self, usage, optionals, positionals, prefix):
+    def _format_usage(self, usage, actions, groups, prefix):
         if prefix is None:
             prefix = _('usage: ')
 
         # if no optionals or positionals are available, usage is just prog
-        if usage is None and not optionals and not positionals:
+        if usage is None and not actions:
             usage = '%(prog)s'
-            
+
         # if optionals and positionals are available, calculate usage
         elif usage is None:
             usage = '%(prog)s' % dict(prog=self._prog)
-                
+
+            # split optionals from positionals
+            optionals = []
+            positionals = []
+            for action in actions:
+                if action.option_strings:
+                    optionals.append(action)
+                else:
+                    positionals.append(action)
+
             # determine width of "usage: PROG" and width of text
             prefix_width = len(prefix) + len(usage) + 1
             prefix_indent = self._current_indent + prefix_width
             text_width = self._width - self._current_indent
 
             # put them on one line if they're short enough
             format = self._format_actions_usage
-            action_usage = format(optionals + positionals)
+            action_usage = format(optionals + positionals, groups)
             if prefix_width + len(action_usage) + 1 < text_width:
                 usage = '%s %s' % (usage, action_usage)
-            
+
             # if they're long, wrap optionals and positionals individually
             else:
-                optional_usage = format(optionals)
-                positional_usage = format(positionals)
+                optional_usage = format(optionals, groups)
+                positional_usage = format(positionals, groups)
                 indent = ' ' * prefix_indent
 
                 # usage is made of PROG, optionals and positionals
                 parts = [usage, ' ']
-                
+
                 # options always get added right after PROG
                 if optional_usage:
                     parts.append(_textwrap.fill(
                         optional_usage, text_width,
                         initial_indent=indent,
                         subsequent_indent=indent).lstrip())
 
                 # if there were options, put arguments on the next line
                 # otherwise, start them right after PROG
                 if positional_usage:
                     part = _textwrap.fill(
                         positional_usage, text_width,
                         initial_indent=indent,
                         subsequent_indent=indent).lstrip()
                     if optional_usage:
                         part = '\n' + indent + part
                     parts.append(part)
                 usage = ''.join(parts)
 
         # prefix with 'usage:'
         return '%s%s\n\n' % (prefix, usage)
 
-    def _format_actions_usage(self, actions):
-        parts = []
-        for action in actions:
-            if action.help is SUPPRESS:
+    def _format_actions_usage(self, actions, groups):
+        # find group indices and identify actions in groups
+        group_actions = _set()
+        inserts = {}
+        for group in groups:
+            try:
+                start = actions.index(group._group_actions[0])
+            except ValueError:
                 continue
+            else:
+                end = start + len(group._group_actions)
+                if actions[start:end] == group._group_actions:
+                    for action in group._group_actions:
+                        group_actions.add(action)
+                    if not group.required:
+                        inserts[start] = '['
+                        inserts[end] = ']'
+                    else:
+                        inserts[start] = '('
+                        inserts[end] = ')'
+                    for i in range(start + 1, end):
+                        inserts[i] = '|'
 
-            # produce all arg strings        
-            if not action.option_strings:
-                parts.append(self._format_args(action, action.dest))
+        # collect all actions format strings
+        parts = []
+        for i, action in enumerate(actions):
+
+            # suppressed arguments are marked with None
+            # remove | separators for suppressed arguments
+            if action.help is SUPPRESS:
+                parts.append(None)
+                if inserts.get(i) == '|':
+                    inserts.pop(i)
+                elif inserts.get(i + 1) == '|':
+                    inserts.pop(i + 1)
+
+            # produce all arg strings
+            elif not action.option_strings:
+                part = self._format_args(action, action.dest)
+
+                # if it's in a group, strip the outer []
+                if action in group_actions:
+                    if part[0] == '[' and part[-1] == ']':
+                        part = part[1:-1]
+
+                # add the action string to the list
+                parts.append(part)
 
             # produce the first way to invoke the option in brackets
             else:
                 option_string = action.option_strings[0]
 
                 # if the Optional doesn't take a value, format is:
                 #    -s or --long
                 if action.nargs == 0:
                     part = '%s' % option_string
 
                 # if the Optional takes a value, format is:
                 #    -s ARGS or --long ARGS
                 else:
                     default = action.dest.upper()
                     args_string = self._format_args(action, default)
                     part = '%s %s' % (option_string, args_string)
 
-                # make it look optional if it's not required
-                if not action.required:
+                # make it look optional if it's not required or in a group
+                if not action.required and action not in group_actions:
                     part = '[%s]' % part
+
+                # add the action string to the list
                 parts.append(part)
 
-        return ' '.join(parts)
+        # insert things at the necessary indices
+        for i in _sorted(inserts, reverse=True):
+            parts[i:i] = [inserts[i]]
+
+        # join all the action items with spaces
+        text = ' '.join([item for item in parts if item is not None])
+
+        # clean up separators for mutually exclusive groups
+        open = r'[\[(]'
+        close = r'[\])]'
+        text = _re.sub(r'(%s) ' % open, r'\1', text)
+        text = _re.sub(r' (%s)' % close, r'\1', text)
+        text = _re.sub(r'%s *%s' % (open, close), r'', text)
+        text = _re.sub(r'\(([^|]*)\)', r'\1', text)
+        text = text.strip()
+
+        # return the text
+        return text
 
     def _format_text(self, text):
         text_width = self._width - self._current_indent
         indent = ' ' * self._current_indent
         return self._fill_text(text, text_width, indent) + '\n\n'
 
     def _format_action(self, action):
         # determine the required width and the entry label
         help_position = min(self._action_max_length + 2,
                             self._max_help_position)
         help_width = self._width - help_position
         action_width = help_position - self._current_indent - 2
         action_header = self._format_action_invocation(action)
 
         # ho nelp; start on same line and add a final newline
         if not action.help:
             tup = self._current_indent, '', action_header
             action_header = '%*s%s\n' % tup
-            
+
         # short action name; start on the same line and pad two spaces
         elif len(action_header) <= action_width:
             tup = self._current_indent, '', action_width, action_header
             action_header = '%*s%-*s  ' % tup
             indent_first = 0
 
         # long action name; start on the next line
         else:
             tup = self._current_indent, '', action_header
             action_header = '%*s%s\n' % tup
             indent_first = help_position
 
         # collect the pieces of the action help
         parts = [action_header]
 
         # if there was help for the action, add lines of help text
         if action.help:
             help_text = self._expand_help(action)
             help_lines = self._split_lines(help_text, help_width)
             parts.append('%*s%s\n' % (indent_first, '', help_lines[0]))
             for line in help_lines[1:]:
                 parts.append('%*s%s\n' % (help_position, '', line))
 
         # or add a newline if the description doesn't end with one
         elif not action_header.endswith('\n'):
             parts.append('\n')
 
         # if there are any sub-actions, add their help as well
         for subaction in self._iter_indented_subactions(action):
             parts.append(self._format_action(subaction))
 
-        # return a single string            
+        # return a single string
         return self._join_parts(parts)
 
     def _format_action_invocation(self, action):
         if not action.option_strings:
-            return self._format_metavar(action, action.dest)
+            metavar, = self._metavar_formatter(action, action.dest)(1)
+            return metavar
 
         else:
             parts = []
 
             # if the Optional doesn't take a value, format is:
             #    -s, --long
             if action.nargs == 0:
                 parts.extend(action.option_strings)
 
             # if the Optional takes a value, format is:
             #    -s ARGS, --long ARGS
             else:
                 default = action.dest.upper()
                 args_string = self._format_args(action, default)
                 for option_string in action.option_strings:
                     parts.append('%s %s' % (option_string, args_string))
 
             return ', '.join(parts)
 
-    def _format_metavar(self, action, default_metavar):
+    def _metavar_formatter(self, action, default_metavar):
         if action.metavar is not None:
-            name = action.metavar
+            result = action.metavar
         elif action.choices is not None:
-            choice_strs = (str(choice) for choice in action.choices)
-            name = '{%s}' % ','.join(choice_strs)
+            choice_strs = [str(choice) for choice in action.choices]
+            result = '{%s}' % ','.join(choice_strs)
         else:
-            name = default_metavar
-        return name
+            result = default_metavar
+
+        def format(tuple_size):
+            if isinstance(result, tuple):
+                return result
+            else:
+                return (result, ) * tuple_size
+        return format
 
     def _format_args(self, action, default_metavar):
-        name = self._format_metavar(action, default_metavar)
+        get_metavar = self._metavar_formatter(action, default_metavar)
         if action.nargs is None:
-            result = name
+            result = '%s' % get_metavar(1)
         elif action.nargs == OPTIONAL:
-            result = '[%s]' % name
+            result = '[%s]' % get_metavar(1)
         elif action.nargs == ZERO_OR_MORE:
-            result = '[%s [%s ...]]' % (name, name)
+            result = '[%s [%s ...]]' % get_metavar(2)
         elif action.nargs == ONE_OR_MORE:
-            result = '%s [%s ...]' % (name, name)
+            result = '%s [%s ...]' % get_metavar(2)
         elif action.nargs is PARSER:
-            result = '%s ...' % name
+            result = '%s ...' % get_metavar(1)
         else:
-            result = ' '.join([name] * action.nargs)
+            formats = ['%s' for _ in range(action.nargs)]
+            result = ' '.join(formats) % get_metavar(action.nargs)
         return result
 
     def _expand_help(self, action):
         params = dict(vars(action), prog=self._prog)
-        for name, value in params.items():
-            if value is SUPPRESS:
+        for name in list(params):
+            if params[name] is SUPPRESS:
                 del params[name]
         if params.get('choices') is not None:
-            choices_str = ', '.join(str(c) for c in params['choices'])
+            choices_str = ', '.join([str(c) for c in params['choices']])
             params['choices'] = choices_str
-        return action.help % params
+        return self._get_help_string(action) % params
 
     def _iter_indented_subactions(self, action):
         try:
             get_subactions = action._get_subactions
         except AttributeError:
             pass
         else:
             self._indent()
             for subaction in get_subactions():
                 yield subaction
             self._dedent()
 
     def _split_lines(self, text, width):
         text = self._whitespace_matcher.sub(' ', text).strip()
         return _textwrap.wrap(text, width)
 
     def _fill_text(self, text, width, indent):
         text = self._whitespace_matcher.sub(' ', text).strip()
         return _textwrap.fill(text, width, initial_indent=indent,
                                            subsequent_indent=indent)
 
+    def _get_help_string(self, action):
+        return action.help
+
+
 class RawDescriptionHelpFormatter(HelpFormatter):
-    
+    """Help message formatter which retains any formatting in descriptions.
+
+    Only the name of this class is considered a public API. All the methods
+    provided by the class are considered an implementation detail.
+    """
+
     def _fill_text(self, text, width, indent):
-        return ''.join(indent + line for line in text.splitlines(True))
+        return ''.join([indent + line for line in text.splitlines(True)])
+
 
 class RawTextHelpFormatter(RawDescriptionHelpFormatter):
-    
+    """Help message formatter which retains formatting of all help text.
+
+    Only the name of this class is considered a public API. All the methods
+    provided by the class are considered an implementation detail.
+    """
+
     def _split_lines(self, text, width):
         return text.splitlines()
 
+
+class ArgumentDefaultsHelpFormatter(HelpFormatter):
+    """Help message formatter which adds default values to argument help.
+
+    Only the name of this class is considered a public API. All the methods
+    provided by the class are considered an implementation detail.
+    """
+
+    def _get_help_string(self, action):
+        help = action.help
+        if '%(default)' not in action.help:
+            if action.default is not SUPPRESS:
+                defaulting_nargs = [OPTIONAL, ZERO_OR_MORE]
+                if action.option_strings or action.nargs in defaulting_nargs:
+                    help += ' (default: %(default)s)'
+        return help
+
+
 # =====================
 # Options and Arguments
 # =====================
 
-class ArgumentError(Exception):
-    """ArgumentError(message, argument)
+def _get_action_name(argument):
+    if argument is None:
+        return None
+    elif argument.option_strings:
+        return  '/'.join(argument.option_strings)
+    elif argument.metavar not in (None, SUPPRESS):
+        return argument.metavar
+    elif argument.dest not in (None, SUPPRESS):
+        return argument.dest
+    else:
+        return None
 
-    Raised whenever there was an error creating or using an argument
-    (optional or positional).
+
+class ArgumentError(Exception):
+    """An error from creating or using an argument (optional or positional).
 
     The string value of this exception is the message, augmented with
     information about the argument that caused it.
     """
-    
+
     def __init__(self, argument, message):
-        if argument.option_strings:
-            self.argument_name =  '/'.join(argument.option_strings)
-        elif argument.metavar not in (None, SUPPRESS):
-            self.argument_name = argument.metavar
-        elif argument.dest not in (None, SUPPRESS):
-            self.argument_name = argument.dest
-        else:
-            self.argument_name = None
+        self.argument_name = _get_action_name(argument)
         self.message = message
 
     def __str__(self):
         if self.argument_name is None:
             format = '%(message)s'
         else:
             format = 'argument %(argument_name)s: %(message)s'
         return format % dict(message=self.message,
                              argument_name=self.argument_name)
 
 # ==============
 # Action classes
 # ==============
 
 class Action(_AttributeHolder):
-    """Action(*strings, **options)
+    """Information about how to convert command line strings to Python objects.
 
-    Action objects hold the information necessary to convert a
-    set of command-line arguments (possibly including an initial option
-    string) into the desired Python object(s).
+    Action objects are used by an ArgumentParser to represent the information
+    needed to parse a single argument from one or more strings from the
+    command line. The keyword arguments to the Action constructor are also
+    all attributes of Action instances.
 
     Keyword Arguments:
 
-    option_strings -- A list of command-line option strings which
-        should be associated with this action.
-    
-    dest -- The name of the attribute to hold the created object(s)
-    
-    nargs -- The number of command-line arguments that should be consumed.
-        By default, one argument will be consumed and a single value will
-        be produced.  Other values include:
-            * N (an integer) consumes N arguments (and produces a list)
-            * '?' consumes zero or one arguments
-            * '*' consumes zero or more arguments (and produces a list)
-            * '+' consumes one or more arguments (and produces a list)
-        Note that the difference between the default and nargs=1 is that
-        with the default, a single value will be produced, while with
-        nargs=1, a list containing a single value will be produced.
-
-    const -- The value to be produced if the option is specified and the
-        option uses an action that takes no values.
-
-    default -- The value to be produced if the option is not specified.
-
-    type -- The type which the command-line arguments should be converted
-        to, should be one of 'string', 'int', 'float', 'complex' or a
-        callable object that accepts a single string argument. If None,
-        'string' is assumed.
-        
-    choices -- A container of values that should be allowed. If not None,
-        after a command-line argument has been converted to the appropriate
-        type, an exception will be raised if it is not a member of this
-        collection.
-
-    required -- True if the action must always be specified at the command
-        line. This is only meaningful for optional command-line arguments.
-
-    help -- The help string describing the argument.
-
-    metavar -- The name to be used for the option's argument with the help
-        string. If None, the 'dest' value will be used as the name.
-    """
+        - option_strings -- A list of command-line option strings which
+            should be associated with this action.
+
+        - dest -- The name of the attribute to hold the created object(s)
+
+        - nargs -- The number of command-line arguments that should be
+            consumed. By default, one argument will be consumed and a single
+            value will be produced.  Other values include:
+                - N (an integer) consumes N arguments (and produces a list)
+                - '?' consumes zero or one arguments
+                - '*' consumes zero or more arguments (and produces a list)
+                - '+' consumes one or more arguments (and produces a list)
+            Note that the difference between the default and nargs=1 is that
+            with the default, a single value will be produced, while with
+            nargs=1, a list containing a single value will be produced.
+
+        - const -- The value to be produced if the option is specified and the
+            option uses an action that takes no values.
+
+        - default -- The value to be produced if the option is not specified.
+
+        - type -- The type which the command-line arguments should be converted
+            to, should be one of 'string', 'int', 'float', 'complex' or a
+            callable object that accepts a single string argument. If None,
+            'string' is assumed.
+
+        - choices -- A container of values that should be allowed. If not None,
+            after a command-line argument has been converted to the appropriate
+            type, an exception will be raised if it is not a member of this
+            collection.
+
+        - required -- True if the action must always be specified at the
+            command line. This is only meaningful for optional command-line
+            arguments.
 
+        - help -- The help string describing the argument.
+
+        - metavar -- The name to be used for the option's argument with the
+            help string. If None, the 'dest' value will be used as the name.
+    """
 
     def __init__(self,
                  option_strings,
                  dest,
                  nargs=None,
                  const=None,
                  default=None,
                  type=None,
                  choices=None,
                  required=False,
                  help=None,
                  metavar=None):
         self.option_strings = option_strings
         self.dest = dest
         self.nargs = nargs
         self.const = const
         self.default = default
         self.type = type
         self.choices = choices
         self.required = required
         self.help = help
         self.metavar = metavar
 
     def _get_kwargs(self):
         names = [
             'option_strings',
             'dest',
             'nargs',
             'const',
             'default',
             'type',
             'choices',
             'help',
-            'metavar'
+            'metavar',
         ]
         return [(name, getattr(self, name)) for name in names]
 
     def __call__(self, parser, namespace, values, option_string=None):
         raise NotImplementedError(_('.__call__() not defined'))
 
+
 class _StoreAction(Action):
+
     def __init__(self,
                  option_strings,
                  dest,
                  nargs=None,
                  const=None,
                  default=None,
                  type=None,
                  choices=None,
                  required=False,
                  help=None,
                  metavar=None):
         if nargs == 0:
             raise ValueError('nargs must be > 0')
         if const is not None and nargs != OPTIONAL:
             raise ValueError('nargs must be %r to supply const' % OPTIONAL)
         super(_StoreAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             nargs=nargs,
             const=const,
             default=default,
             type=type,
             choices=choices,
             required=required,
             help=help,
             metavar=metavar)
 
     def __call__(self, parser, namespace, values, option_string=None):
         setattr(namespace, self.dest, values)
 
+
 class _StoreConstAction(Action):
+
     def __init__(self,
                  option_strings,
                  dest,
                  const,
                  default=None,
                  required=False,
                  help=None,
                  metavar=None):
         super(_StoreConstAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             nargs=0,
             const=const,
             default=default,
             required=required,
             help=help)
 
     def __call__(self, parser, namespace, values, option_string=None):
         setattr(namespace, self.dest, self.const)
 
+
 class _StoreTrueAction(_StoreConstAction):
+
     def __init__(self,
                  option_strings,
                  dest,
                  default=False,
                  required=False,
                  help=None):
         super(_StoreTrueAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             const=True,
             default=default,
             required=required,
             help=help)
 
+
 class _StoreFalseAction(_StoreConstAction):
+
     def __init__(self,
                  option_strings,
                  dest,
                  default=True,
                  required=False,
                  help=None):
         super(_StoreFalseAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             const=False,
             default=default,
             required=required,
             help=help)
-    
+
+
 class _AppendAction(Action):
+
     def __init__(self,
                  option_strings,
                  dest,
                  nargs=None,
                  const=None,
                  default=None,
                  type=None,
                  choices=None,
                  required=False,
                  help=None,
                  metavar=None):
         if nargs == 0:
             raise ValueError('nargs must be > 0')
         if const is not None and nargs != OPTIONAL:
             raise ValueError('nargs must be %r to supply const' % OPTIONAL)
         super(_AppendAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             nargs=nargs,
             const=const,
             default=default,
             type=type,
             choices=choices,
             required=required,
             help=help,
             metavar=metavar)
 
     def __call__(self, parser, namespace, values, option_string=None):
-        _ensure_value(namespace, self.dest, []).append(values)
+        items = _copy.copy(_ensure_value(namespace, self.dest, []))
+        items.append(values)
+        setattr(namespace, self.dest, items)
+
 
 class _AppendConstAction(Action):
+
     def __init__(self,
                  option_strings,
                  dest,
                  const,
                  default=None,
                  required=False,
                  help=None,
                  metavar=None):
         super(_AppendConstAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             nargs=0,
             const=const,
             default=default,
             required=required,
             help=help,
             metavar=metavar)
 
     def __call__(self, parser, namespace, values, option_string=None):
-        _ensure_value(namespace, self.dest, []).append(self.const)
+        items = _copy.copy(_ensure_value(namespace, self.dest, []))
+        items.append(self.const)
+        setattr(namespace, self.dest, items)
+
 
 class _CountAction(Action):
+
     def __init__(self,
                  option_strings,
                  dest,
                  default=None,
                  required=False,
                  help=None):
         super(_CountAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             nargs=0,
             default=default,
             required=required,
             help=help)
 
     def __call__(self, parser, namespace, values, option_string=None):
         new_count = _ensure_value(namespace, self.dest, 0) + 1
         setattr(namespace, self.dest, new_count)
 
+
 class _HelpAction(Action):
+
     def __init__(self,
                  option_strings,
                  dest=SUPPRESS,
                  default=SUPPRESS,
                  help=None):
         super(_HelpAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             default=default,
             nargs=0,
             help=help)
 
     def __call__(self, parser, namespace, values, option_string=None):
         parser.print_help()
         parser.exit()
 
+
 class _VersionAction(Action):
+
     def __init__(self,
                  option_strings,
                  dest=SUPPRESS,
                  default=SUPPRESS,
                  help=None):
         super(_VersionAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             default=default,
             nargs=0,
             help=help)
 
     def __call__(self, parser, namespace, values, option_string=None):
         parser.print_version()
         parser.exit()
-        
+
+
 class _SubParsersAction(Action):
 
     class _ChoicesPseudoAction(Action):
+
         def __init__(self, name, help):
             sup = super(_SubParsersAction._ChoicesPseudoAction, self)
             sup.__init__(option_strings=[], dest=name, help=help)
 
-        
     def __init__(self,
                  option_strings,
                  prog,
                  parser_class,
                  dest=SUPPRESS,
                  help=None,
                  metavar=None):
-        
+
         self._prog_prefix = prog
         self._parser_class = parser_class
         self._name_parser_map = {}
         self._choices_actions = []
 
         super(_SubParsersAction, self).__init__(
             option_strings=option_strings,
             dest=dest,
             nargs=PARSER,
             choices=self._name_parser_map,
             help=help,
             metavar=metavar)
 
     def add_parser(self, name, **kwargs):
         # set prog from the existing prefix
         if kwargs.get('prog') is None:
             kwargs['prog'] = '%s %s' % (self._prog_prefix, name)
 
         # create a pseudo-action to hold the choice help
         if 'help' in kwargs:
             help = kwargs.pop('help')
             choice_action = self._ChoicesPseudoAction(name, help)
             self._choices_actions.append(choice_action)
 
         # create the parser and add it to the map
         parser = self._parser_class(**kwargs)
         self._name_parser_map[name] = parser
         return parser
 
     def _get_subactions(self):
         return self._choices_actions
 
     def __call__(self, parser, namespace, values, option_string=None):
         parser_name = values[0]
         arg_strings = values[1:]
 
         # set the parser name if requested
         if self.dest is not SUPPRESS:
             setattr(namespace, self.dest, parser_name)
 
-        # select the parser        
+        # select the parser
         try:
             parser = self._name_parser_map[parser_name]
         except KeyError:
             tup = parser_name, ', '.join(self._name_parser_map)
             msg = _('unknown parser %r (choices: %s)' % tup)
             raise ArgumentError(self, msg)
 
         # parse all the remaining options into the namespace
         parser.parse_args(arg_strings, namespace)
 
 
 # ==============
 # Type classes
 # ==============
 
 class FileType(object):
     """Factory for creating file object types
 
     Instances of FileType are typically passed as type= arguments to the
     ArgumentParser add_argument() method.
 
     Keyword Arguments:
-    mode -- A string indicating how the file is to be opened. Accepts the
-        same values as the builtin open() function.
-    bufsize -- The file's desired buffer size. Accepts the same values as
-        the builtin open() function.
-    """    
+        - mode -- A string indicating how the file is to be opened. Accepts the
+            same values as the builtin open() function.
+        - bufsize -- The file's desired buffer size. Accepts the same values as
+            the builtin open() function.
+    """
+
     def __init__(self, mode='r', bufsize=None):
         self._mode = mode
         self._bufsize = bufsize
-    
+
     def __call__(self, string):
         # the special argument "-" means sys.std{in,out}
         if string == '-':
-            if self._mode == 'r':
+            if 'r' in self._mode:
                 return _sys.stdin
-            elif self._mode == 'w':
+            elif 'w' in self._mode:
                 return _sys.stdout
             else:
                 msg = _('argument "-" with mode %r' % self._mode)
                 raise ValueError(msg)
 
         # all other arguments are used as file names
         if self._bufsize:
             return open(string, self._mode, self._bufsize)
         else:
             return open(string, self._mode)
 
+    def __repr__(self):
+        args = [self._mode, self._bufsize]
+        args_str = ', '.join([repr(arg) for arg in args if arg is not None])
+        return '%s(%s)' % (type(self).__name__, args_str)
 
 # ===========================
 # Optional and Positional Parsing
 # ===========================
 
 class Namespace(_AttributeHolder):
+    """Simple object for storing attributes.
+
+    Implements equality by attribute names and values, and provides a simple
+    string representation.
+    """
 
     def __init__(self, **kwargs):
-        for name, value in kwargs.iteritems():
-            setattr(self, name, value)
+        for name in kwargs:
+            setattr(self, name, kwargs[name])
 
     def __eq__(self, other):
         return vars(self) == vars(other)
 
     def __ne__(self, other):
         return not (self == other)
 
 
 class _ActionsContainer(object):
+
     def __init__(self,
                  description,
                  prefix_chars,
                  argument_default,
                  conflict_handler):
         super(_ActionsContainer, self).__init__()
 
         self.description = description
         self.argument_default = argument_default
         self.prefix_chars = prefix_chars
         self.conflict_handler = conflict_handler
 
         # set up registries
         self._registries = {}
 
         # register actions
         self.register('action', None, _StoreAction)
         self.register('action', 'store', _StoreAction)
         self.register('action', 'store_const', _StoreConstAction)
         self.register('action', 'store_true', _StoreTrueAction)
         self.register('action', 'store_false', _StoreFalseAction)
         self.register('action', 'append', _AppendAction)
         self.register('action', 'append_const', _AppendConstAction)
         self.register('action', 'count', _CountAction)
         self.register('action', 'help', _HelpAction)
         self.register('action', 'version', _VersionAction)
         self.register('action', 'parsers', _SubParsersAction)
-        
+
         # raise an exception if the conflict handler is invalid
         self._get_handler()
 
         # action storage
-        self._optional_actions_list = []
-        self._positional_actions_list = []
-        self._positional_actions_full_list = []
-        self._option_strings = {}
+        self._actions = []
+        self._option_string_actions = {}
+
+        # groups
+        self._action_groups = []
+        self._mutually_exclusive_groups = []
 
         # defaults storage
         self._defaults = {}
 
+        # determines whether an "option" looks like a negative number
+        self._negative_number_matcher = _re.compile(r'^-\d+|-\d*.\d+$')
+
+        # whether or not there are any optionals that look like negative
+        # numbers -- uses a list so it can be shared and edited
+        self._has_negative_number_optionals = []
+
     # ====================
     # Registration methods
     # ====================
-
     def register(self, registry_name, value, object):
         registry = self._registries.setdefault(registry_name, {})
         registry[value] = object
 
     def _registry_get(self, registry_name, value, default=None):
         return self._registries[registry_name].get(value, default)
 
     # ==================================
     # Namespace default settings methods
     # ==================================
-
     def set_defaults(self, **kwargs):
         self._defaults.update(kwargs)
 
         # if these defaults match any existing arguments, replace
         # the previous default on the object with the new one
-        for action_list in [self._option_strings.values(),
-                            self._positional_actions_full_list]:
-            for action in action_list:
-                if action.dest in kwargs:
-                    action.default = kwargs[action.dest]
+        for action in self._actions:
+            if action.dest in kwargs:
+                action.default = kwargs[action.dest]
 
     # =======================
     # Adding argument actions
     # =======================
-
     def add_argument(self, *args, **kwargs):
         """
         add_argument(dest, ..., name=value, ...)
         add_argument(option_string, option_string, ..., name=value, ...)
         """
 
         # if no positional args are supplied or only one is supplied and
         # it doesn't look like an option string, parse a positional
         # argument
         chars = self.prefix_chars
         if not args or len(args) == 1 and args[0][0] not in chars:
             kwargs = self._get_positional_kwargs(*args, **kwargs)
 
         # otherwise, we're adding an optional argument
         else:
             kwargs = self._get_optional_kwargs(*args, **kwargs)
 
         # if no default was supplied, use the parser-level default
         if 'default' not in kwargs:
             dest = kwargs['dest']
             if dest in self._defaults:
                 kwargs['default'] = self._defaults[dest]
             elif self.argument_default is not None:
                 kwargs['default'] = self.argument_default
-            
+
         # create the action object, and add it to the parser
         action_class = self._pop_action_class(kwargs)
         action = action_class(**kwargs)
         return self._add_action(action)
 
+    def add_argument_group(self, *args, **kwargs):
+        group = _ArgumentGroup(self, *args, **kwargs)
+        self._action_groups.append(group)
+        return group
+
+    def add_mutually_exclusive_group(self, **kwargs):
+        group = _MutuallyExclusiveGroup(self, **kwargs)
+        self._mutually_exclusive_groups.append(group)
+        return group
+
     def _add_action(self, action):
         # resolve any conflicts
         self._check_conflict(action)
 
-        # add to optional or positional list
-        if action.option_strings:
-            self._optional_actions_list.append(action)
-        else:
-            self._positional_actions_list.append(action)
-            self._positional_actions_full_list.append(action)
+        # add to actions list
+        self._actions.append(action)
         action.container = self
 
         # index the action by any option strings it has
         for option_string in action.option_strings:
-            self._option_strings[option_string] = action
+            self._option_string_actions[option_string] = action
+
+        # set the flag if any option strings look like negative numbers
+        for option_string in action.option_strings:
+            if self._negative_number_matcher.match(option_string):
+                if not self._has_negative_number_optionals:
+                    self._has_negative_number_optionals.append(True)
 
         # return the created action
         return action
 
+    def _remove_action(self, action):
+        self._actions.remove(action)
+
     def _add_container_actions(self, container):
-        for action in container._optional_actions_list:
-            self._add_action(action)
-        for action in container._positional_actions_list:
-            self._add_action(action)
+        # collect groups by titles
+        title_group_map = {}
+        for group in self._action_groups:
+            if group.title in title_group_map:
+                msg = _('cannot merge actions - two groups are named %r')
+                raise ValueError(msg % (group.title))
+            title_group_map[group.title] = group
+
+        # map each action to its group
+        group_map = {}
+        for group in container._action_groups:
+
+            # if a group with the title exists, use that, otherwise
+            # create a new group matching the container's group
+            if group.title not in title_group_map:
+                title_group_map[group.title] = self.add_argument_group(
+                    title=group.title,
+                    description=group.description,
+                    conflict_handler=group.conflict_handler)
+
+            # map the actions to their new group
+            for action in group._group_actions:
+                group_map[action] = title_group_map[group.title]
+
+        # add all actions to this container or their group
+        for action in container._actions:
+            group_map.get(action, self)._add_action(action)
 
     def _get_positional_kwargs(self, dest, **kwargs):
         # make sure required is not specified
         if 'required' in kwargs:
             msg = _("'required' is an invalid argument for positionals")
             raise TypeError(msg)
-        
+
+        # mark positional arguments as required if at least one is
+        # always required
+        if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]:
+            kwargs['required'] = True
+        if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:
+            kwargs['required'] = True
+
         # return the keyword arguments with no option strings
         return dict(kwargs, dest=dest, option_strings=[])
 
     def _get_optional_kwargs(self, *args, **kwargs):
         # determine short and long option strings
         option_strings = []
         long_option_strings = []
         for option_string in args:
             # error on one-or-fewer-character option strings
             if len(option_string) < 2:
                 msg = _('invalid option string %r: '
                         'must be at least two characters long')
                 raise ValueError(msg % option_string)
 
             # error on strings that don't start with an appropriate prefix
             if not option_string[0] in self.prefix_chars:
                 msg = _('invalid option string %r: '
                         'must start with a character %r')
                 tup = option_string, self.prefix_chars
                 raise ValueError(msg % tup)
 
             # error on strings that are all prefix characters
-            if not (set(option_string) - set(self.prefix_chars)):
+            if not (_set(option_string) - _set(self.prefix_chars)):
                 msg = _('invalid option string %r: '
                         'must contain characters other than %r')
                 tup = option_string, self.prefix_chars
                 raise ValueError(msg % tup)
 
             # strings starting with two prefix characters are long options
-            option_strings.append(option_string)            
+            option_strings.append(option_string)
             if option_string[0] in self.prefix_chars:
                 if option_string[1] in self.prefix_chars:
                     long_option_strings.append(option_string)
 
         # infer destination, '--foo-bar' -> 'foo_bar' and '-x' -> 'x'
         dest = kwargs.pop('dest', None)
         if dest is None:
             if long_option_strings:
                 dest_option_string = long_option_strings[0]
             else:
                 dest_option_string = option_strings[0]
             dest = dest_option_string.lstrip(self.prefix_chars)
             dest = dest.replace('-', '_')
 
         # return the updated keyword arguments
         return dict(kwargs, dest=dest, option_strings=option_strings)
 
     def _pop_action_class(self, kwargs, default=None):
         action = kwargs.pop('action', default)
         return self._registry_get('action', action, action)
 
     def _get_handler(self):
         # determine function from conflict handler string
         handler_func_name = '_handle_conflict_%s' % self.conflict_handler
         try:
             return getattr(self, handler_func_name)
         except AttributeError:
             msg = _('invalid conflict_resolution value: %r')
             raise ValueError(msg % self.conflict_handler)
 
     def _check_conflict(self, action):
 
-        # find all options that conflict with this option        
+        # find all options that conflict with this option
         confl_optionals = []
         for option_string in action.option_strings:
-            if option_string in self._option_strings:
-                confl_optional = self._option_strings[option_string]
+            if option_string in self._option_string_actions:
+                confl_optional = self._option_string_actions[option_string]
                 confl_optionals.append((option_string, confl_optional))
 
         # resolve any conflicts
         if confl_optionals:
             conflict_handler = self._get_handler()
             conflict_handler(action, confl_optionals)
 
     def _handle_conflict_error(self, action, conflicting_actions):
         message = _('conflicting option string(s): %s')
-        conflict_string = ', '.join(option_string
-                                    for option_string, action
-                                    in conflicting_actions)
+        conflict_string = ', '.join([option_string
+                                     for option_string, action
+                                     in conflicting_actions])
         raise ArgumentError(action, message % conflict_string)
 
     def _handle_conflict_resolve(self, action, conflicting_actions):
 
-        # remove all conflicting options        
+        # remove all conflicting options
         for option_string, action in conflicting_actions:
-            
+
             # remove the conflicting option
             action.option_strings.remove(option_string)
-            self._option_strings.pop(option_string, None)
+            self._option_string_actions.pop(option_string, None)
 
             # if the option now has no option string, remove it from the
             # container holding it
             if not action.option_strings:
-                action.container._optional_actions_list.remove(action)
+                action.container._remove_action(action)
 
 
 class _ArgumentGroup(_ActionsContainer):
 
     def __init__(self, container, title=None, description=None, **kwargs):
         # add any missing keyword arguments by checking the container
         update = kwargs.setdefault
         update('conflict_handler', container.conflict_handler)
         update('prefix_chars', container.prefix_chars)
         update('argument_default', container.argument_default)
         super_init = super(_ArgumentGroup, self).__init__
         super_init(description=description, **kwargs)
-        
+
+        # group attributes
         self.title = title
+        self._group_actions = []
+
+        # share most attributes with the container
         self._registries = container._registries
-        self._positional_actions_full_list = container._positional_actions_full_list
-        self._option_strings = container._option_strings
+        self._actions = container._actions
+        self._option_string_actions = container._option_string_actions
         self._defaults = container._defaults
+        self._has_negative_number_optionals = \
+            container._has_negative_number_optionals
+
+    def _add_action(self, action):
+        action = super(_ArgumentGroup, self)._add_action(action)
+        self._group_actions.append(action)
+        return action
+
+    def _remove_action(self, action):
+        super(_ArgumentGroup, self)._remove_action(action)
+        self._group_actions.remove(action)
+
+
+class _MutuallyExclusiveGroup(_ArgumentGroup):
+
+    def __init__(self, container, required=False):
+        super(_MutuallyExclusiveGroup, self).__init__(container)
+        self.required = required
+        self._container = container
+
+    def _add_action(self, action):
+        if action.required:
+            msg = _('mutually exclusive arguments must be optional')
+            raise ValueError(msg)
+        action = self._container._add_action(action)
+        self._group_actions.append(action)
+        return action
+
+    def _remove_action(self, action):
+        self._container._remove_action(action)
+        self._group_actions.remove(action)
 
 
 class ArgumentParser(_AttributeHolder, _ActionsContainer):
+    """Object for parsing command line strings into Python objects.
+
+    Keyword Arguments:
+        - prog -- The name of the program (default: sys.argv[0])
+        - usage -- A usage message (default: auto-generated from arguments)
+        - description -- A description of what the program does
+        - epilog -- Text following the argument descriptions
+        - version -- Add a -v/--version option with the given version string
+        - parents -- Parsers whose arguments should be copied into this one
+        - formatter_class -- HelpFormatter class for printing help messages
+        - prefix_chars -- Characters that prefix optional arguments
+        - fromfile_prefix_chars -- Characters that prefix files containing
+            additional arguments
+        - argument_default -- The default value for all arguments
+        - conflict_handler -- String indicating how to handle conflicts
+        - add_help -- Add a -h/-help option
+    """
 
     def __init__(self,
                  prog=None,
                  usage=None,
                  description=None,
                  epilog=None,
                  version=None,
                  parents=[],
                  formatter_class=HelpFormatter,
                  prefix_chars='-',
+                 fromfile_prefix_chars=None,
                  argument_default=None,
                  conflict_handler='error',
                  add_help=True):
 
         superinit = super(ArgumentParser, self).__init__
         superinit(description=description,
                   prefix_chars=prefix_chars,
                   argument_default=argument_default,
                   conflict_handler=conflict_handler)
 
         # default setting for prog
         if prog is None:
             prog = _os.path.basename(_sys.argv[0])
 
         self.prog = prog
         self.usage = usage
         self.epilog = epilog
         self.version = version
         self.formatter_class = formatter_class
+        self.fromfile_prefix_chars = fromfile_prefix_chars
         self.add_help = add_help
 
-        self._argument_group_class = _ArgumentGroup
-        self._has_subparsers = False
-        self._argument_groups = []
+        add_group = self.add_argument_group
+        self._positionals = add_group(_('positional arguments'))
+        self._optionals = add_group(_('optional arguments'))
+        self._subparsers = None
 
         # register types
         def identity(string):
             return string
         self.register('type', None, identity)
 
         # add help and version arguments if necessary
         # (using explicit default to override global argument_default)
         if self.add_help:
             self.add_argument(
                 '-h', '--help', action='help', default=SUPPRESS,
                 help=_('show this help message and exit'))
         if self.version:
             self.add_argument(
                 '-v', '--version', action='version', default=SUPPRESS,
                 help=_("show program's version number and exit"))
 
         # add parent arguments and defaults
         for parent in parents:
             self._add_container_actions(parent)
             try:
                 defaults = parent._defaults
             except AttributeError:
                 pass
             else:
                 self._defaults.update(defaults)
 
-        # determines whether an "option" looks like a negative number
-        self._negative_number_matcher = _re.compile(r'^-\d+|-\d*.\d+$')
-                
-
     # =======================
     # Pretty __repr__ methods
     # =======================
-
     def _get_kwargs(self):
         names = [
             'prog',
             'usage',
             'description',
             'version',
             'formatter_class',
             'conflict_handler',
             'add_help',
         ]
         return [(name, getattr(self, name)) for name in names]
 
     # ==================================
     # Optional/Positional adding methods
     # ==================================
-
-    def add_argument_group(self, *args, **kwargs):
-        group = self._argument_group_class(self, *args, **kwargs)
-        self._argument_groups.append(group)
-        return group
-
     def add_subparsers(self, **kwargs):
-        if self._has_subparsers:
+        if self._subparsers is not None:
             self.error(_('cannot have multiple subparser arguments'))
-        
+
         # add the parser class to the arguments if it's not present
         kwargs.setdefault('parser_class', type(self))
 
+        if 'title' in kwargs or 'description' in kwargs:
+            title = _(kwargs.pop('title', 'subcommands'))
+            description = _(kwargs.pop('description', None))
+            self._subparsers = self.add_argument_group(title, description)
+        else:
+            self._subparsers = self._positionals
+
         # prog defaults to the usage message of this parser, skipping
         # optional arguments and with no "usage:" prefix
         if kwargs.get('prog') is None:
             formatter = self._get_formatter()
-            formatter.add_usage(self.usage, [],
-                                self._get_positional_actions(), '')
+            positionals = self._get_positional_actions()
+            groups = self._mutually_exclusive_groups
+            formatter.add_usage(self.usage, positionals, groups, '')
             kwargs['prog'] = formatter.format_help().strip()
 
         # create the parsers action and add it to the positionals list
         parsers_class = self._pop_action_class(kwargs, 'parsers')
         action = parsers_class(option_strings=[], **kwargs)
-        self._positional_actions_list.append(action)
-        self._positional_actions_full_list.append(action)
-        self._has_subparsers = True
+        self._subparsers._add_action(action)
 
         # return the created parsers action
         return action
 
-    def _add_container_actions(self, container):
-        super(ArgumentParser, self)._add_container_actions(container)
-        try:
-            groups = container._argument_groups
-        except AttributeError:
-            pass
+    def _add_action(self, action):
+        if action.option_strings:
+            self._optionals._add_action(action)
         else:
-            for group in groups:
-                new_group = self.add_argument_group(
-                    title=group.title,
-                    description=group.description,
-                    conflict_handler=group.conflict_handler)
-                new_group._add_container_actions(group)
+            self._positionals._add_action(action)
+        return action
 
     def _get_optional_actions(self):
-        actions = []
-        actions.extend(self._optional_actions_list)
-        for argument_group in self._argument_groups:
-            actions.extend(argument_group._optional_actions_list)
-        return actions
+        return [action
+                for action in self._actions
+                if action.option_strings]
 
     def _get_positional_actions(self):
-        return list(self._positional_actions_full_list)
-
+        return [action
+                for action in self._actions
+                if not action.option_strings]
 
     # =====================================
     # Command line argument parsing methods
     # =====================================
-
     def parse_args(self, args=None, namespace=None):
+        args, argv = self.parse_known_args(args, namespace)
+        if argv:
+            msg = _('unrecognized arguments: %s')
+            self.error(msg % ' '.join(argv))
+        return args
+
+    def parse_known_args(self, args=None, namespace=None):
         # args default to the system args
         if args is None:
             args = _sys.argv[1:]
 
         # default Namespace built from parser defaults
         if namespace is None:
             namespace = Namespace()
-            
+
         # add any action defaults that aren't present
-        optional_actions = self._get_optional_actions()
-        positional_actions = self._get_positional_actions()
-        for action in optional_actions + positional_actions:
+        for action in self._actions:
             if action.dest is not SUPPRESS:
                 if not hasattr(namespace, action.dest):
                     if action.default is not SUPPRESS:
                         default = action.default
-                        if isinstance(action.default, basestring):
+                        if isinstance(action.default, _basestring):
                             default = self._get_value(action, default)
                         setattr(namespace, action.dest, default)
 
         # add any parser defaults that aren't present
-        for dest, value in self._defaults.iteritems():
+        for dest in self._defaults:
             if not hasattr(namespace, dest):
-                setattr(namespace, dest, value)
-            
+                setattr(namespace, dest, self._defaults[dest])
+
         # parse the arguments and exit if there are any errors
         try:
-            result = self._parse_args(args, namespace)
-        except ArgumentError, err:
+            return self._parse_known_args(args, namespace)
+        except ArgumentError:
+            err = _sys.exc_info()[1]
             self.error(str(err))
 
-        # make sure all required optionals are present
-        for action in self._get_optional_actions():
-            if action.required:
-                if getattr(result, action.dest, None) is None:
-                    opt_strs = '/'.join(action.option_strings)
-                    msg = _('option %s is required' % opt_strs)
-                    self.error(msg)
-
-        # return the parsed arguments                    
-        return result
-
-    def _parse_args(self, arg_strings, namespace):            
+    def _parse_known_args(self, arg_strings, namespace):
+        # replace arg strings that are file references
+        if self.fromfile_prefix_chars is not None:
+            arg_strings = self._read_args_from_files(arg_strings)
+
+        # map all mutually exclusive arguments to the other arguments
+        # they can't occur with
+        action_conflicts = {}
+        for mutex_group in self._mutually_exclusive_groups:
+            group_actions = mutex_group._group_actions
+            for i, mutex_action in enumerate(mutex_group._group_actions):
+                conflicts = action_conflicts.setdefault(mutex_action, [])
+                conflicts.extend(group_actions[:i])
+                conflicts.extend(group_actions[i + 1:])
 
         # find all option indices, and determine the arg_string_pattern
         # which has an 'O' if there is an option at an index,
         # an 'A' if there is an argument, or a '-' if there is a '--'
         option_string_indices = {}
         arg_string_pattern_parts = []
         arg_strings_iter = iter(arg_strings)
         for i, arg_string in enumerate(arg_strings_iter):
 
             # all args after -- are non-options
             if arg_string == '--':
                 arg_string_pattern_parts.append('-')
                 for arg_string in arg_strings_iter:
                     arg_string_pattern_parts.append('A')
 
             # otherwise, add the arg to the arg strings
             # and note the index if it was an option
             else:
                 option_tuple = self._parse_optional(arg_string)
                 if option_tuple is None:
                     pattern = 'A'
                 else:
                     option_string_indices[i] = option_tuple
                     pattern = 'O'
                 arg_string_pattern_parts.append(pattern)
 
         # join the pieces together to form the pattern
         arg_strings_pattern = ''.join(arg_string_pattern_parts)
 
         # converts arg strings to the appropriate and then takes the action
+        seen_actions = _set()
+        seen_non_default_actions = _set()
+
         def take_action(action, argument_strings, option_string=None):
+            seen_actions.add(action)
             argument_values = self._get_values(action, argument_strings)
+
+            # error if this argument is not allowed with other previously
+            # seen arguments, assuming that actions that use the default
+            # value don't really count as "present"
+            if argument_values is not action.default:
+                seen_non_default_actions.add(action)
+                for conflict_action in action_conflicts.get(action, []):
+                    if conflict_action in seen_non_default_actions:
+                        msg = _('not allowed with argument %s')
+                        action_name = _get_action_name(conflict_action)
+                        raise ArgumentError(action, msg % action_name)
+
             # take the action if we didn't receive a SUPPRESS value
-            # (e.g. from a default)            
+            # (e.g. from a default)
             if argument_values is not SUPPRESS:
                 action(self, namespace, argument_values, option_string)
 
         # function to convert arg_strings into an optional action
         def consume_optional(start_index):
-            
-            # determine the optional action and parse any explicit
-            # argument out of the option string
+
+            # get the optional identified at this index
             option_tuple = option_string_indices[start_index]
             action, option_string, explicit_arg = option_tuple
 
-            # loop because single-dash options can be chained
+            # identify additional optionals in the same arg string
             # (e.g. -xyz is the same as -x -y -z if no args are required)
             match_argument = self._match_argument
             action_tuples = []
             while True:
 
-                # if we found no optional action, raise an error
+                # if we found no optional action, skip it
                 if action is None:
-                    self.error(_('no such option: %s') % option_string)
+                    extras.append(arg_strings[start_index])
+                    return start_index + 1
 
                 # if there is an explicit argument, try to match the
                 # optional's string arguments to only this
                 if explicit_arg is not None:
                     arg_count = match_argument(action, 'A')
 
                     # if the action is a single-dash option and takes no
                     # arguments, try to parse more single-dash options out
                     # of the tail of the option string
                     chars = self.prefix_chars
                     if arg_count == 0 and option_string[1] not in chars:
                         action_tuples.append((action, [], option_string))
-                        parse_optional = self._parse_optional
                         for char in self.prefix_chars:
-                            option_string = char + explicit_arg
-                            option_tuple = parse_optional(option_string)
-                            if option_tuple[0] is not None:
+                            option_string = char + explicit_arg[0]
+                            explicit_arg = explicit_arg[1:] or None
+                            optionals_map = self._option_string_actions
+                            if option_string in optionals_map:
+                                action = optionals_map[option_string]
                                 break
                         else:
                             msg = _('ignored explicit argument %r')
                             raise ArgumentError(action, msg % explicit_arg)
 
-                        # set the action, etc. for the next loop iteration
-                        action, option_string, explicit_arg = option_tuple
-
                     # if the action expect exactly one argument, we've
                     # successfully matched the option; exit the loop
                     elif arg_count == 1:
                         stop = start_index + 1
                         args = [explicit_arg]
                         action_tuples.append((action, args, option_string))
                         break
 
                     # error if a double-dash option did not use the
                     # explicit argument
                     else:
                         msg = _('ignored explicit argument %r')
                         raise ArgumentError(action, msg % explicit_arg)
-                
+
                 # if there is no explicit argument, try to match the
                 # optional's string arguments with the following strings
                 # if successful, exit the loop
                 else:
                     start = start_index + 1
                     selected_patterns = arg_strings_pattern[start:]
                     arg_count = match_argument(action, selected_patterns)
                     stop = start + arg_count
                     args = arg_strings[start:stop]
                     action_tuples.append((action, args, option_string))
                     break
 
             # add the Optional to the list and return the index at which
             # the Optional's string args stopped
             assert action_tuples
             for action, args, option_string in action_tuples:
                 take_action(action, args, option_string)
             return stop
 
         # the list of Positionals left to be parsed; this is modified
         # by consume_positionals()
         positionals = self._get_positional_actions()
 
         # function to convert arg_strings into positional actions
         def consume_positionals(start_index):
             # match as many Positionals as possible
             match_partial = self._match_arguments_partial
             selected_pattern = arg_strings_pattern[start_index:]
             arg_counts = match_partial(positionals, selected_pattern)
 
             # slice off the appropriate arg strings for each Positional
             # and add the Positional and its args to the list
             for action, arg_count in zip(positionals, arg_counts):
                 args = arg_strings[start_index: start_index + arg_count]
                 start_index += arg_count
                 take_action(action, args)
 
             # slice off the Positionals that we just parsed and return the
             # index at which the Positionals' string args stopped
             positionals[:] = positionals[len(arg_counts):]
             return start_index
 
         # consume Positionals and Optionals alternately, until we have
         # passed the last option string
+        extras = []
         start_index = 0
         if option_string_indices:
             max_option_string_index = max(option_string_indices)
         else:
             max_option_string_index = -1
         while start_index <= max_option_string_index:
-            
+
             # consume any Positionals preceding the next option
-            next_option_string_index = min(
+            next_option_string_index = min([
                 index
                 for index in option_string_indices
-                if index >= start_index)
+                if index >= start_index])
             if start_index != next_option_string_index:
                 positionals_end_index = consume_positionals(start_index)
 
                 # only try to parse the next optional if we didn't consume
                 # the option string during the positionals parsing
                 if positionals_end_index > start_index:
                     start_index = positionals_end_index
                     continue
                 else:
                     start_index = positionals_end_index
 
             # if we consumed all the positionals we could and we're not
-            # at the index of an option string, there were unparseable
-            # arguments
+            # at the index of an option string, there were extra arguments
             if start_index not in option_string_indices:
-                msg = _('extra arguments found: %s')
-                extras = arg_strings[start_index:next_option_string_index]
-                self.error(msg % ' '.join(extras))
+                strings = arg_strings[start_index:next_option_string_index]
+                extras.extend(strings)
+                start_index = next_option_string_index
 
             # consume the next optional and any arguments for it
             start_index = consume_optional(start_index)
 
         # consume any positionals following the last Optional
         stop_index = consume_positionals(start_index)
 
-        # if we didn't consume all the argument strings, there were too
-        # many supplied
-        if stop_index != len(arg_strings):
-            extras = arg_strings[stop_index:]
-            self.error(_('extra arguments found: %s') % ' '.join(extras))
+        # if we didn't consume all the argument strings, there were extras
+        extras.extend(arg_strings[stop_index:])
 
         # if we didn't use all the Positional objects, there were too few
         # arg strings supplied.
         if positionals:
             self.error(_('too few arguments'))
 
-        # return the updated namespace
-        return namespace
+        # make sure all required actions were present
+        for action in self._actions:
+            if action.required:
+                if action not in seen_actions:
+                    name = _get_action_name(action)
+                    self.error(_('argument %s is required') % name)
+
+        # make sure all required groups had one option present
+        for group in self._mutually_exclusive_groups:
+            if group.required:
+                for action in group._group_actions:
+                    if action in seen_non_default_actions:
+                        break
+
+                # if no actions were used, report the error
+                else:
+                    names = [_get_action_name(action)
+                             for action in group._group_actions
+                             if action.help is not SUPPRESS]
+                    msg = _('one of the arguments %s is required')
+                    self.error(msg % ' '.join(names))
+
+        # return the updated namespace and the extra arguments
+        return namespace, extras
+
+    def _read_args_from_files(self, arg_strings):
+        # expand arguments referencing files
+        new_arg_strings = []
+        for arg_string in arg_strings:
+
+            # for regular arguments, just add them back into the list
+            if arg_string[0] not in self.fromfile_prefix_chars:
+                new_arg_strings.append(arg_string)
+
+            # replace arguments referencing files with the file content
+            else:
+                try:
+                    args_file = open(arg_string[1:])
+                    try:
+                        arg_strings = args_file.read().splitlines()
+                        arg_strings = self._read_args_from_files(arg_strings)
+                        new_arg_strings.extend(arg_strings)
+                    finally:
+                        args_file.close()
+                except IOError:
+                    err = _sys.exc_info()[1]
+                    self.error(str(err))
+
+        # return the modified argument list
+        return new_arg_strings
 
     def _match_argument(self, action, arg_strings_pattern):
         # match the pattern for this action to the arg strings
         nargs_pattern = self._get_nargs_pattern(action)
         match = _re.match(nargs_pattern, arg_strings_pattern)
 
-        # raise an exception if we weren't able to find a match        
+        # raise an exception if we weren't able to find a match
         if match is None:
             nargs_errors = {
-                None:_('expected one argument'),
-                OPTIONAL:_('expected at most one argument'),
-                ONE_OR_MORE:_('expected at least one argument')
+                None: _('expected one argument'),
+                OPTIONAL: _('expected at most one argument'),
+                ONE_OR_MORE: _('expected at least one argument'),
             }
             default = _('expected %s argument(s)') % action.nargs
             msg = nargs_errors.get(action.nargs, default)
             raise ArgumentError(action, msg)
 
         # return the number of arguments matched
         return len(match.group(1))
 
     def _match_arguments_partial(self, actions, arg_strings_pattern):
         # progressively shorten the actions list by slicing off the
         # final actions until we find a match
         result = []
-        for i in xrange(len(actions), 0, -1):
+        for i in range(len(actions), 0, -1):
             actions_slice = actions[:i]
-            pattern = ''.join(self._get_nargs_pattern(action)
-                              for action in actions_slice)
+            pattern = ''.join([self._get_nargs_pattern(action)
+                               for action in actions_slice])
             match = _re.match(pattern, arg_strings_pattern)
             if match is not None:
-                result.extend(len(string) for string in match.groups())
+                result.extend([len(string) for string in match.groups()])
                 break
 
         # return the list of arg string counts
         return result
-    
+
     def _parse_optional(self, arg_string):
+        # if it's an empty string, it was meant to be a positional
+        if not arg_string:
+            return None
+
         # if it doesn't start with a prefix, it was meant to be positional
         if not arg_string[0] in self.prefix_chars:
             return None
-        
+
         # if it's just dashes, it was meant to be positional
         if not arg_string.strip('-'):
             return None
 
         # if the option string is present in the parser, return the action
-        if arg_string in self._option_strings:
-            action = self._option_strings[arg_string]
+        if arg_string in self._option_string_actions:
+            action = self._option_string_actions[arg_string]
             return action, arg_string, None
 
         # search through all possible prefixes of the option string
         # and all actions in the parser for possible interpretations
-        option_tuples = []
-        prefix_tuples = self._get_option_prefix_tuples(arg_string)
-        for option_string in self._option_strings:
-            for option_prefix, explicit_arg in prefix_tuples:
-                if option_string.startswith(option_prefix):
-                    action = self._option_strings[option_string]
-                    tup = action, option_string, explicit_arg
-                    option_tuples.append(tup)
-                    break
+        option_tuples = self._get_option_tuples(arg_string)
 
         # if multiple actions match, the option string was ambiguous
         if len(option_tuples) > 1:
-            options = ', '.join(opt_str for _, opt_str, _ in option_tuples)
+            options = ', '.join([option_string
+                for action, option_string, explicit_arg in option_tuples])
             tup = arg_string, options
             self.error(_('ambiguous option: %s could match %s') % tup)
 
         # if exactly one action matched, this segmentation is good,
         # so return the parsed action
         elif len(option_tuples) == 1:
             option_tuple, = option_tuples
             return option_tuple
 
         # if it was not found as an option, but it looks like a negative
         # number, it was meant to be positional
+        # unless there are negative-number-like options
         if self._negative_number_matcher.match(arg_string):
+            if not self._has_negative_number_optionals:
+                return None
+
+        # if it contains a space, it was meant to be a positional
+        if ' ' in arg_string:
             return None
 
         # it was meant to be an optional but there is no such option
         # in this parser (though it might be a valid option in a subparser)
         return None, arg_string, None
 
-    def _get_option_prefix_tuples(self, option_string):
+    def _get_option_tuples(self, option_string):
         result = []
-        
+
         # option strings starting with two prefix characters are only
         # split at the '='
         chars = self.prefix_chars
         if option_string[0] in chars and option_string[1] in chars:
             if '=' in option_string:
                 option_prefix, explicit_arg = option_string.split('=', 1)
             else:
                 option_prefix = option_string
                 explicit_arg = None
-            tup = option_prefix, explicit_arg
-            result.append(tup)
+            for option_string in self._option_string_actions:
+                if option_string.startswith(option_prefix):
+                    action = self._option_string_actions[option_string]
+                    tup = action, option_string, explicit_arg
+                    result.append(tup)
+
+        # single character options can be concatenated with their arguments
+        # but multiple character options always have to have their argument
+        # separate
+        elif option_string[0] in chars and option_string[1] not in chars:
+            option_prefix = option_string
+            explicit_arg = None
+            short_option_prefix = option_string[:2]
+            short_explicit_arg = option_string[2:]
+
+            for option_string in self._option_string_actions:
+                if option_string == short_option_prefix:
+                    action = self._option_string_actions[option_string]
+                    tup = action, option_string, short_explicit_arg
+                    result.append(tup)
+                elif option_string.startswith(option_prefix):
+                    action = self._option_string_actions[option_string]
+                    tup = action, option_string, explicit_arg
+                    result.append(tup)
 
-        # option strings starting with a single prefix character are
-        # split at all indices
+        # shouldn't ever get here
         else:
-            for first_index, char in enumerate(option_string):
-                if char not in self.prefix_chars:
-                    break
-            for i in xrange(len(option_string), first_index, -1):
-                tup = option_string[:i], option_string[i:] or None
-                result.append(tup)
+            self.error(_('unexpected option string: %s') % option_string)
 
-        # return the collected prefix tuples
-        return result                
+        # return the collected option tuples
+        return result
 
     def _get_nargs_pattern(self, action):
         # in all examples below, we have to allow for '--' args
         # which are represented as '-' in the pattern
         nargs = action.nargs
-        
+
         # the default (None) is assumed to be a single argument
         if nargs is None:
             nargs_pattern = '(-*A-*)'
 
         # allow zero or one arguments
         elif nargs == OPTIONAL:
             nargs_pattern = '(-*A?-*)'
 
         # allow zero or more arguments
         elif nargs == ZERO_OR_MORE:
             nargs_pattern = '(-*[A-]*)'
 
         # allow one or more arguments
         elif nargs == ONE_OR_MORE:
             nargs_pattern = '(-*A[A-]*)'
 
         # allow one argument followed by any number of options or arguments
         elif nargs is PARSER:
             nargs_pattern = '(-*A[-AO]*)'
 
-        # all others should be integers            
+        # all others should be integers
         else:
             nargs_pattern = '(-*%s-*)' % '-*'.join('A' * nargs)
 
-        # if this is an optional action, -- is not allowed        
+        # if this is an optional action, -- is not allowed
         if action.option_strings:
             nargs_pattern = nargs_pattern.replace('-*', '')
             nargs_pattern = nargs_pattern.replace('-', '')
 
         # return the pattern
         return nargs_pattern
 
     # ========================
     # Value conversion methods
     # ========================
-
     def _get_values(self, action, arg_strings):
         # for everything but PARSER args, strip out '--'
         if action.nargs is not PARSER:
             arg_strings = [s for s in arg_strings if s != '--']
-        
+
         # optional argument produces a default when not present
         if not arg_strings and action.nargs == OPTIONAL:
             if action.option_strings:
                 value = action.const
             else:
                 value = action.default
-            if isinstance(value, basestring):
+            if isinstance(value, _basestring):
                 value = self._get_value(action, value)
                 self._check_value(action, value)
 
         # when nargs='*' on a positional, if there were no command-line
         # args, use the default if it is anything other than None
         elif (not arg_strings and action.nargs == ZERO_OR_MORE and
               not action.option_strings):
             if action.default is not None:
                 value = action.default
             else:
                 value = arg_strings
             self._check_value(action, value)
-        
+
         # single argument or optional argument produces a single value
         elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
             arg_string, = arg_strings
             value = self._get_value(action, arg_string)
             self._check_value(action, value)
 
         # PARSER arguments convert all values, but check only the first
         elif action.nargs is PARSER:
-            value = list(self._get_value(action, v) for v in arg_strings)
+            value = [self._get_value(action, v) for v in arg_strings]
             self._check_value(action, value[0])
 
         # all other types of nargs produce a list
         else:
-            value = list(self._get_value(action, v) for v in arg_strings)
+            value = [self._get_value(action, v) for v in arg_strings]
             for v in value:
                 self._check_value(action, v)
 
-        # return the converted value            
+        # return the converted value
         return value
 
     def _get_value(self, action, arg_string):
         type_func = self._registry_get('type', action.type, action.type)
-        if not callable(type_func):
+        if not hasattr(type_func, '__call__'):
             msg = _('%r is not callable')
             raise ArgumentError(action, msg % type_func)
-        
+
         # convert the value to the appropriate type
         try:
             result = type_func(arg_string)
 
         # TypeErrors or ValueErrors indicate errors
         except (TypeError, ValueError):
             name = getattr(action.type, '__name__', repr(action.type))
             msg = _('invalid %s value: %r')
             raise ArgumentError(action, msg % (name, arg_string))
 
         # return the converted value
         return result
 
     def _check_value(self, action, value):
         # converted value must be one of the choices (if specified)
         if action.choices is not None and value not in action.choices:
             tup = value, ', '.join(map(repr, action.choices))
             msg = _('invalid choice: %r (choose from %s)') % tup
             raise ArgumentError(action, msg)
 
-        
-
     # =======================
     # Help-formatting methods
     # =======================
-
     def format_usage(self):
         formatter = self._get_formatter()
-        formatter.add_usage(self.usage,
-                            self._get_optional_actions(),
-                            self._get_positional_actions())
+        formatter.add_usage(self.usage, self._actions,
+                            self._mutually_exclusive_groups)
         return formatter.format_help()
 
     def format_help(self):
         formatter = self._get_formatter()
 
         # usage
-        formatter.add_usage(self.usage,
-                            self._get_optional_actions(),
-                            self._get_positional_actions())
+        formatter.add_usage(self.usage, self._actions,
+                            self._mutually_exclusive_groups)
 
-        # description        
+        # description
         formatter.add_text(self.description)
 
-        # positionals
-        formatter.start_section(_('positional arguments'))
-        formatter.add_arguments(self._positional_actions_list)
-        formatter.end_section()
-
-        # optionals
-        formatter.start_section(_('optional arguments'))
-        formatter.add_arguments(self._optional_actions_list)
-        formatter.end_section()
-
-        # user-defined groups
-        for argument_group in self._argument_groups:
-            formatter.start_section(argument_group.title)
-            formatter.add_text(argument_group.description)
-            formatter.add_arguments(argument_group._positional_actions_list)
-            formatter.add_arguments(argument_group._optional_actions_list)
+        # positionals, optionals and user-defined groups
+        for action_group in self._action_groups:
+            formatter.start_section(action_group.title)
+            formatter.add_text(action_group.description)
+            formatter.add_arguments(action_group._group_actions)
             formatter.end_section()
 
         # epilog
         formatter.add_text(self.epilog)
 
         # determine help from format above
         return formatter.format_help()
 
     def format_version(self):
         formatter = self._get_formatter()
         formatter.add_text(self.version)
         return formatter.format_help()
 
     def _get_formatter(self):
         return self.formatter_class(prog=self.prog)
 
     # =====================
     # Help-printing methods
     # =====================
-
     def print_usage(self, file=None):
         self._print_message(self.format_usage(), file)
 
     def print_help(self, file=None):
         self._print_message(self.format_help(), file)
 
     def print_version(self, file=None):
         self._print_message(self.format_version(), file)
 
     def _print_message(self, message, file=None):
         if message:
             if file is None:
                 file = _sys.stderr
             file.write(message)
 
-
     # ===============
     # Exiting methods
     # ===============
-
     def exit(self, status=0, message=None):
         if message:
             _sys.stderr.write(message)
         _sys.exit(status)
 
     def error(self, message):
         """error(message: string)
 
         Prints a usage message incorporating the message to stderr and
         exits.
-        
+
         If you override this in a subclass, it should not return -- it
         should either exit or raise an exception.
         """
         self.print_usage(_sys.stderr)
         self.exit(2, _('%s: error: %s\n') % (self.prog, message))

IPython/kernel/tests/engineservicetest.py

@@ -1,374 +1,377 b''
 # encoding: utf-8
 
 """Test template for complete engine object"""
 
 __docformat__ = "restructuredtext en"
 
 #-------------------------------------------------------------------------------
 #  Copyright (C) 2008  The IPython Development Team
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
 #-------------------------------------------------------------------------------
 
 #-------------------------------------------------------------------------------
 # Imports
 #-------------------------------------------------------------------------------
 
 import cPickle as pickle
 
 from twisted.internet import defer, reactor
 from twisted.python import failure
 from twisted.application import service
 import zope.interface as zi
 
 from IPython.kernel import newserialized
 from IPython.kernel import error
 from IPython.kernel.pickleutil import can, uncan
 import IPython.kernel.engineservice as es
 from IPython.kernel.core.interpreter import Interpreter
 from IPython.testing.parametric import Parametric, parametric
 
 #-------------------------------------------------------------------------------
 # Tests
 #-------------------------------------------------------------------------------
 
     
 # A sequence of valid commands run through execute
 validCommands = ['a=5',
                  'b=10',
                  'a=5; b=10; c=a+b',
                  'import math; 2.0*math.pi',
                  """def f():
     result = 0.0
     for i in range(10):
         result += i
 """,
                  'if 1<2: a=5',
                  """import time
 time.sleep(0.1)""",
                 """from math import cos;
 x = 1.0*cos(0.5)""", # Semicolons lead to Discard ast nodes that should be discarded
                 """s = 1
 s = set()
    """, # Trailing whitespace should be allowed.
                 """import math
 math.cos(1.0)""", # Test a method call with a discarded return value
                 """x=1.0234
 a=5; b=10""", # Test an embedded semicolon
                 """x=1.0234
 a=5; b=10;""" # Test both an embedded and trailing semicolon
                  ]
                  
 # A sequence of commands that raise various exceptions
 invalidCommands = [('a=1/0',ZeroDivisionError),
                    ('print v',NameError),
                    ('l=[];l[0]',IndexError),
                    ("d={};d['a']",KeyError),
                    ("assert 1==0",AssertionError),
                    ("import abababsdbfsbaljasdlja",ImportError),
                    ("raise Exception()",Exception)]
 
 def testf(x):
     return 2.0*x
 
 globala = 99
 
 def testg(x):
     return  globala*x
 
 class IEngineCoreTestCase(object):
     """Test an IEngineCore implementer."""
 
     def createShell(self):
         return Interpreter()
 
     def catchQueueCleared(self, f):
         try:
             f.raiseException()
         except error.QueueCleared:
             pass
         
     def testIEngineCoreInterface(self):
         """Does self.engine claim to implement IEngineCore?"""
         self.assert_(es.IEngineCore.providedBy(self.engine))
         
     def testIEngineCoreInterfaceMethods(self):
         """Does self.engine have the methods and attributes in IEngineCore."""
         for m in list(es.IEngineCore):
             self.assert_(hasattr(self.engine, m))
     
     def testIEngineCoreDeferreds(self):
         d = self.engine.execute('a=5')
         d.addCallback(lambda _: self.engine.pull('a'))
         d.addCallback(lambda _: self.engine.get_result())
         d.addCallback(lambda _: self.engine.keys())
         d.addCallback(lambda _: self.engine.push(dict(a=10)))
         return d
     
     def runTestExecute(self, cmd):
         self.shell = Interpreter()
         actual = self.shell.execute(cmd)
         def compare(computed):
             actual['id'] = computed['id']
             self.assertEquals(actual, computed)
         d = self.engine.execute(cmd)
         d.addCallback(compare)
         return d
     
     @parametric
     def testExecute(cls):
         return [(cls.runTestExecute, cmd) for cmd in validCommands]
 
     def runTestExecuteFailures(self, cmd, exc):
         def compare(f):
             self.assertRaises(exc, f.raiseException)
         d = self.engine.execute(cmd)
         d.addErrback(compare)
         return d
     
     @parametric
     def testExecuteFailuresEngineService(cls):
         return [(cls.runTestExecuteFailures, cmd, exc)
                 for cmd, exc in invalidCommands]
     
     def runTestPushPull(self, o):
         d = self.engine.push(dict(a=o))
         d.addCallback(lambda r: self.engine.pull('a'))
         d.addCallback(lambda r: self.assertEquals(o,r))
         return d
     
     @parametric
     def testPushPull(cls):
         objs = [10,"hi there",1.2342354,{"p":(1,2)},None]
         return [(cls.runTestPushPull, o) for o in objs]
         
     def testPullNameError(self):
         d = self.engine.push(dict(a=5))
         d.addCallback(lambda _:self.engine.reset())
         d.addCallback(lambda _: self.engine.pull("a"))
         d.addErrback(lambda f: self.assertRaises(NameError, f.raiseException))
         return d        
     
     def testPushPullFailures(self):
         d = self.engine.pull('a')
         d.addErrback(lambda f: self.assertRaises(NameError, f.raiseException))
         d.addCallback(lambda _: self.engine.execute('l = lambda x: x'))
         d.addCallback(lambda _: self.engine.pull('l'))
         d.addErrback(lambda f: self.assertRaises(pickle.PicklingError, f.raiseException))
         d.addCallback(lambda _: self.engine.push(dict(l=lambda x: x)))
         d.addErrback(lambda f: self.assertRaises(pickle.PicklingError, f.raiseException))
         return d
         
     def testPushPullArray(self):
         try:
             import numpy
         except:
             return
         a = numpy.random.random(1000)
         d = self.engine.push(dict(a=a))
         d.addCallback(lambda _: self.engine.pull('a'))
         d.addCallback(lambda b: b==a)
         d.addCallback(lambda c: c.all())
         return self.assertDeferredEquals(d, True)
         
     def testPushFunction(self):
                     
         d = self.engine.push_function(dict(f=testf))
         d.addCallback(lambda _: self.engine.execute('result = f(10)'))
         d.addCallback(lambda _: self.engine.pull('result'))
         d.addCallback(lambda r: self.assertEquals(r, testf(10)))
         return d
 
     def testPullFunction(self):
         d = self.engine.push_function(dict(f=testf, g=testg))
         d.addCallback(lambda _: self.engine.pull_function(('f','g')))
         d.addCallback(lambda r: self.assertEquals(r[0](10), testf(10)))
         return d
         
     def testPushFunctionGlobal(self):
         """Make sure that pushed functions pick up the user's namespace for globals."""
         d = self.engine.push(dict(globala=globala))
         d.addCallback(lambda _: self.engine.push_function(dict(g=testg)))
         d.addCallback(lambda _: self.engine.execute('result = g(10)'))
         d.addCallback(lambda _: self.engine.pull('result'))
         d.addCallback(lambda r: self.assertEquals(r, testg(10)))
         return d
         
     def testGetResultFailure(self):
         d = self.engine.get_result(None)
         d.addErrback(lambda f: self.assertRaises(IndexError, f.raiseException))
         d.addCallback(lambda _: self.engine.get_result(10))
         d.addErrback(lambda f: self.assertRaises(IndexError, f.raiseException))
         return d
 
     def runTestGetResult(self, cmd):
         self.shell = Interpreter()
         actual = self.shell.execute(cmd)
         def compare(computed):
             actual['id'] = computed['id']
             self.assertEquals(actual, computed)
         d = self.engine.execute(cmd)
         d.addCallback(lambda r: self.engine.get_result(r['number']))
         d.addCallback(compare)
         return d
     
     @parametric
     def testGetResult(cls):
         return [(cls.runTestGetResult, cmd) for cmd in validCommands]
     
     def testGetResultDefault(self):
         cmd = 'a=5'
         shell = self.createShell()
         shellResult = shell.execute(cmd)
         def popit(dikt, key):
             dikt.pop(key)
             return dikt
         d = self.engine.execute(cmd)
         d.addCallback(lambda _: self.engine.get_result())
         d.addCallback(lambda r: self.assertEquals(shellResult, popit(r,'id')))
         return d
 
     def testKeys(self):
         d = self.engine.keys()
         d.addCallback(lambda s: isinstance(s, list))
         d.addCallback(lambda r: self.assertEquals(r, True))
         return d
             
 Parametric(IEngineCoreTestCase)
             
 class IEngineSerializedTestCase(object):
     """Test an IEngineCore implementer."""
         
     def testIEngineSerializedInterface(self):
         """Does self.engine claim to implement IEngineCore?"""
         self.assert_(es.IEngineSerialized.providedBy(self.engine))
         
     def testIEngineSerializedInterfaceMethods(self):
         """Does self.engine have the methods and attributes in IEngineCore."""
         for m in list(es.IEngineSerialized):
             self.assert_(hasattr(self.engine, m))
        
     def testIEngineSerializedDeferreds(self):
         dList = []
         d = self.engine.push_serialized(dict(key=newserialized.serialize(12345)))
         self.assert_(isinstance(d, defer.Deferred))
         dList.append(d)
         d = self.engine.pull_serialized('key')
         self.assert_(isinstance(d, defer.Deferred))
         dList.append(d)
         D = defer.DeferredList(dList)
         return D
                 
     def testPushPullSerialized(self):
         objs = [10,"hi there",1.2342354,{"p":(1,2)}]
         d = defer.succeed(None)
         for o in objs:
             self.engine.push_serialized(dict(key=newserialized.serialize(o)))
             value = self.engine.pull_serialized('key')
             value.addCallback(lambda serial: newserialized.IUnSerialized(serial).getObject())
             d = self.assertDeferredEquals(value,o,d)
         return d
 
     def testPullSerializedFailures(self):
         d = self.engine.pull_serialized('a')
         d.addErrback(lambda f: self.assertRaises(NameError, f.raiseException))
         d.addCallback(lambda _: self.engine.execute('l = lambda x: x'))
         d.addCallback(lambda _: self.engine.pull_serialized('l'))
         d.addErrback(lambda f: self.assertRaises(pickle.PicklingError, f.raiseException))
         return d
 
 Parametric(IEngineSerializedTestCase)
 
 class IEngineQueuedTestCase(object):
     """Test an IEngineQueued implementer."""
         
     def testIEngineQueuedInterface(self):
         """Does self.engine claim to implement IEngineQueued?"""
         self.assert_(es.IEngineQueued.providedBy(self.engine))
         
     def testIEngineQueuedInterfaceMethods(self):
         """Does self.engine have the methods and attributes in IEngineQueued."""
         for m in list(es.IEngineQueued):
             self.assert_(hasattr(self.engine, m))
             
     def testIEngineQueuedDeferreds(self): 
         dList = []
         d = self.engine.clear_queue()
         self.assert_(isinstance(d, defer.Deferred))
         dList.append(d)
         d = self.engine.queue_status()
         self.assert_(isinstance(d, defer.Deferred))
         dList.append(d)
         D = defer.DeferredList(dList)
         return D
             
     def testClearQueue(self):
         result = self.engine.clear_queue()
         d1 = self.assertDeferredEquals(result, None)
         d1.addCallback(lambda _: self.engine.queue_status())
         d2 = self.assertDeferredEquals(d1, {'queue':[], 'pending':'None'})
         return d2
         
     def testQueueStatus(self):
         result = self.engine.queue_status()
         result.addCallback(lambda r: 'queue' in r and 'pending' in r)
         d = self.assertDeferredEquals(result, True)
         return d
 
 Parametric(IEngineQueuedTestCase)
 
 class IEnginePropertiesTestCase(object):
     """Test an IEngineProperties implementor."""
     
     def testIEnginePropertiesInterface(self):
         """Does self.engine claim to implement IEngineProperties?"""
         self.assert_(es.IEngineProperties.providedBy(self.engine))
     
     def testIEnginePropertiesInterfaceMethods(self):
         """Does self.engine have the methods and attributes in IEngineProperties."""
         for m in list(es.IEngineProperties):
             self.assert_(hasattr(self.engine, m))
     
     def testGetSetProperties(self):
         dikt = dict(a=5, b='asdf', c=True, d=None, e=range(5))
         d = self.engine.set_properties(dikt)
         d.addCallback(lambda r: self.engine.get_properties())
         d = self.assertDeferredEquals(d, dikt)
         d.addCallback(lambda r: self.engine.get_properties(('c',)))
         d = self.assertDeferredEquals(d, {'c': dikt['c']})
         d.addCallback(lambda r: self.engine.set_properties(dict(c=False)))
         d.addCallback(lambda r: self.engine.get_properties(('c', 'd')))
         d = self.assertDeferredEquals(d, dict(c=False, d=None))
         return d
     
     def testClearProperties(self):
         dikt = dict(a=5, b='asdf', c=True, d=None, e=range(5))
         d = self.engine.set_properties(dikt)
         d.addCallback(lambda r: self.engine.clear_properties())
         d.addCallback(lambda r: self.engine.get_properties())
         d = self.assertDeferredEquals(d, {})
         return d
     
     def testDelHasProperties(self):
         dikt = dict(a=5, b='asdf', c=True, d=None, e=range(5))
         d = self.engine.set_properties(dikt)
         d.addCallback(lambda r: self.engine.del_properties(('b','e')))
         d.addCallback(lambda r: self.engine.has_properties(('a','b','c','d','e')))
         d = self.assertDeferredEquals(d, [True, False, True, True, False])
         return d
     
     def testStrictDict(self):
-        s = """from IPython.kernel.engineservice import get_engine
-p = get_engine(%s).properties"""%self.engine.id
+        s = """from IPython.kernel.engineservice import get_engine; p = get_engine(%s).properties"""%self.engine.id
         d = self.engine.execute(s)
+        # These 3 lines cause a weird testing error on some platforms (OS X).
+        # I am leaving them here in case they are masking some really 
+        # weird reactor issue.  For now I will just keep my eye on this.
         d.addCallback(lambda r: self.engine.execute("p['a'] = lambda _:None"))
         d.addErrback(lambda f: self.assertRaises(error.InvalidProperty,
             f.raiseException))
+        # Below here seems to be fine
         d.addCallback(lambda r: self.engine.execute("p['a'] = range(5)"))
         d.addCallback(lambda r: self.engine.execute("p['a'].append(5)"))
         d.addCallback(lambda r: self.engine.get_properties('a'))
         d = self.assertDeferredEquals(d, dict(a=range(5)))
         return d
         
 Parametric(IEnginePropertiesTestCase)

docs/source/changes.txt

@@ -1,577 +1,544 b''
 .. Developers should add in this file, during each release cycle, information
 .. about important changes they've made, in a summary format that's meant for
 .. end users.  For each release we normally have three sections: features,  bug
 .. fixes and api breakage.
 .. Please remember to credit the authors of the contributions by name,
 .. especially when they are new users or developers who do not regularly
 .. participate  in IPython's development.
 
 .. _changes:
 
 ==========
 What's new
 ==========
 
-.. contents::
-..
-    1  Release dev
-      1.1  New features
-      1.2  Bug fixes
-      1.3  Backwards incompatible changes
-    2  Release 0.10
-      2.1  New features
-      2.2  Bug fixes
-      2.3  Backwards incompatible changes
-    3  Release 0.9.1
-    4  Release 0.9
-      4.1  New features
-      4.2  Bug fixes
-      4.3  Backwards incompatible changes
-      4.4  Changes merged in from IPython1
-        4.4.1  New features
-        4.4.2  Bug fixes
-        4.4.3  Backwards incompatible changes
-    5  Release 0.8.4
-    6  Release 0.8.3
-    7  Release 0.8.2
-    8  Older releases
-..
-
-Release dev
-===========
-
-New features
-------------
-
-Bug fixes
----------
-
-Backwards incompatible changes
-------------------------------
-
-
 Release 0.10
 ============
 
 This release brings months of slow but steady development, and will be the last
 before a major restructuring and cleanup of IPython's internals that is already
 under way.  For this reason, we hope that 0.10 will be a stable and robust
 release so that while users adapt to some of the API changes that will come
 with the refactoring that will become IPython 0.11, they can safely use 0.10 in
 all existing projects with minimal changes (if any).
 
 IPython 0.10 is now a medium-sized project, with roughly (as reported by David
 Wheeler's :command:`sloccount` utility) 40750 lines of Python code, and a diff
 between 0.9.1 and this release that contains almost 28000 lines of code and
 documentation.  Our documentation, in PDF format, is a 495-page long PDF
 document (also available in HTML format, both generated from the same sources).
 
 Many users and developers contributed code, features, bug reports and ideas to
 this release.  Please do not hesitate in contacting us if we've failed to
 acknowledge your contribution here.  In particular, for this release we have
 contribution from the following people, a mix of new and regular names (in
 alphabetical order by first name):
 
 * Alexander Clausen: fix #341726.
 * Brian Granger: lots of work everywhere (features, bug fixes, etc).
 * Daniel Ashbrook: bug report on MemoryError during compilation, now fixed.
 * Darren Dale: improvements to documentation build system, feedback, design
   ideas.
 * Fernando Perez: various places.
 * Gaël Varoquaux: core code, ipythonx GUI, design discussions, etc. Lots...
 * John Hunter: suggestions, bug fixes, feedback.
 * Jorgen Stenarson: work on many fronts, tests, fixes, win32 support, etc.
 * Laurent Dufréchou: many improvements to ipython-wx standalone app.
 * Lukasz Pankowski: prefilter, `%edit`, demo improvements.
 * Matt Foster: TextMate support in `%edit`.
 * Nathaniel Smith: fix #237073.
 * Pauli Virtanen: fixes and improvements to extensions, documentation.
 * Prabhu Ramachandran: improvements to `%timeit`.
 * Robert Kern: several extensions.
 * Sameer D'Costa: help on critical bug #269966.
 * Stephan Peijnik: feedback on Debian compliance and many man pages.
+* Steven Bethard: we are now shipping his :mod:`argparse` module.
 * Tom Fetherston: many improvements to :mod:`IPython.demo` module.
 * Ville Vainio: lots of work everywhere (features, bug fixes, etc).
 * Vishal Vasta: ssh support in ipcluster.
 * Walter Doerwald: work on the :mod:`IPython.ipipe` system.
 
 Below we give an overview of new features, bug fixes and backwards-incompatible
 changes.  For a detailed account of every change made, feel free to view the
 project log with :command:`bzr log`.
 
 New features
 ------------
 
 * New `%paste` magic automatically extracts current contents of clipboard and
   pastes it directly, while correctly handling code that is indented or
   prepended with `>>>` or `...` python prompt markers.  A very useful new
   feature contributed by Robert Kern.
 
 * IPython 'demos', created with the :mod:`IPython.demo` module, can now be
   created from files on disk or strings in memory.  Other fixes and
   improvements to the demo system, by Tom Fetherston.
 
 * Added :func:`find_cmd` function to :mod:`IPython.platutils` module, to find
   commands in a cross-platform manner.
 
 * Many improvements and fixes to Gaël Varoquaux's :command:`ipythonx`, a
   WX-based lightweight IPython instance that can be easily embedded in other WX
   applications.  These improvements have made it possible to now have an
   embedded IPython in Mayavi and other tools.
 
 * :class:`MultiengineClient` objects now have a :meth:`benchmark` method.
 
 * The manual now includes a full set of auto-generated API documents from the
   code sources, using Sphinx and some of our own support code.  We are now
   using the `Numpy Documentation Standard`_  for all docstrings, and we have
   tried to update as many existing ones as possible to this format.
 
-.. _Numpy Documentation Standard: http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard
-
 * The new :mod:`IPython.Extensions.ipy_pretty` extension by Robert Kern
   provides configurable pretty-printing.
 
 * Many improvements to the :command:`ipython-wx` standalone WX-based IPython
   application by Laurent Dufréchou.  It can optionally run in a thread, and
   this can be toggled at runtime (allowing the loading of Matplotlib in a
   running session without ill effects).
 
 * IPython includes a copy of Steven Bethard's argparse_ in the
   :mod:`IPython.external` package, so we can use it internally and it is also
   available to any IPython user.  By installing it in this manner, we ensure
   zero conflicts with any system-wide installation you may already have while
-  minimizing external dependencies for new users.
+  minimizing external dependencies for new users.  In IPython 0.10, We ship
+  argparse version 1.0.
 
 * An improved and much more robust test suite, that runs groups of tests in
   separate subprocesses using either Nose or Twisted's :command:`trial` runner
   to ensure proper management of Twisted-using code.  The test suite degrades
   gracefully if optional dependencies are not available, so that the
   :command:`iptest` command can be run with only Nose installed and nothing
   else.  We also have more and cleaner test decorators to better select tests
   depending on runtime conditions, do setup/teardown, etc.
 
 * The new ipcluster now has a fully working ssh mode that should work on
   Linux, Unix and OS X.  Thanks to Vishal Vatsa for implementing this!
 
 * The wonderful TextMate editor can now be used with %edit on OS X.  Thanks
   to Matt Foster for this patch.
 
 * The documentation regarding parallel uses of IPython, including MPI and PBS,
   has been significantly updated and improved.
 
 * The developer guidelines in the documentation have been updated to explain
   our workflow using :command:`bzr` and Launchpad.
   
 * Fully refactored :command:`ipcluster` command line program for starting
   IPython clusters.  This new version is a complete rewrite and 1) is fully
   cross platform (we now use Twisted's process management), 2) has much
   improved performance, 3) uses subcommands for different types of clusters, 4)
   uses argparse for parsing command line options, 5) has better support for
   starting clusters using :command:`mpirun`, 6) has experimental support for
   starting engines using PBS.  It can also reuse FURL files, by appropriately
   passing options to its subcommands.  However, this new version of ipcluster
   should be considered a technology preview.  We plan on changing the API in
   significant ways before it is final.
 
-* The :mod:`argparse` module has been added to :mod:`IPython.external`.
-
 * Full description of the security model added to the docs.
 
 * cd completer: show bookmarks if no other completions are available.
 
 * sh profile: easy way to give 'title' to prompt: assign to variable
   '_prompt_title'. It looks like this::
       
         [~]|1> _prompt_title = 'sudo!'
         sudo![~]|2>
 
 * %edit: If you do '%edit pasted_block', pasted_block variable gets updated
   with new data (so repeated editing makes sense)
 
+.. _Numpy Documentation Standard: http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard
+
+.. _argparse: http://code.google.com/p/argparse/
+
 Bug fixes
 ---------
 
 * Fix #368719, removed top-level debian/ directory to make the job of Debian
   packagers easier.
   
 * Fix #291143 by including man pages contributed by Stephan Peijnik from the
   Debian project.
 
 * Fix #358202, effectively a race condition, by properly synchronizing file
   creation at cluster startup time.
 
 * `%timeit` now handles correctly functions that take a long time to execute
   even the first time, by not repeating them.
 
 * Fix #239054, releasing of references after exiting.
 
 * Fix #341726, thanks to Alexander Clausen.
 
 * Fix #269966.  This long-standing and very difficult bug (which is actually a
   problem in Python itself) meant long-running sessions would inevitably grow
   in memory size, often with catastrophic consequences if users had large
   objects in their scripts.  Now, using `%run` repeatedly should not cause any
   memory leaks.  Special thanks to John Hunter and Sameer D'Costa for their
   help with this bug.
 
 * Fix #295371, bug in `%history`.
 
 * Improved support for py2exe.
 
 * Fix #270856: IPython hangs with PyGTK
 
 * Fix #270998: A magic with no docstring breaks the '%magic magic'
 
 * fix #271684: -c startup commands screw up raw vs. native history
 
 * Numerous bugs on Windows with the new ipcluster have been fixed.
 
 * The ipengine and ipcontroller scripts now handle missing furl files
   more gracefully by giving better error messages.
 
 * %rehashx: Aliases no longer contain dots. python3.0 binary
   will create alias python30. Fixes:
   #259716 "commands with dots in them don't work"
 
 * %cpaste: %cpaste -r repeats the last pasted block.
   The block is assigned to pasted_block even if code
   raises exception.
 
 * Bug #274067 'The code in get_home_dir is broken for py2exe' was
   fixed.
 
+* Many other small bug fixes not listed here by number (see the bzr log for
+  more info).
+  
 Backwards incompatible changes
 ------------------------------
 
 * `ipykit` and related files were unmaintained and have been removed.
 
 * The :func:`IPython.genutils.doctest_reload` does not actually call
   `reload(doctest)` anymore, as this was causing many problems with the test
   suite.  It still resets `doctest.master` to None.
 
 * While we have not deliberately broken Python 2.4 compatibility, only minor
   testing was done with Python 2.4, while 2.5 and 2.6 were fully tested.  But
   if you encounter problems with 2.4, please do report them as bugs.
 
 * The :command:`ipcluster` now requires a mode argument; for example to start a
   cluster on the local machine with 4 engines, you must now type::
 
     $ ipcluster local -n 4
 
 * The controller now has a ``-r`` flag that needs to be used if you want to
   reuse existing furl files.  Otherwise they are deleted (the default).
 
 * Remove ipy_leo.py. You can use :command:`easy_install ipython-extension` to
   get it.  (done to decouple it from ipython release cycle)
 
 
 Release 0.9.1
 =============
 
 This release was quickly made to restore compatibility with Python 2.4, which
 version 0.9 accidentally broke.  No new features were introduced, other than
 some additional testing support for internal use.
 
 
 Release 0.9
 ===========
 
 New features
 ------------
 
 * All furl files and security certificates are now put in a read-only
   directory named ~./ipython/security.
 
 * A single function :func:`get_ipython_dir`, in :mod:`IPython.genutils` that
   determines the user's IPython directory in a robust manner.
 
 * Laurent's WX application has been given a top-level script called
   ipython-wx, and it has received numerous fixes. We expect this code to be
   architecturally better integrated with Gael's WX 'ipython widget' over the
   next few releases.
 
 * The Editor synchronization work by Vivian De Smedt has been merged in.  This
   code adds a number of new editor hooks to synchronize with editors under
   Windows.
 
 * A new, still experimental but highly functional, WX shell by Gael Varoquaux.
   This work was sponsored by Enthought, and while it's still very new, it is
   based on a more cleanly organized arhictecture of the various IPython
   components. We will continue to develop this over the next few releases as a
   model for GUI components that use IPython.
 
 * Another GUI frontend, Cocoa based (Cocoa is the OSX native GUI framework),
   authored by Barry Wark.  Currently the WX and the Cocoa ones have slightly
   different internal organizations, but the whole team is working on finding
   what the right abstraction points are for a unified codebase.
 
 * As part of the frontend work, Barry Wark also implemented an experimental
   event notification system that various ipython components can use.  In the
   next release the implications and use patterns of this system regarding the
   various GUI options will be worked out.
 
 * IPython finally has a full test system, that can test docstrings with
   IPython-specific functionality.  There are still a few pieces missing for it
   to be widely accessible to all users (so they can run the test suite at any
   time and report problems), but it now works for the developers.  We are
   working hard on continuing to improve it, as this was probably IPython's
   major Achilles heel (the lack of proper test coverage made it effectively
   impossible to do large-scale refactoring).  The full test suite can now
   be run using the :command:`iptest` command line program.
 
 * The notion of a task has been completely reworked.  An `ITask` interface has
   been created.  This interface defines the methods that tasks need to
   implement.  These methods are now responsible for things like submitting
   tasks and processing results.  There are two basic task types:
   :class:`IPython.kernel.task.StringTask` (this is the old `Task` object, but
   renamed) and the new :class:`IPython.kernel.task.MapTask`, which is based on
   a function.
 
 * A new interface, :class:`IPython.kernel.mapper.IMapper` has been defined to
   standardize the idea of a `map` method.  This interface has a single `map`
   method that has the same syntax as the built-in `map`.  We have also defined
   a `mapper` factory interface that creates objects that implement
   :class:`IPython.kernel.mapper.IMapper` for different controllers.  Both the
   multiengine and task controller now have mapping capabilties.
 
 * The parallel function capabilities have been reworks.  The major changes are
   that i) there is now an `@parallel` magic that creates parallel functions,
   ii) the syntax for mulitple variable follows that of `map`, iii) both the
   multiengine and task controller now have a parallel function implementation.
 
 * All of the parallel computing capabilities from `ipython1-dev` have been
   merged into IPython proper.  This resulted in the following new subpackages:
   :mod:`IPython.kernel`, :mod:`IPython.kernel.core`, :mod:`IPython.config`,
   :mod:`IPython.tools` and :mod:`IPython.testing`.
 
 * As part of merging in the `ipython1-dev` stuff, the `setup.py` script and
   friends have been completely refactored.  Now we are checking for
   dependencies using the approach that matplotlib uses.
 
 * The documentation has been completely reorganized to accept the 
   documentation from `ipython1-dev`.
 
 * We have switched to using Foolscap for all of our network protocols in
   :mod:`IPython.kernel`.  This gives us secure connections that are both
   encrypted and authenticated.
 
 * We have a brand new `COPYING.txt` files that describes the IPython license
   and copyright. The biggest change is that we are putting "The IPython
   Development Team" as the copyright holder. We give more details about
   exactly what this means in this file. All developer should read this and use
   the new banner in all IPython source code files.
 
 * sh profile: ./foo runs foo as system command, no need to do !./foo anymore
 
 * String lists now support ``sort(field, nums = True)`` method (to easily sort
   system command output). Try it with ``a = !ls -l ; a.sort(1, nums=1)``.
 
 * '%cpaste foo' now assigns the pasted block as string list, instead of string
 
 * The ipcluster script now run by default with no security.  This is done
   because the main usage of the script is for starting things on localhost.
   Eventually when ipcluster is able to start things on other hosts, we will put
   security back.
 
 * 'cd --foo' searches directory history for string foo, and jumps to that dir.
   Last part of dir name is checked first. If no matches for that are found,
   look at the whole path.
 
   
 Bug fixes
 ---------
 
 * The Windows installer has been fixed.  Now all IPython scripts have ``.bat``
   versions created.  Also, the Start Menu shortcuts have been updated.
 
 * The colors escapes in the multiengine client are now turned off on win32 as
   they don't print correctly.
 
 * The :mod:`IPython.kernel.scripts.ipengine` script was exec'ing
   mpi_import_statement incorrectly, which was leading the engine to crash when
   mpi was enabled.
 
 * A few subpackages had missing ``__init__.py`` files.
 
 * The documentation is only created if Sphinx is found.  Previously, the
   ``setup.py`` script would fail if it was missing.
 
 * Greedy ``cd`` completion has been disabled again (it was enabled in 0.8.4) as
   it caused problems on certain platforms.
           
 
 Backwards incompatible changes
 ------------------------------
 
 * The ``clusterfile`` options of the :command:`ipcluster` command has been
   removed as it was not working and it will be replaced soon by something much
   more robust.
 
 * The :mod:`IPython.kernel` configuration now properly find the user's
   IPython directory.
 
 * In ipapi, the :func:`make_user_ns` function has been replaced with
   :func:`make_user_namespaces`, to support dict subclasses in namespace
   creation.
 
 * :class:`IPython.kernel.client.Task` has been renamed
   :class:`IPython.kernel.client.StringTask` to make way for new task types.
 
 * The keyword argument `style` has been renamed `dist` in `scatter`, `gather`
   and `map`.
 
 * Renamed the values that the rename `dist` keyword argument can have from
   `'basic'` to `'b'`.
 
 * IPython has a larger set of dependencies if you want all of its capabilities.
   See the `setup.py` script for details.
 
 * The constructors for :class:`IPython.kernel.client.MultiEngineClient` and 
   :class:`IPython.kernel.client.TaskClient` no longer take the (ip,port) tuple.
   Instead they take the filename of a file that contains the FURL for that
   client.  If the FURL file is in your IPYTHONDIR, it will be found automatically
   and the constructor can be left empty.
 
 * The asynchronous clients in :mod:`IPython.kernel.asyncclient` are now created 
   using the factory functions :func:`get_multiengine_client` and 
   :func:`get_task_client`.  These return a `Deferred` to the actual client.
 
 * The command line options to `ipcontroller` and `ipengine` have changed to
   reflect the new Foolscap network protocol and the FURL files.  Please see the
   help for these scripts for details.
 
 * The configuration files for the kernel have changed because of the Foolscap
   stuff.  If you were using custom config files before, you should delete them
   and regenerate new ones.
 
 Changes merged in from IPython1
 -------------------------------
 
 New features
 ............
 
 * Much improved ``setup.py`` and ``setupegg.py`` scripts.  Because Twisted and
   zope.interface are now easy installable, we can declare them as dependencies
   in our setupegg.py script.
 
 * IPython is now compatible with Twisted 2.5.0 and 8.x.
 
 * Added a new example of how to use :mod:`ipython1.kernel.asynclient`.
 
 * Initial draft of a process daemon in :mod:`ipython1.daemon`.  This has not
   been merged into IPython and is still in `ipython1-dev`.
   
 * The ``TaskController`` now has methods for getting the queue status.
 
 * The ``TaskResult`` objects not have information about how long the task
   took to run.
   
 * We are attaching additional attributes to exceptions ``(_ipython_*)`` that 
   we use to carry additional info around.
   
 * New top-level module :mod:`asyncclient` that has asynchronous versions (that
   return deferreds) of the client classes.  This is designed to users who want
   to run their own Twisted reactor.
   
 * All the clients in :mod:`client` are now based on Twisted.  This is done by 
   running the Twisted reactor in a separate thread and using the
   :func:`blockingCallFromThread` function that is in recent versions of Twisted.
 
 * Functions can now be pushed/pulled to/from engines using
   :meth:`MultiEngineClient.push_function` and
   :meth:`MultiEngineClient.pull_function`.
 
 * Gather/scatter are now implemented in the client to reduce the work load
   of the controller and improve performance.
 
 * Complete rewrite of the IPython docuementation.  All of the documentation
   from the IPython website has been moved into docs/source as restructured
   text documents.  PDF and HTML documentation are being generated using 
   Sphinx.
 
 * New developer oriented documentation: development guidelines and roadmap. 
 
 * Traditional ``ChangeLog`` has been changed to a more useful ``changes.txt``
   file that is organized by release and is meant to provide something more
   relevant for users.
 
 Bug fixes
 .........
 
 * Created a proper ``MANIFEST.in`` file to create source distributions.
 
 * Fixed a bug in the ``MultiEngine`` interface.  Previously, multi-engine 
   actions were being collected with a :class:`DeferredList` with 
   ``fireononeerrback=1``.  This meant that methods were returning 
   before all engines had given their results.  This was causing extremely odd 
   bugs in certain cases. To fix this problem, we have 1) set 
   ``fireononeerrback=0`` to make sure all results (or exceptions) are in 
   before returning and 2) introduced a :exc:`CompositeError` exception 
   that wraps all of the engine exceptions.  This is a huge change as it means 
   that users will have to catch :exc:`CompositeError` rather than the actual
   exception.
 
 Backwards incompatible changes
 ..............................
 
 * All names have been renamed to conform to the lowercase_with_underscore
   convention.  This will require users to change references to all names like
   ``queueStatus`` to ``queue_status``.
 
 * Previously, methods like :meth:`MultiEngineClient.push` and
   :meth:`MultiEngineClient.push` used ``*args`` and ``**kwargs``.  This was
   becoming a problem as we weren't able to introduce new keyword arguments into
   the API.  Now these methods simple take a dict or sequence.  This has also
   allowed us to get rid of the ``*All`` methods like :meth:`pushAll` and
   :meth:`pullAll`.  These things are now handled with the ``targets`` keyword
   argument that defaults to ``'all'``.
 
 * The :attr:`MultiEngineClient.magicTargets` has been renamed to
   :attr:`MultiEngineClient.targets`. 
 
 * All methods in the MultiEngine interface now accept the optional keyword
   argument ``block``.
 
 * Renamed :class:`RemoteController` to :class:`MultiEngineClient` and 
   :class:`TaskController` to :class:`TaskClient`.
 
 * Renamed the top-level module from :mod:`api` to :mod:`client`.
 
 * Most methods in the multiengine interface now raise a :exc:`CompositeError`
   exception that wraps the user's exceptions, rather than just raising the raw
   user's exception.
 
 * Changed the ``setupNS`` and ``resultNames`` in the ``Task`` class to ``push`` 
   and ``pull``.
 
   
 Release 0.8.4
 =============
 
 This was a quick release to fix an unfortunate bug that slipped into the 0.8.3
 release.  The ``--twisted`` option was disabled, as it turned out to be broken
 across several platforms.
 
 
 Release 0.8.3
 =============
   
 * pydb is now disabled by default (due to %run -d problems). You can enable
   it by passing -pydb command line argument to IPython. Note that setting
   it in config file won't work.
 
   
 Release 0.8.2
 =============
 
 * %pushd/%popd behave differently; now "pushd /foo" pushes CURRENT directory 
   and jumps to /foo. The current behaviour is closer to the documented 
   behaviour, and should not trip anyone.
 
   
 Older releases
 ==============
 
 Changes in earlier releases of IPython are described in the older file
 ``ChangeLog``.  Please refer to this document for details.
 

docs/source/development/overview.txt

@@ -1,494 +1,515 b''
 .. _development:
 
 ==============================
 IPython development guidelines
 ==============================
 
 
 Overview
 ========
 
 This document describes IPython from the perspective of developers. Most
 importantly, it gives information for people who want to contribute to the
 development of IPython. So if you want to help out, read on!
 
 How to contribute to IPython
 ============================
 
 IPython development is done using Bazaar [Bazaar]_ and Launchpad [Launchpad]_.
 This makes it easy for people to contribute to the development of IPython.
 There are several ways in which you can join in.
 
 If you have a small change that you want to send to the team, you can edit your
 bazaar checkout of IPython (see below) in-place, and ask bazaar for the
 differences::
 
     $ cd /path/to/your/copy/of/ipython
     $ bzr diff > my_fixes.diff
 
 This produces a patch file with your fixes, which we can apply to the source
 tree.  This file should then be attached to a ticket in our `bug tracker
 <https://bugs.launchpad.net/ipython>`_, indicating what it does.
 
 This model of creating small, self-contained patches works very well and there
 are open source projects that do their entire development this way.  However,
 in IPython we have found that for tracking larger changes, making use of
 bazaar's full capabilities in conjunction with Launchpad's code hosting
 services makes for a much better experience.
 
 Making your own branch of IPython allows you to refine your changes over time,
 track the development of the main team, and propose your own full version of
 the code for others to use and review, with a minimum amount of fuss.  The next
 parts of this document will explain how to do this.
 
 Install Bazaar and create a Launchpad account
 ---------------------------------------------
 
 First make sure you have installed Bazaar (see their `website
 <http://bazaar-vcs.org/>`_). To see that Bazaar is installed and knows about
 you, try the following::
 
     $ bzr whoami
     Joe Coder <[e-mail]>
 
 This should display your name and email. Next, you will want to create an
 account on the `Launchpad website <http://www.launchpad.net>`_ and setup your
 ssh keys. For more information of setting up your ssh keys, see `this link
 <https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair>`_.
 
 Get the main IPython branch from Launchpad
 ------------------------------------------
 
 Now, you can get a copy of the main IPython development branch (we call this
 the "trunk")::
 
     $ bzr branch lp:ipython
 
 Create a working branch
 -----------------------
 
 When working on IPython, you won't actually make edits directly to the
 :file:`lp:ipython` branch. Instead, you will create a separate branch for your
 changes. For now, let's assume you want to do your work in a branch named
 "ipython-mybranch". Create this branch by doing::
 
     $ bzr branch ipython ipython-mybranch
 
 When you actually create a branch, you will want to give it a name that
 reflects the nature of the work that you will be doing in it, like
 "install-docs-update".
 
 Make edits in your working branch
 ---------------------------------
 
 Now you are ready to actually make edits in your :file:`ipython-mybranch`
 branch. Before doing this, it is helpful to install this branch so you can
 test your changes as you work. This is easiest if you have setuptools
 installed. Then, just do::
 
     $ cd ipython-mybranch
     $ python setupegg.py develop
 
 Now, make some changes. After a while, you will want to commit your changes.
 This let's Bazaar know that you like the changes you have made and gives you
 an opportunity to keep a nice record of what you have done. This looks like
 this::
 
     $ ...do work in ipython-mybranch...
     $ bzr commit -m "the commit message goes here"
 
 Please note that since we now don't use an old-style linear ChangeLog (that
 tends to cause problems with distributed version control systems), you should
 ensure that your log messages are reasonably detailed.  Use a docstring-like
 approach in the commit messages (including the second line being left
 *blank*)::
 
   Single line summary of  changes being committed.
 
   * more details when warranted ...
   * including crediting outside contributors if they sent the
     code/bug/idea!
 
 As you work, you will repeat this edit/commit cycle many times. If you work on
 your branch for a long time, you will also want to get the latest changes from
 the :file:`lp:ipython` branch. This can be done with the following sequence of
 commands::
 
     $ ls
     ipython
     ipython-mybranch
     
     $ cd ipython
     $ bzr pull
     $ cd ../ipython-mybranch
     $ bzr merge ../ipython
     $ bzr commit -m "Merging changes from trunk"
 
 Along the way, you should also run the IPython test suite.  You can do this
 using the :command:`iptest` command (which is basically a customized version of
 :command:`nosetests`)::
 
     $ cd
     $ iptest
 
 The :command:`iptest` command will also pick up and run any tests you have
 written.  See :ref:`_devel_testing` for further details on the testing system.
 
 
 Post your branch and request a code review
 ------------------------------------------
 
 Once you are done with your edits, you should post your branch on Launchpad so
 that other IPython developers can review the changes and help you merge your
 changes into the main development branch. To post your branch on Launchpad,
 do::
 
     $ cd ipython-mybranch
     $ bzr push lp:~yourusername/ipython/ipython-mybranch
 
 Then, go to the `IPython Launchpad site <www.launchpad.net/ipython>`_, and you
 should see your branch under the "Code" tab. If you click on your branch, you
 can provide a short description of the branch as well as mark its status. Most
 importantly, you should click the link that reads "Propose for merging into
 another branch". What does this do?
 
 This let's the other IPython developers know that your branch is ready to be
 reviewed and merged into the main development branch. During this review
 process, other developers will give you feedback and help you get your code
 ready to be merged. What types of things will we be looking for:
 
 * All code is documented.
 * All code has tests.
 * The entire IPython test suite passes.
 
 Once your changes have been reviewed and approved, someone will merge them
 into the main development branch.
 
+
+Some notes for core developers when merging third-party contributions
+=====================================================================
+
+Core developers, who ultimately merge any approved branch (from themselves,
+another developer, or any third-party contribution) will typically use
+:command:`bzr merge` to merge the branch into the trunk and push it to the main
+Launcphad site.  This is a short list of things to keep in mind when doing this
+process, so that the project history is easy to understand in the long run, and
+that generating release notes is as painless and accurate as possible.
+
+- When you merge any non-trivial functionality (from one small bug fix to a big
+  feature branch), please remember to always edit the changes_ file
+  accordingly.  This file has one main section for each release, and if you
+  edit it as you go, noting what new features, bug fixes or API changes you
+  have made, the release notes will be almost finished when they are needed
+  later.  This is much easier if done when you merge the work, rather than
+  weeks or months later by re-reading a massive Bazaar log.
+
+- When big merges are done, the practice of putting a summary commit message in
+  the merge is *extremely* useful.  It makes this kind of job much nicer,
+  because that summary log message can be almost copy/pasted without changes,
+  if it was well written, rather than dissecting the next-level messages from
+  the individual commits.
+
+- It's important that we remember to always credit who gave us something if
+  it's not the committer.  In general, we have been fairly good on this front,
+  this is just a reminder to keep things up.  As a note, if you are ever
+  committing something that is completely (or almost so) a third-party
+  contribution, do the commit as::
+
+    $ bzr commit --author="Someone Else"
+
+  This way it will show that name separately in the log, which makes it even
+  easier to spot.  Obviously we often rework third party contributions
+  extensively, but this is still good to keep in mind for cases when we don't
+  touch the code too much.
+
+
 Documentation
 =============
 
 Standalone documentation
 ------------------------
 
 All standalone documentation should be written in plain text (``.txt``) files
 using reStructuredText [reStructuredText]_ for markup and formatting. All such
 documentation should be placed in directory :file:`docs/source` of the IPython
 source tree. The documentation in this location will serve as the main source
 for IPython documentation and all existing documentation should be converted
 to this format.
 
 To build the final documentation, we use Sphinx [Sphinx]_.  Once you have
 Sphinx installed, you can build the html docs yourself by doing::
 
     $ cd ipython-mybranch/docs
     $ make html
 
 Docstring format
 ----------------
 
 Good docstrings are very important. All new code should have docstrings that
 are formatted using reStructuredText for markup and formatting, since it is
 understood by a wide variety of tools. Details about using reStructuredText
 for docstrings can be found `here
 <http://epydoc.sourceforge.net/manual-othermarkup.html>`_.
 
 Additional PEPs of interest regarding documentation of code:
 
 * `Docstring Conventions <http://www.python.org/peps/pep-0257.html>`_
 * `Docstring Processing System Framework <http://www.python.org/peps/pep-0256.html>`_
 * `Docutils Design Specification <http://www.python.org/peps/pep-0258.html>`_
 
 
 Coding conventions
 ==================
 
 General
 -------
 
 In general, we'll try to follow the standard Python style conventions as
 described here:
 
 * `Style Guide for Python Code  <http://www.python.org/peps/pep-0008.html>`_  
 
 
 Other comments:
 
 * In a large file, top level classes and functions should be 
   separated by 2-3 lines to make it easier to separate them visually.
 * Use 4 spaces for indentation.
 * Keep the ordering of methods the same in classes that have the same 
   methods.  This is particularly true for classes that implement an interface.
 
 Naming conventions
 ------------------
 
 In terms of naming conventions, we'll follow the guidelines from the `Style
 Guide for Python Code`_.
 
 For all new IPython code (and much existing code is being refactored), we'll
 use:
 
 * All ``lowercase`` module names.
 
 * ``CamelCase`` for class names.
 
 * ``lowercase_with_underscores`` for methods, functions, variables and
   attributes.
 
 There are, however, some important exceptions to these rules. In some cases,
 IPython code will interface with packages (Twisted, Wx, Qt) that use other
 conventions. At some level this makes it impossible to adhere to our own
 standards at all times. In particular, when subclassing classes that use other
 naming conventions, you must follow their naming conventions. To deal with
 cases like this, we propose the following policy:
 
 * If you are subclassing a class that uses different conventions, use its
   naming conventions throughout your subclass.  Thus, if you are creating a
   Twisted Protocol class, used Twisted's
   ``namingSchemeForMethodsAndAttributes.``
 
 * All IPython's official interfaces should use our conventions.  In some cases
   this will mean that you need to provide shadow names (first implement
   ``fooBar`` and then ``foo_bar = fooBar``).  We want to avoid this at all
   costs, but it will probably be necessary at times.  But, please use this
   sparingly!
 
 Implementation-specific *private* methods will use
 ``_single_underscore_prefix``. Names with a leading double underscore will
 *only* be used in special cases, as they makes subclassing difficult (such
 names are not easily seen by child classes).
 
 Occasionally some run-in lowercase names are used, but mostly for very short
 names or where we are implementing methods very similar to existing ones in a
 base class (like ``runlines()`` where ``runsource()`` and ``runcode()`` had
 established precedent).
 
 The old IPython codebase has a big mix of classes and modules prefixed with an
 explicit ``IP``. In Python this is mostly unnecessary, redundant and frowned
 upon, as namespaces offer cleaner prefixing. The only case where this approach
 is justified is for classes which are expected to be imported into external
 namespaces and a very generic name (like Shell) is too likely to clash with
 something else. We'll need to revisit this issue as we clean up and refactor
 the code, but in general we should remove as many unnecessary ``IP``/``ip``
 prefixes as possible. However, if a prefix seems absolutely necessary the more
 specific ``IPY`` or ``ipy`` are preferred.
 
 .. _devel_testing:
 
 Testing system
 ==============
 
 It is extremely important that all code contributed to IPython has tests.
 Tests should be written as unittests, doctests or as entities that the Nose
 [Nose]_ testing package will find. Regardless of how the tests are written, we
 will use Nose for discovering and running the tests. Nose will be required to
 run the IPython test suite, but will not be required to simply use IPython.
 
 Tests of Twisted using code need to follow two additional guidelines:
 
 1. Twisted using tests should be written by subclassing the :class:`TestCase`
    class that comes with :mod:`twisted.trial.unittest`. 
 
 2. All :class:`Deferred` instances that are created in the test must be
    properly chained and the final one *must* be the return value of the test
    method.
 
 When these two things are done, Nose will be able to run the tests and the
 twisted reactor will be handled correctly.
 
 Each subpackage in IPython should have its own :file:`tests` directory that
 contains all of the tests for that subpackage. This allows each subpackage to
 be self-contained.  A good convention to follow is to have a file named
 :file:`test_foo.py` for each module :file:`foo.py` in the package.  This makes
 it easy to organize the tests, though like most conventions, it's OK to break
 it if logic and common sense dictate otherwise.
 
 If a subpackage has any dependencies beyond the Python standard library, the
 tests for that subpackage should be skipped if the dependencies are not
 found. This is very important so users don't get tests failing simply because
 they don't have dependencies.  We ship a set of decorators in the
 :mod:`IPython.testing` package to tag tests that may be platform-specific or
 otherwise may have restrictions; if the existing ones don't fit your needs, add
 a new decorator in that location so other tests can reuse it.
 
 To run the IPython test suite, use the :command:`iptest` command that is
 installed with IPython (if you are using IPython in-place, without installing
 it, you can find this script in the :file:`scripts` directory)::
 
     $ iptest
 
-This command runs Nose with the proper options and extensions.  By default,
-:command:`iptest` runs the entire IPython test suite (skipping tests that may
-be platform-specific or which depend on tools you may not have).  But you can
-also use it to run only one specific test file, or a specific test function.
-For example, this will run only the :file:`test_magic` file from the test
-suite::
+This command colects all IPython tests into separate groups, and then calls
+either Nose with the proper options and extensions, or Twisted's
+:command:`trial`.  This ensures that tests that need the Twisted reactor
+management facilities execute separate of Nose.  If any individual test group
+fails, :command:`iptest` will print what you need to type so you can rerun that
+particular test group alone for debugging.
+
+By default, :command:`iptest` runs the entire IPython test
+suite (skipping tests that may be platform-specific or which depend on tools
+you may not have).  But you can also use it to run only one specific test file,
+or a specific test function.  For example, this will run only the
+:file:`test_magic` file from the test suite::
 
     $ iptest IPython.tests.test_magic
     ----------------------------------------------------------------------
     Ran 10 tests in 0.348s
 
     OK (SKIP=3)
     Deleting object: second_pass
 
 while the ``path:function`` syntax allows you to select a specific function in
 that file to run::
 
     $ iptest IPython.tests.test_magic:test_obj_del
     ----------------------------------------------------------------------
     Ran 1 test in 0.204s
 
     OK
 
 Since :command:`iptest` is based on nosetests, you can pass it any regular
 nosetests option.  For example, you can use ``--pdb`` or ``--pdb-failures`` to
 automatically activate the interactive Pdb debugger on errors or failures.  See
 the nosetests documentation for further details.
 
-.. warning::
-
-   Note that right now we have a nasty interaction between ipdoctest and
-   twisted.  Until we figure this out, please use the following instructions to
-   ensure that at least you run all the tests.
-
-Right now, if you now run::
-
-    $ iptest [any options] [any submodules]
-
-it will NOT load ipdoctest but won't cause any Twisted problems.
-
-Once you're happy that you didn't break Twisted, run::
-
-    $ iptest --with-ipdoctest [any options] [any submodules]
-
-This MAY give a Twisted AlreadyCalledError exception at the end, but it will
-also correctly load up all of the ipython-specific tests and doctests.
-
-The above can be made easier with a trivial shell alias::
-
-    $ alias iptest2='iptest --with-ipdoctest'
-
-So that you can run::
-
-    $ iptest ...
-    # Twisted happy
-    # iptest2 ...
-    # ignore possible Twisted error, this checks all the rest.
-
 
 A few tips for writing tests
 ----------------------------
 
 You can write tests either as normal test files, using all the conventions that
 Nose recognizes, or as doctests.  Note that *all* IPython functions should have
 at least one example that serves as a doctest, whenever technically feasible.
 However, example doctests should only be in the main docstring if they are *a
 good example*, i.e. if they convey useful information about the function.  If
 you simply would like to write a test as a doctest, put it in a separate test
 file and write a no-op function whose only purpose is its docstring.
 
 Note, however, that in a file named :file:`test_X`, functions whose only test
 is their docstring (as a doctest) and which have no test functionality of their
 own, should be called *doctest_foo* instead of *test_foo*, otherwise they get
 double-counted (the empty function call is counted as a test, which just
 inflates tests numbers artificially).  This restriction does not apply to
 functions in files with other names, due to how Nose discovers tests.
 
 You can use IPython examples in your docstrings.  Those can make full use of
 IPython functionality (magics, variable substitution, etc), but be careful to
 keep them generic enough that they run identically on all Operating Systems.
 
 The prompts in your doctests can be either of the plain Python ``>>>`` variety
 or ``In [1]:`` IPython style.  Since this is the IPython system, after all, we
 encourage you to use IPython prompts throughout, unless you are illustrating a
 specific aspect of the normal prompts (such as the ``%doctest_mode`` magic).
 
 If a test isn't safe to run inside the main nose process (e.g. because it loads
 a GUI toolkit), consider running it in a subprocess and capturing its output
 for evaluation and test decision later.  Here is an example of how to do it, by
 relying on the builtin ``_ip`` object that contains the public IPython api as
 defined in :mod:`IPython.ipapi`::
 
    def test_obj_del():
        """Test that object's __del__ methods are called on exit."""
        test_dir = os.path.dirname(__file__)
        del_file = os.path.join(test_dir,'obj_del.py')
        out = _ip.IP.getoutput('ipython %s' % del_file)
        nt.assert_equals(out,'object A deleted')
 
 
 
 If a doctest contains input whose output you don't want to verify identically
 via doctest (random output, an object id, etc), you can mark a docstring with
 ``#random``.  All of these test will have their code executed but no output
 checking will be done::
 
        >>> 1+3
        junk goes here...  # random
 
        >>> 1+2
        again,  anything goes #random
        if multiline, the random mark is only needed once.
 
        >>> 1+2
        You can also put the random marker at the end:
        # random
 
        >>> 1+2
        # random
        .. or at the beginning.
 
 In a case where you want an *entire* docstring to be executed but not verified
 (this only serves to check that the code runs without crashing, so it should be
 used very sparingly), you can put ``# all-random`` in the docstring.
        
 .. _devel_config:
 
 Release checklist
 =================
 
 Most of the release process is automated by the :file:`release` script in the
 :file:`tools` directory.  This is just a handy reminder for the release manager.
 
+#. First, run :file:`build_release`, which does all the file checking and
+   building that the real release script will do.  This will let you do test
+   installations, check that the build procedure runs OK, etc.  You may want to
+   disable a few things like multi-version RPM building while testing, because
+   otherwise the build takes really long.
+
 #. Run the release script, which makes the tar.gz, eggs and Win32 .exe
    installer.  It posts them to the site and registers the release with PyPI.
 
 #. Updating the website with announcements and links to the updated
    changes.txt in html form. Remember to put a short note both on the news
    page of the site and on Launcphad.
 
 #. Drafting a short release announcement with i) highlights and ii) a link to
    the html changes.txt.
 
 #. Make sure that the released version of the docs is live on the site.
 
 #. Celebrate!
 
 Porting to 3.0
 ==============
 
 There are no definite plans for porting of IPython to python 3. The major 
 issue is the dependency on twisted framework for the networking/threading 
 stuff. It is possible that it the traditional IPython interactive console 
 could be ported more easily since it has no such dependency. Here are a few
 things that will need to be considered when doing such a port especially
 if we want to have a codebase that works directly on both 2.x and 3.x.
 
 	1. The syntax for exceptions changed (PEP 3110). The old 
 	   `except exc, var` changed to `except exc as var`. At last
 	   count there was 78 occurences of this usage in the codebase.  This
 	   is  a particularly problematic issue, because it's not easy to
 	   implement it in a 2.5-compatible way.
 
 Because it is quite difficult to support simultaneously Python 2.5 and 3.x, we
 will likely at some point put out a release that requires strictly 2.6 and
 abandons 2.5 compatibility.  This will then allow us to port the code to using
 :func:`print` as a function, `except exc as var` syntax, etc.  But as of
 version 0.11 at least, we will retain Python 2.5 compatibility.
 
 
 .. [Bazaar] Bazaar. http://bazaar-vcs.org/
 .. [Launchpad] Launchpad. http://www.launchpad.net/ipython
 .. [reStructuredText] reStructuredText.  http://docutils.sourceforge.net/rst.html
 .. [Sphinx] Sphinx. http://sphinx.pocoo.org/
 .. [Nose] Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/