Merge pull request #9200 from Carreau/rc2

Factor build process (and gen what's new)

docs/source/whatsnew/development.rst

@@ -1,39 +1,23 b''
 =====================
  Development version
 =====================
 
 This document describes in-flight development work.
 
 .. warning::
 
     Please do not edit this file by hand (doing so will likely cause merge
     conflicts for other Pull Requests). Instead, create a new file in the
     `docs/source/whatsnew/pr` folder
 
 
-IPython debugger (IPdb) now supports the number of context lines for the
-``where`` (and ``w``) commands. The `context` keyword is also available in various APIs.
-See PR :ghpull:`9097`
 
-YouTube video will now show thumbnail when exported to a media that do not support video. (:ghpull:`9086`)
-
-Add warning when running `ipython <subcommand>` when subcommand is deprecated. `jupyter` should now be used.
-
-Code in `%pinfo` (also known as `??`) are now highlighter (:ghpull:`8947`)
-
-`%aimport` now support module completion. (:ghpull:`8884`)
-
-`ipdb` output is now colored ! (:ghpull:`8842`)
-
-Add ability to transpose columns for completion: (:ghpull:`8748`)
-
-Many many docs improvements and bug fixes, you can see the list of changes [add link to GitHub diff]
 
 .. DO NOT EDIT THIS LINE BEFORE RELEASE. FEATURE INSERTION POINT.
 
 
 Backwards incompatible changes
 ------------------------------
 
 
-.. DO NOT EDIT THIS LINE BEFORE RELEASE. INCOMPAT INSERTION POINT.
  No newline at end of file
+.. DO NOT EDIT THIS LINE BEFORE RELEASE. INCOMPAT INSERTION POINT.

docs/source/whatsnew/version4.rst

@@ -1,46 +1,69 b''
 ============
  4.x Series
 ============
 
+
+IPython 4.1
+===========
+
+Release February, 2016. IPython 4.1 contain mostly bug fixes. It though contain
+a few improvement.
+
+
+- IPython debugger (IPdb) now supports the number of context lines for the
+  ``where`` (and ``w``) commands. The `context` keyword is also available in
+  various APIs. See PR :ghpull:`9097`
+- YouTube video will now show thumbnail when exported to a media that do not
+  support video. (:ghpull:`9086`)
+- Add warning when running `ipython <subcommand>` when subcommand is
+  deprecated. `jupyter` should now be used.
+- Code in `%pinfo` (also known as `??`) are now highlighter (:ghpull:`8947`)
+- `%aimport` now support module completion. (:ghpull:`8884`)
+- `ipdb` output is now colored ! (:ghpull:`8842`)
+- Add ability to transpose columns for completion: (:ghpull:`8748`)
+
+Many many docs improvements and bug fixes, you can see the
+`list of changes <https://github.com/ipython/ipython/compare/4.0.0...4.1.0>`_
+
 IPython 4.0
 ===========
 
 Released August, 2015
 
 IPython 4.0 is the first major release after the Big Split.
 IPython no longer contains the notebook, qtconsole, etc. which have moved to
 `jupyter <https://jupyter.readthedocs.org>`_.
 IPython subprojects, such as `IPython.parallel <https://ipyparallel.readthedocs.org>`_ and `widgets <https://ipywidgets.readthedocs.org>`_ have moved to their own repos as well.
 
 The following subpackages are deprecated:
 
 - IPython.kernel (now jupyter_client and ipykernel)
 - IPython.consoleapp (now jupyter_client.consoleapp)
 - IPython.nbformat (now nbformat)
 - IPython.nbconvert (now nbconvert)
 - IPython.html (now notebook)
 - IPython.parallel (now ipyparallel)
 - IPython.utils.traitlets (now traitlets)
 - IPython.config (now traitlets.config)
 - IPython.qt (now qtconsole)
 - IPython.terminal.console (now jupyter_console)
 
 and a few other utilities.
 
 Shims for the deprecated subpackages have been added,
 so existing code should continue to work with a warning about the new home.
 
 There are few changes to the code beyond the reorganization and some bugfixes.
 
 IPython highlights:
 
 - Public APIs for discovering IPython paths is moved from :mod:`IPython.utils.path` to :mod:`IPython.paths`.
   The old function locations continue to work with deprecation warnings.
 - Code raising ``DeprecationWarning``
   entered by the user in an interactive session will now display the warning by
   default. See :ghpull:`8480` an :ghissue:`8478`.
 - The `--deep-reload` flag and the corresponding options to inject `dreload` or
   `reload` into the interactive namespace have been deprecated, and will be
   removed in future versions. You should now explicitly import `reload` from
   `IPython.lib.deepreload` to use it.
 

tools/build_release

@@ -1,28 +1,32 b''
 #!/usr/bin/env python
 """IPython release build script.
 """
-
 import os
 from shutil import rmtree
 
-from toollib import sh, pjoin, get_ipdir, cd, compile_tree, execfile, sdists, wheels
+from toollib import sh, pjoin, get_ipdir, cd, compile_tree, execfile, sdists, buildwheels
+
+def build_release():
+
+    # Get main ipython dir, this will raise if it doesn't pass some checks
+    ipdir = get_ipdir()
+    cd(ipdir)
 
-# Get main ipython dir, this will raise if it doesn't pass some checks
-ipdir = get_ipdir()
-cd(ipdir)
+    # Load release info
+    execfile(pjoin('IPython', 'core', 'release.py'), globals())
 
-# Load release info
-execfile(pjoin('IPython', 'core', 'release.py'), globals())
+    # Check that everything compiles
+    compile_tree('*')
 
-# Check that everything compiles
-compile_tree('*')
+    # Cleanup
+    for d in ['build', 'dist', pjoin('docs', 'build'), pjoin('docs', 'dist'),
+              pjoin('docs', 'source', 'api', 'generated')]:
+        if os.path.isdir(d):
+            rmtree(d)
 
-# Cleanup
-for d in ['build', 'dist', pjoin('docs', 'build'), pjoin('docs', 'dist'),
-          pjoin('docs', 'source', 'api', 'generated')]:
-    if os.path.isdir(d):
-        rmtree(d)
+    # Build source and binary distros
+    sh(sdists)
+    buildwheels()
 
-# Build source and binary distros
-sh(sdists)
-sh(wheels)
+if __name__ == '__main__':
+    build_release()

tools/release

@@ -1,88 +1,88 b''
 #!/usr/bin/env python
 """IPython release script.
 
 This should ONLY be run at real release time.
 """
 from __future__ import print_function
 
 import os
 import sys
 
 from toollib import (get_ipdir, pjoin, cd, execfile, sh, archive,
-        sdists, archive_user, archive_dir)
+        sdists, archive_user, archive_dir, buildwheels)
 from gh_api import post_download
 
 # Get main ipython dir, this will raise if it doesn't pass some checks
 ipdir = get_ipdir()
 tooldir = pjoin(ipdir, 'tools')
 distdir = pjoin(ipdir, 'dist')
 
 # Where I keep static backups of each release
 ipbackupdir = os.path.expanduser('~/ipython/backup')
 if not os.path.exists(ipbackupdir):
     os.makedirs(ipbackupdir)
 
 # Start in main IPython dir
 cd(ipdir)
 
 # Load release info
 version = None
 execfile(pjoin('IPython','core','release.py'), globals())
 
 # Build site addresses for file uploads
 release_site = '%s/release/%s' % (archive, version)
 backup_site =  '%s/backup/' % archive
 
 # Start actual release process
 print()
 print('Releasing IPython')
 print('=================')
 print()
 print('Version:', version)
 print()
 print('Source IPython directory:', ipdir)
 print()
 
 # Perform local backup, go to tools dir to run it.
 cd(tooldir)
 sh('./make_tarball.py')
 sh('mv ipython-*.tgz %s' % ipbackupdir)
 
 # Build release files
 sh('./build_release %s' % ipdir)
 
 # Not Registering with PyPI, registering with setup.py is insecure as communication is not encrypted
 cd(ipdir)
 
 # Upload all files
 sh(sdists)
-for py in ('2', '3'):
-    sh('python%s setupegg.py bdist_wheel' % py)
+
+buildwheels()
 
 if 'upload' not in sys.argv:
     print("`./release upload` to register and release")
     sys.exit(0)
 
 
 print('Will not upload with setuptools as upload connection is insecure. Please use `twine upload dist/*` to upload the files to PyPI')
 
 cd(distdir)
 print( 'Uploading distribution files to GitHub...')
 
 for fname in os.listdir('.'):
     # TODO: update to GitHub releases API
     continue
     print('uploading %s to GitHub' % fname)
     desc = "IPython %s source distribution" % version
     post_download("ipython/ipython", fname, description=desc)
 
 # Make target dir if it doesn't exist
 print('Uploading IPython to backup site.')
 sh('ssh %s "mkdir -p %s/release/%s" ' % (archive_user, archive_dir, version))
 sh('scp * %s' % release_site)
 
 print( 'Uploading backup files...')
 cd(ipbackupdir)
 sh('scp `ls -1tr *tgz | tail -1` %s' % backup_site)
 
 print('Done!')

tools/toollib.py

@@ -1,64 +1,66 b''
 """Various utilities common to IPython release and maintenance tools.
 """
 from __future__ import print_function
 
 # Library imports
 import os
 
 # Useful shorthands
 pjoin = os.path.join
 cd = os.chdir
 
 # Constants
 
 # SSH root address of the archive site
 archive_user = '[e-mail]'
 archive_dir = 'archive.ipython.org'
 archive = '%s:%s' % (archive_user, archive_dir)
 
 # Build commands
 # Source dists
 sdists = './setup.py sdist --formats=gztar,zip'
 # Binary dists
-wheels = './setupegg.py bdist_wheel'
+def buildwheels():
+    for py in ('2', '3'):
+        sh('python%s setupegg.py bdist_wheel' % py)
 
 # Utility functions
 def sh(cmd):
     """Run system command in shell, raise SystemExit if it returns an error."""
     print("$", cmd)
     stat = os.system(cmd)
     #stat = 0  # Uncomment this and comment previous to run in debug mode
     if stat:
         raise SystemExit("Command %s failed with code: %s" % (cmd, stat))
 
 # Backwards compatibility
 c = sh
 
 def get_ipdir():
     """Get IPython directory from command line, or assume it's the one above."""
 
     # Initialize arguments and check location
     ipdir = pjoin(os.path.dirname(__file__), os.pardir)
 
     ipdir = os.path.abspath(ipdir)
 
     cd(ipdir)
     if not os.path.isdir('IPython') and os.path.isfile('setup.py'):
         raise SystemExit('Invalid ipython directory: %s' % ipdir)
     return ipdir
 
 
 def compile_tree(folder='.'):
     """Compile all Python files below current directory."""
     stat = os.system('python -m compileall {}'.format(folder))
     if stat:
         msg = '*** ERROR: Some Python files in tree do NOT compile! ***\n'
         msg += 'See messages above for the actual file that produced it.\n'
         raise SystemExit(msg)
 
 try:
     execfile
 except NameError:
     def execfile(fname, globs, locs=None):
         locs = locs or globs
         exec(compile(open(fname).read(), fname, "exec"), globs, locs)

tools/update_whatsnew.py

@@ -1,78 +1,82 b''
 #!/usr/bin/env python
 """Update the What's New doc (development version)
 
 This collects the snippets from whatsnew/pr/, moves their content into
 whatsnew/development.rst (chronologically ordered), and deletes the snippets.
 """
 
 import io
-import os
+import sys
 from glob import glob
 from os.path import dirname, basename, abspath, join as pjoin
 from subprocess import check_call, check_output
 
 repo_root = dirname(dirname(abspath(__file__)))
 whatsnew_dir = pjoin(repo_root, 'docs', 'source', 'whatsnew')
 pr_dir = pjoin(whatsnew_dir, 'pr')
 target = pjoin(whatsnew_dir, 'development.rst')
 
 FEATURE_MARK = ".. DO NOT EDIT THIS LINE BEFORE RELEASE. FEATURE INSERTION POINT."
 INCOMPAT_MARK = ".. DO NOT EDIT THIS LINE BEFORE RELEASE. INCOMPAT INSERTION POINT."
 
 # 1. Collect the whatsnew snippet files ---------------------------------------
 
 files = set(glob(pjoin(pr_dir, '*.rst')))
 # Ignore explanatory and example files
 files.difference_update({pjoin(pr_dir, f) for f in {
                          'incompat-switching-to-perl.rst',
                          'antigravity-feature.rst'}
                          })
 
+if not files:
+    print("No automatic update available for what's new")
+    sys.exit(0)
+
 
 def getmtime(f):
     return check_output(['git', 'log', '-1', '--format="%ai"', '--', f])
 
 files = sorted(files, key=getmtime)
 
 features, incompats = [], []
 for path in files:
     with io.open(path, encoding='utf-8') as f:
         try:
             content = f.read().rstrip()
         except Exception as e:
             raise Exception('Error reading "{}"'.format(f)) from e
 
     if basename(path).startswith('incompat-'):
         incompats.append(content)
     else:
         features.append(content)
 
 # Put the insertion markers back on the end, so they're ready for next time.
 feature_block = '\n\n'.join(features + [FEATURE_MARK])
 incompat_block = '\n\n'.join(incompats + [INCOMPAT_MARK])
 
 # 2. Update the target file ---------------------------------------------------
 
 with io.open(target, encoding='utf-8') as f:
     content = f.read()
 
 assert content.count(FEATURE_MARK) == 1
 assert content.count(INCOMPAT_MARK) == 1
 
 content = content.replace(FEATURE_MARK, feature_block)
 content = content.replace(INCOMPAT_MARK, incompat_block)
 
 # Clean trailing whitespace
 content = '\n'.join(l.rstrip() for l in content.splitlines())
 
 with io.open(target, 'w', encoding='utf-8') as f:
     f.write(content)
 
 # 3. Stage the changes in git -------------------------------------------------
 
 for file in files:
     check_call(['git', 'rm', file])
 
 check_call(['git', 'add', target])
 
 print("Merged what's new changes. Check the diff and commit the change.")