Backport PR #9211: Release process tweaks

Things I ran into while releasing 4.1.1.

The only one that might be controversial is removing the `python -m compileall` check before we build packages. Because this is hardcoded to `python`, it runs on Python 2 on my system, and fails on one of our tools scripts that happens to be Python 3 only. We could make this smarter, but I don't think it's worth it. We now have continuous integration and editors with built in static analysis, so if we make a file invalid Python syntax we'll know about it well before release.

I also ran into a problem in that I don't have access to archive.ipython.org (I'm sure I did on one of my other computers, but I forget which). There should probably be a check for that somewhere earlier in the build process, but I haven't included it here.

docs/source/coredev/release_process.rst

@@ -1,185 +1,185 b''
 .. _release_process:
 
 =======================
 IPython release process
 =======================
 
 This document contains the process that is used to create an IPython release.
 
 Conveniently, the `release` script in the `tools` directory of the `IPython`
 repository automates most of the release process. This document serves as a
 handy reminder and checklist for the release manager.
 
 1. Set Environment variables
 ----------------------------
 
 Set environment variables to document previous release tag, current
 release milestone, current release version, and git tag::
 
     PREV_RELEASE=rel-1.0.0
     MILESTONE=1.1
     VERSION=1.1.0
     TAG="rel-$VERSION"
     BRANCH=master
 
 These variables may be used later to copy/paste as answers to the script
 questions instead of typing the appropriate command when the time comes. These
 variables are not used by the scripts directly; therefore, there is no need to
 `export` the variables.
 
 2. Create GitHub stats and finish release note
 ----------------------------------------------
 
 .. note::
 
     Before generating the GitHub stats, verify that all closed issues and
     pull requests have `appropriate milestones <https://github.com/ipython/ipython/wiki/Dev%3A-GitHub-workflow#milestones>`_.
     `This search <https://github.com/ipython/ipython/issues?q=is%3Aclosed+no%3Amilestone+is%3Aissue>`_
     should return no results before creating the GitHub stats.
 
 If a major release:
 
     - merge any pull request notes into what's new::
 
           python tools/update_whatsnew.py
 
     - update `docs/source/whatsnew/development.rst`, to ensure it covers
       the major release features
     - move the contents of `development.rst` to `versionX.rst` where `X` is
       the numerical release version
     - generate summary of GitHub contributions, which can be done with::
 
           python tools/github_stats.py --milestone $MILESTONE > stats.rst
 
       which may need some manual cleanup of `stats.rst`. Add the cleaned
       `stats.rst` results to `docs/source/whatsnew/github-stats-X.rst` where
       `X` is the numerical release version. If creating a major release, make
       a new `github-stats-X.rst` file; if creating a minor release, the
       content from `stats.rst` may simply be added to the top of an existing
       `github-stats-X.rst` file.
 
 To find duplicates and update `.mailmap`, use::
 
     git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
 
 3. Make sure the repository is clean
 ------------------------------------
 
 of any file that could be problematic.
    Remove all non-tracked files with:
 
    .. code::
 
        git clean -xfdi
 
    This will ask for confirmation before removing all untracked files. Make
    sure the ``dist/`` folder is clean to avoid any stale builds from
    previous build attempts.
 
 
 
 4. Run the `tools/build_release` script
 ---------------------------------------
 
 Running `tools/build_release` does all the file checking and building that
 the real release script will do. This makes test installations, checks that
 the build procedure runs OK, and tests other steps in the release process.
 
 We encourage creating a test build of the docs as well.
 
 5. Create and push the new tag
 ------------------------------
 
 Edit `IPython/core/release.py` to have the current version.
 
 in particular, update version number and ``_version_extra`` content in
 ``IPython/core/release.py``.
 
 Make sure the version number matches pep440, in particular, `rc` and `beta` are
 not separated by `.` or the `sdist` and `bdist` will appear as different
 releases. For example, a valid version number for a release candidate (rc)
 release is: ``1.3rc1``. Notice that there is no separator between the '3' and
 the 'r'.
 
 
-Commit the changes to release.py and jsversion::
+Commit the changes to release.py::
 
     git commit -am "release $VERSION"
     git push origin $BRANCH
 
 Create and push the tag::
 
     git tag -am "release $VERSION" "$TAG"
     git push origin --tags
 
 Update release.py back to `x.y-dev` or `x.y-maint`, and push::
 
     git commit -am "back to development"
     git push origin $BRANCH
 
 6. Get a fresh clone
 --------------------
 
 Get a fresh clone of the tag for building the release::
 
     cd /tmp
     git clone --depth 1 https://github.com/ipython/ipython.git -b "$TAG"
 
 7. Run the release script
 -------------------------
 
 Run the `release` script, this step requires having a current wheel, Python >=3.4 and Python 2.7.::
 
     cd tools && ./release
 
 This makes the tarballs, zipfiles, and wheels, and put them under the `dist/`
 folder. Be sure to test the ``wheel`` and the ``sdist`` locally before uploading
 them to PyPI. 
 
 Use the following to actually upload the result of the build:
 
     ./release upload
 
 It should posts them to ``archive.ipython.org`` and registers the release
 with PyPI if you have the various authorisations. 
 
 You might need to use `twine <https://github.com/pypa/twine>`_ (`twine upload
 dist/*`) manually to actually upload on PyPI. Unlike setuptools, twine is able
 to upload packages over SSL.
 
 
 8. Draft a short release announcement
 -------------------------------------
 
 The announcement should include:
 
 - release highlights
 - a link to the html version of the *What's new* section of the documentation
 - a link to upgrade or installation tips (if necessary)
 
 Post the announcement to the mailing list and or blog, and link from Twitter.
 
 9. Update milestones on GitHub
 ------------------------------
 
 These steps will bring milestones up to date:
 
 - close the just released milestone
 - open a new milestone for the next release (x, y+1), if the milestone doesn't
   exist already
 
 10. Update the IPython website
 ------------------------------
 
 The IPython website should document the new release:
 
 - add release announcement (news, announcements)
 - update current version and download links
 - update links on the documentation page (especially if a major release)
 
 11. Celebrate!
 --------------
 
 Celebrate the release and please thank the contributors for their work. Great
 job!
 

tools/build_release

@@ -1,32 +1,29 b''
 #!/usr/bin/env python
 """IPython release build script.
 """
 import os
 from shutil import rmtree
 
-from toollib import sh, pjoin, get_ipdir, cd, compile_tree, execfile, sdists, buildwheels
+from toollib import sh, pjoin, get_ipdir, cd, execfile, sdists, buildwheels
 
 def build_release():
 
     # Get main ipython dir, this will raise if it doesn't pass some checks
     ipdir = get_ipdir()
     cd(ipdir)
 
     # Load release info
     execfile(pjoin('IPython', 'core', 'release.py'), globals())
 
-    # Check that everything compiles
-    compile_tree('*')
-
     # Cleanup
     for d in ['build', 'dist', pjoin('docs', 'build'), pjoin('docs', 'dist'),
               pjoin('docs', 'source', 'api', 'generated')]:
         if os.path.isdir(d):
             rmtree(d)
 
     # Build source and binary distros
     sh(sdists)
     buildwheels()
 
 if __name__ == '__main__':
     build_release()

tools/toollib.py

@@ -1,66 +1,57 b''
 """Various utilities common to IPython release and maintenance tools.
 """
 from __future__ import print_function
 
 # Library imports
 import os
 
 # Useful shorthands
 pjoin = os.path.join
 cd = os.chdir
 
 # Constants
 
 # SSH root address of the archive site
 archive_user = '[e-mail]'
 archive_dir = 'archive.ipython.org'
 archive = '%s:%s' % (archive_user, archive_dir)
 
 # Build commands
 # Source dists
 sdists = './setup.py sdist --formats=gztar,zip'
 # Binary dists
 def buildwheels():
     for py in ('2', '3'):
         sh('python%s setupegg.py bdist_wheel' % py)
 
 # Utility functions
 def sh(cmd):
     """Run system command in shell, raise SystemExit if it returns an error."""
     print("$", cmd)
     stat = os.system(cmd)
     #stat = 0  # Uncomment this and comment previous to run in debug mode
     if stat:
         raise SystemExit("Command %s failed with code: %s" % (cmd, stat))
 
 # Backwards compatibility
 c = sh
 
 def get_ipdir():
     """Get IPython directory from command line, or assume it's the one above."""
 
     # Initialize arguments and check location
     ipdir = pjoin(os.path.dirname(__file__), os.pardir)
 
     ipdir = os.path.abspath(ipdir)
 
     cd(ipdir)
     if not os.path.isdir('IPython') and os.path.isfile('setup.py'):
         raise SystemExit('Invalid ipython directory: %s' % ipdir)
     return ipdir
 
-
-def compile_tree(folder='.'):
-    """Compile all Python files below current directory."""
-    stat = os.system('python -m compileall {}'.format(folder))
-    if stat:
-        msg = '*** ERROR: Some Python files in tree do NOT compile! ***\n'
-        msg += 'See messages above for the actual file that produced it.\n'
-        raise SystemExit(msg)
-
 try:
-    execfile
+    execfile = execfile
 except NameError:
     def execfile(fname, globs, locs=None):
         locs = locs or globs
         exec(compile(open(fname).read(), fname, "exec"), globs, locs)