Merge pull request #10241 from takluyver/release-integrate-twine

Upload to PyPI in release upload script

docs/source/coredev/release_process.rst

@@ -1,259 +1,246 b''
 .. _release_process:
 
 =======================
 IPython release process
 =======================
 
 This document contains the process that is used to create an IPython release.
 
 Conveniently, the ``release`` script in the ``tools`` directory of the ``IPython``
 repository automates most of the release process. This document serves as a
 handy reminder and checklist for the release manager. 
 
 During the release process, you might need the extra following dependencies:
 
  - ``keyring`` to access your GitHub authentication tokens
  - ``graphviz`` to generate some graphs in the documentation
  
 Make sure you have all the required dependencies to run the tests as well.
  
 
 1. Set Environment variables
 ----------------------------
 
 Set environment variables to document previous release tag, current
 release milestone, current release version, and git tag.
 
 These variables may be used later to copy/paste as answers to the script
 questions instead of typing the appropriate command when the time comes. These
 variables are not used by the scripts directly; therefore, there is no need to
 ``export`` them. The format for bash is as follows, but note that these values
 are just an example valid only for the 5.0 release; you'll need to update them
 for the release you are actually making::
 
     PREV_RELEASE=4.2.1
     MILESTONE=5.0
     VERSION=5.0.0
     BRANCH=master
 
 
 2. Create GitHub stats and finish release note
 ----------------------------------------------
 
 .. note::
 
     This step is optional if making a Beta or RC release.
 
 .. note::
 
     Before generating the GitHub stats, verify that all closed issues and pull
     requests have `appropriate milestones
     <https://github.com/ipython/ipython/wiki/Dev:-GitHub-workflow#milestones>`_.
     `This search
     <https://github.com/ipython/ipython/issues?q=is%3Aclosed+no%3Amilestone+is%3Aissue>`_
     should return no results before creating the GitHub stats.
 
 If a major release:
 
     - merge any pull request notes into what's new::
 
           python tools/update_whatsnew.py
 
     - update ``docs/source/whatsnew/development.rst``, to ensure it covers
       the major release features
 
     - move the contents of ``development.rst`` to ``versionX.rst`` where ``X`` is
       the numerical release version
 
     - generate summary of GitHub contributions, which can be done with::
 
           python tools/github_stats.py --milestone $MILESTONE > stats.rst
 
       which may need some manual cleanup of ``stats.rst``. Add the cleaned
       ``stats.rst`` results to ``docs/source/whatsnew/github-stats-X.rst``
       where ``X`` is the numerical release version (don't forget to add it to
       the git repo as well). If creating a major release, make a new
       ``github-stats-X.rst`` file; if creating a minor release, the content
       from ``stats.rst`` may simply be added to the top of an existing
       ``github-stats-X.rst`` file.  Finally, edit
       ``docs/source/whatsnew/index.rst`` to list the new ``github-stats-X``
       file you just created and remove temporarily the first entry called
       ``development`` (you'll need to add it back after release).
 
       Make sure that the stats file has a header or it won't be rendered in
       the final documentation. 
 
 To find duplicates and update `.mailmap`, use::
 
     git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
 
 If a minor release you might need to do some of the above points manually, and
 forward port the changes.
 
 3. Make sure the repository is clean
 ------------------------------------
 
 of any file that could be problematic.
    Remove all non-tracked files with:
 
    .. code::
 
        git clean -xfdi
 
    This will ask for confirmation before removing all untracked files. Make
    sure the ``dist/`` folder is clean to avoid any stale builds from
    previous build attempts.
 
 
 4. Update the release version number
 ------------------------------------
 
 Edit ``IPython/core/release.py`` to have the current version.
 
 in particular, update version number and ``_version_extra`` content in
 ``IPython/core/release.py``.
 
 Step 5 will validate your changes automatically, but you might still want to
 make sure the version number matches pep440.
 
 In particular, ``rc`` and ``beta`` are not separated by ``.`` or the ``sdist``
 and ``bdist`` will appear as different releases. For example, a valid version
 number for a release candidate (rc) release is: ``1.3rc1``. Notice that there
 is no separator between the '3' and the 'r'. Check the environment variable
 ``$VERSION`` as well.
 
 You will likely just have to modify/comment/uncomment one of the lines setting
 ``_version_extra``
 
 
 5. Run the `tools/build_release` script
 ---------------------------------------
 
 Running ``tools/build_release`` does all the file checking and building that
 the real release script will do. This makes test installations, checks that
 the build procedure runs OK, and tests other steps in the release process.
 
 The ``build_release`` script will in particular verify that the version number
 match PEP 440, in order to avoid surprise at the time of build upload.
 
 We encourage creating a test build of the docs as well.
 
 6. Create and push the new tag
 ------------------------------
 
 Commit the changes to release.py::
 
     git commit -am "release $VERSION"
     git push origin $BRANCH
 
 Create and push the tag::
 
     git tag -am "release $VERSION" "$VERSION"
     git push origin --tags
 
 Update release.py back to ``x.y-dev`` or ``x.y-maint``, and re-add the
 ``development`` entry in ``docs/source/whatsnew/index.rst`` and push::
 
     git commit -am "back to development"
     git push origin $BRANCH
 
 7. Get a fresh clone
 --------------------
 
 Get a fresh clone of the tag for building the release::
 
     cd /tmp
     git clone --depth 1 https://github.com/ipython/ipython.git -b "$VERSION"
     cd ipython
 
 .. note::
 
-    You can aslo cleanup the current working repository with ``git clean -xfdi``
+    You can also cleanup the current working repository with ``git clean -xfdi``
 
 8. Run the release script
 -------------------------
 
-.. important::
-
-    These steps cover instructions for creating releases of IPython 5.x LTS and
-    IPython 6.x. Ignore release steps for Python 2 when releasing IPython 6.x
-    which no longer supports Python 2. 
-
 Run the ``release`` script, this step requires having a current wheel, Python
 >=3.4 and Python 2.7.::
 
     ./tools/release
 
 This makes the tarballs and wheels, and puts them under the ``dist/``
 folder. Be sure to test the ``wheels``  and the ``sdist`` locally before
 uploading them to PyPI. We do not use an universal wheel as each wheel
 installs an ``ipython2`` or ``ipython3`` script, depending on the version of
 Python it is built for. Using an universal wheel would prevent this.
 
 Use the following to actually upload the result of the build::
 
     ./tools/release upload
 
-It should posts them to ``archive.ipython.org``.
-
-You will need to use `twine <https://github.com/pypa/twine>`_ ) manually to
-actually upload on PyPI. Unlike setuptools, twine is able to upload packages
-over SSL::
-
-    twine upload dist/*
-
+It should posts them to ``archive.ipython.org`` and to PyPI.
 
 PyPI/Warehouse will automatically hide previous releases. If you are uploading
 a non-stable version, make sure to log-in to PyPI and un-hide previous version.
 
 
 9. Draft a short release announcement
 -------------------------------------
 
 The announcement should include:
 
 - release highlights
 - a link to the html version of the *What's new* section of the documentation
 - a link to upgrade or installation tips (if necessary)
 
 Post the announcement to the mailing list and or blog, and link from Twitter.
 
 .. note::
 
     If you are doing a RC or Beta, you can likely skip the next steps.
 
 10. Update milestones on GitHub
 -------------------------------
 
 These steps will bring milestones up to date:
 
 - close the just released milestone
 - open a new milestone for the next release (x, y+1), if the milestone doesn't
   exist already
 
 11. Update the IPython website
 ------------------------------
 
 The IPython website should document the new release:
 
 - add release announcement (news, announcements)
 - update current version and download links
 - update links on the documentation page (especially if a major release)
 
 12. Update readthedocs
 ----------------------
 
 Make sure to update readthedocs and set the latest tag as stable, as well as
 checking that previous release is still building under its own tag. 
 
 13. Update the Conda-Forge feedstock
 ------------------------------------
 
 Follow the instructions on `the repository <https://github.com/conda-forge/ipython-feedstock>`_
 
 14. Celebrate!
 --------------
 
 Celebrate the release and please thank the contributors for their work. Great
 job!
 

tools/release

@@ -1,89 +1,94 b''
 #!/usr/bin/env python
 """IPython release script.
 
 This should ONLY be run at real release time.
 """
 from __future__ import print_function
 
 import os
+from subprocess import call
 import sys
 
 from toollib import (get_ipdir, pjoin, cd, execfile, sh, archive,
         sdists, archive_user, archive_dir, buildwheels)
 from gh_api import post_download
 
 # Get main ipython dir, this will raise if it doesn't pass some checks
 ipdir = get_ipdir()
 tooldir = pjoin(ipdir, 'tools')
 distdir = pjoin(ipdir, 'dist')
 
 # Where I keep static backups of each release
 ipbackupdir = os.path.expanduser('~/ipython/backup')
 if not os.path.exists(ipbackupdir):
     os.makedirs(ipbackupdir)
 
 # Start in main IPython dir
 cd(ipdir)
 
 # Load release info
 version = None
 execfile(pjoin('IPython','core','release.py'), globals())
 
 # Build site addresses for file uploads
 release_site = '%s/release/%s' % (archive, version)
 backup_site =  '%s/backup/' % archive
 
 # Start actual release process
 print()
 print('Releasing IPython')
 print('=================')
 print()
 print('Version:', version)
 print()
 print('Source IPython directory:', ipdir)
 print()
 
 # Perform local backup, go to tools dir to run it.
 cd(tooldir)
 
 if 'upload' in sys.argv:
     cd(distdir)
-    print( 'Uploading distribution files to GitHub...')
+    #print( 'Uploading distribution files to GitHub...')
 
     for fname in os.listdir('.'):
         # TODO: update to GitHub releases API
         continue
         print('uploading %s to GitHub' % fname)
         desc = "IPython %s source distribution" % version
         post_download("ipython/ipython", fname, description=desc)
 
     # Make target dir if it doesn't exist
-    print('Uploading IPython to backup site.')
+    print('1. Uploading IPython to archive.ipython.org')
     sh('ssh %s "mkdir -p %s/release/%s" ' % (archive_user, archive_dir, version))
     sh('scp * %s' % release_site)
 
-    print( 'Uploading backup files...')
+    print('2. Uploading backup files...')
     cd(ipbackupdir)
     sh('scp `ls -1tr *tgz | tail -1` %s' % backup_site)
 
-    print('Done!')
-    print('Use `twine upload dist/*` to upload the files to PyPI')
+    print('3. Uploading to PyPI using twine')
+    cd(distdir)
+    call(['twine', 'upload'] + os.listdir('.'))
+
 else:
+    # Build, but don't upload
+
+    # Make backup tarball
     sh('./make_tarball.py')
     sh('mv ipython-*.tgz %s' % ipbackupdir)
 
     # Build release files
     sh('./build_release %s' % ipdir)
 
-    # Not Registering with PyPI, registering with setup.py is insecure as communication is not encrypted
     cd(ipdir)
 
     # Upload all files
     sh(sdists)
 
     buildwheels()
-    print("`./release upload` to upload source distribution on github and ipython archive")
+    print("`./release upload` to upload source distribution on PyPI and ipython archive")
     sys.exit(0)