Update the release process to attempt reproducible builds.

This include some refactoring of the current scripts as well as the
introduction of a script to repack the tar.gz in order to remove a
couple of artifact the build environment. Mostly timestamps, extra
files, and groups ownership of files.

tools/retar.py

@@ -0,0 +1,60 b''
+"""
+Un-targz and retargz a targz file to ensure reproducible build. 
+
+usage: 
+
+    $ export SOURCE_DATE_EPOCH=$(date +%s)
+    ...
+    $ python retar.py <tarfile.gz>
+
+The process of creating an sdist can be non-reproducible:
+  - directory created during the process get a mtime of the creation date; 
+  - gziping files embed the timestamp of fo zip creation. 
+
+This will untar-retar; ensuring that all mtime > SOURCE_DATE_EPOCH will be set
+equal to SOURCE_DATE_EPOCH.
+
+"""
+
+import tarfile
+import sys
+import os
+import gzip
+import io
+
+if len(sys.argv) > 2:
+    raise ValueError("Too many arguments")
+
+
+timestamp = int(os.environ["SOURCE_DATE_EPOCH"])
+
+old_buf = io.BytesIO()
+with open(sys.argv[1], "rb") as f:
+    old_buf.write(f.read())
+old_buf.seek(0)
+old = tarfile.open(fileobj=old_buf, mode="r:gz")
+
+buf = io.BytesIO()
+new = tarfile.open(fileobj=buf, mode="w", format=tarfile.GNU_FORMAT)
+for i, m in enumerate(old):
+    data = None
+    # mutation does not work, copy
+    if m.name.endswith('.DS_Store'):
+        continue
+    m2 = tarfile.TarInfo(m.name)
+    m2.mtime = min(timestamp, m.mtime)
+    m2.size = m.size
+    m2.type = m.type
+    m2.linkname = m.linkname
+    if m.isdir():
+        data = old.extractfile(m)
+        new.addfile(m2, data)
+    else:
+        new.addfile(m2)
+new.close()
+old.close()
+
+buf.seek(0)
+with open(sys.argv[1], "wb") as f:
+    with gzip.GzipFile('', "wb", fileobj=f, mtime=timestamp) as gzf:
+        gzf.write(buf.read())

docs/source/coredev/index.rst

@@ -1,303 +1,320 b''
 .. _core_developer_guide:
 
 =================================
 Guide for IPython core Developers
 =================================
 
 This guide documents the development of IPython itself.  Alternatively,
 developers of third party tools and libraries that use IPython should see the
 :doc:`../development/index`.
 
 
 For instructions on how to make a developer install see :ref:`devinstall`.
 
 Backporting Pull requests
 =========================
 
 All pull requests should usually be made against ``master``, if a Pull Request
 need to be backported to an earlier release; then it should be tagged with the
 correct ``milestone``.
 
 If you tag a pull request with a milestone **before** merging the pull request,
 and the base ref is ``master``, then our backport bot should automatically create
 a corresponding pull-request that backport on the correct branch.
 
 If you have write access to the IPython repository you can also just mention the
 **backport bot** to do the work for you. The bot is evolving so instructions may
 be different. At the time of this writing you can use::
 
     @meeseeksdev[bot] backport [to] <branchname>
 
 The bot will attempt to backport the current pull-request and issue a PR if
 possible. 
 
 .. note::
 
     The ``@`` and ``[bot]`` when mentioning the bot should be optional and can
     be omitted.
 
 If the pull request cannot be automatically backported, the bot should tell you
 so on the PR and apply a "Need manual backport" tag to the origin PR.
 
 .. _release_process:
 
 IPython release process
 =======================
 
 This document contains the process that is used to create an IPython release.
 
 Conveniently, the ``release`` script in the ``tools`` directory of the ``IPython``
 repository automates most of the release process. This document serves as a
 handy reminder and checklist for the release manager.
 
 During the release process, you might need the extra following dependencies:
 
  - ``keyring`` to access your GitHub authentication tokens
  - ``graphviz`` to generate some graphs in the documentation
  - ``ghpro`` to generate the stats
 
 Make sure you have all the required dependencies to run the tests as well.
 
 You can try to ``source tools/release_helper.sh`` when releasing via bash, it 
 should guide you through most of the process.
 
 
 1. Set Environment variables
 ----------------------------
 
 Set environment variables to document previous release tag, current
 release milestone, current release version, and git tag.
 
 These variables may be used later to copy/paste as answers to the script
 questions instead of typing the appropriate command when the time comes. These
 variables are not used by the scripts directly; therefore, there is no need to
 ``export`` them. The format for bash is as follows, but note that these values
 are just an example valid only for the 5.0 release; you'll need to update them
 for the release you are actually making::
 
     PREV_RELEASE=4.2.1
     MILESTONE=5.0
     VERSION=5.0.0
     BRANCH=master
 
+For `reproducibility of builds <https://reproducible-builds.org/specs/source-date-epoch/>`_,
+we recommend setting ``SOURCE_DATE_EPOCH`` prior to running the build; record the used value
+of ``SOURCE_DATE_EPOCH`` as it may not be available from build artifact. You
+should be able to use ``date +%s`` to get a formatted timestamp::
+
+    SOURCE_DATE_EPOCH=$(date +%s)
+
 
 2. Create GitHub stats and finish release note
 ----------------------------------------------
 
 .. note::
 
     This step is optional if making a Beta or RC release.
 
 .. note::
 
     Before generating the GitHub stats, verify that all closed issues and pull
     requests have `appropriate milestones
     <https://github.com/ipython/ipython/wiki/Dev:-GitHub-workflow#milestones>`_.
     `This search
     <https://github.com/ipython/ipython/issues?q=is%3Aclosed+no%3Amilestone+is%3Aissue>`_
     should return no results before creating the GitHub stats.
 
 If a major release:
 
     - merge any pull request notes into what's new::
 
           python tools/update_whatsnew.py
 
     - update ``docs/source/whatsnew/development.rst``, to ensure it covers
       the major release features
 
     - move the contents of ``development.rst`` to ``versionX.rst`` where ``X`` is
       the numerical release version
 
     - generate summary of GitHub contributions, which can be done with::
 
           python tools/github_stats.py --milestone $MILESTONE > stats.rst
 
       which may need some manual cleanup of ``stats.rst``. Add the cleaned
       ``stats.rst`` results to ``docs/source/whatsnew/github-stats-X.rst``
       where ``X`` is the numerical release version (don't forget to add it to
       the git repository as well). If creating a major release, make a new
       ``github-stats-X.rst`` file; if creating a minor release, the content
       from ``stats.rst`` may simply be added to the top of an existing
       ``github-stats-X.rst`` file.
 
     - Edit ``docs/source/whatsnew/index.rst`` to list the new ``github-stats-X``
       file you just created.
 
     - You do not need to temporarily remove the first entry called
       ``development``, nor re-add it after the release, it will automatically be
       hidden when releasing a stable version of IPython (if ``_version_extra``
       in ``release.py`` is an empty string.
 
       Make sure that the stats file has a header or it won't be rendered in
       the final documentation.
 
 To find duplicates and update `.mailmap`, use::
 
     git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
 
 If a minor release you might need to do some of the above points manually, and
 forward port the changes.
 
 3. Make sure the repository is clean
 ------------------------------------
 
 of any file that could be problematic.
    Remove all non-tracked files with:
 
    .. code::
 
        git clean -xfdi
 
    This will ask for confirmation before removing all untracked files. Make
    sure the ``dist/`` folder is clean to avoid any stale builds from
    previous build attempts.
 
 
 4. Update the release version number
 ------------------------------------
 
 Edit ``IPython/core/release.py`` to have the current version.
 
 in particular, update version number and ``_version_extra`` content in
 ``IPython/core/release.py``.
 
 Step 5 will validate your changes automatically, but you might still want to
 make sure the version number matches pep440.
 
 In particular, ``rc`` and ``beta`` are not separated by ``.`` or the ``sdist``
 and ``bdist`` will appear as different releases. For example, a valid version
 number for a release candidate (rc) release is: ``1.3rc1``. Notice that there
 is no separator between the '3' and the 'r'. Check the environment variable
 ``$VERSION`` as well.
 
 You will likely just have to modify/comment/uncomment one of the lines setting
 ``_version_extra``
 
 
 5. Run the `tools/build_release` script
 ---------------------------------------
 
 Running ``tools/build_release`` does all the file checking and building that
 the real release script will do. This makes test installations, checks that
 the build procedure runs OK, and tests other steps in the release process.
 
 The ``build_release`` script will in particular verify that the version number
 match PEP 440, in order to avoid surprise at the time of build upload.
 
 We encourage creating a test build of the docs as well. 
 
 6. Create and push the new tag
 ------------------------------
 
 Commit the changes to release.py::
 
     git commit -am "release $VERSION" -S
     git push origin $BRANCH
 
 (omit the ``-S`` if you are no signing the package)
 
 Create and push the tag::
 
     git tag -am "release $VERSION" "$VERSION" -s
     git push origin $VERSION
 
 (omit the ``-s`` if you are no signing the package)
 
 Update release.py back to ``x.y-dev`` or ``x.y-maint`` commit and push::
 
     git commit -am "back to development" -S
     git push origin $BRANCH
 
 (omit the ``-S`` if you are no signing the package)
 
 Now checkout the tag we just made::
 
     git checkout $VERSION
 
 7. Run the release script
 -------------------------
 
 Run the ``release`` script, this step requires having a current wheel, Python
 >=3.4 and Python 2.7.::
 
     ./tools/release
 
 This makes the tarballs and wheels, and puts them under the ``dist/``
 folder. Be sure to test the ``wheels``  and the ``sdist`` locally before
 uploading them to PyPI. We do not use an universal wheel as each wheel
 installs an ``ipython2`` or ``ipython3`` script, depending on the version of
 Python it is built for. Using an universal wheel would prevent this.
 
+Check the shasum of files with::
+
+    shasum -a 256 dist/*
+
+and takes notes of them you might need them to update the conda-forge recipes.
+Rerun the command and check the hash have not changed::
+
+    ./tools/release
+    shasum -a 256 dist/*
+
 Use the following to actually upload the result of the build::
 
     ./tools/release upload
 
 It should posts them to ``archive.ipython.org`` and to PyPI.
 
 PyPI/Warehouse will automatically hide previous releases. If you are uploading
 a non-stable version, make sure to log-in to PyPI and un-hide previous version.
 
 
 8. Draft a short release announcement
 -------------------------------------
 
 The announcement should include:
 
 - release highlights
 - a link to the html version of the *What's new* section of the documentation
 - a link to upgrade or installation tips (if necessary)
 
 Post the announcement to the mailing list and or blog, and link from Twitter.
 
 .. note::
 
     If you are doing a RC or Beta, you can likely skip the next steps.
 
 9. Update milestones on GitHub
 -------------------------------
 
 These steps will bring milestones up to date:
 
 - close the just released milestone
 - open a new milestone for the next release (x, y+1), if the milestone doesn't
   exist already
 
 10. Update the IPython website
 ------------------------------
 
 The IPython website should document the new release:
 
 - add release announcement (news, announcements)
 - update current version and download links
 - update links on the documentation page (especially if a major release)
 
 11. Update readthedocs
 ----------------------
 
 Make sure to update readthedocs and set the latest tag as stable, as well as
 checking that previous release is still building under its own tag.
 
 12. Update the Conda-Forge feedstock
 ------------------------------------
 
 Follow the instructions on `the repository <https://github.com/conda-forge/ipython-feedstock>`_
 
 13. Celebrate!
 --------------
 
 Celebrate the release and please thank the contributors for their work. Great
 job!
 
 
 
 Old Documentation
 =================
 
 Out of date documentation is still available and have been kept for archival purposes.
 
 .. note::
 
   Developers documentation used to be on the IPython wiki, but are now out of
   date. The wiki is though still available for historical reasons: `Old IPython
   GitHub Wiki.  <https://github.com/ipython/ipython/wiki/Dev:-Index>`_

setup.py

@@ -1,261 +1,261 b''
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 """Setup script for IPython.
 
 Under Posix environments it works like a typical setup.py script.
 Under Windows, the command sdist is not supported, since IPython
 requires utilities which are not available under Windows."""
 
 #-----------------------------------------------------------------------------
 #  Copyright (c) 2008-2011, IPython Development Team.
 #  Copyright (c) 2001-2007, Fernando Perez <[e-mail]>
 #  Copyright (c) 2001, Janko Hauser <[e-mail]>
 #  Copyright (c) 2001, Nathaniel Gray <[e-mail]>
 #
 #  Distributed under the terms of the Modified BSD License.
 #
 #  The full license is in the file COPYING.rst, distributed with this software.
 #-----------------------------------------------------------------------------
 
 import os
 import sys
 
 # **Python version check**
 #
 # This check is also made in IPython/__init__, don't forget to update both when
 # changing Python version requirements.
 if sys.version_info < (3, 6):
     pip_message = 'This may be due to an out of date pip. Make sure you have pip >= 9.0.1.'
     try:
         import pip
         pip_version = tuple([int(x) for x in pip.__version__.split('.')[:3]])
         if pip_version < (9, 0, 1) :
             pip_message = 'Your pip version is out of date, please install pip >= 9.0.1. '\
             'pip {} detected.'.format(pip.__version__)
         else:
             # pip is new enough - it must be something else
             pip_message = ''
     except Exception:
         pass
 
 
     error = """
 IPython 7.10+ supports Python 3.6 and above, following NEP 29.
 When using Python 2.7, please install IPython 5.x LTS Long Term Support version.
 Python 3.3 and 3.4 were supported up to IPython 6.x.
 Python 3.5 was supported with IPython 7.0 to 7.9.
 
 See IPython `README.rst` file for more information:
 
     https://github.com/ipython/ipython/blob/master/README.rst
 
 Python {py} detected.
 {pip}
 """.format(py=sys.version_info, pip=pip_message )
 
     print(error, file=sys.stderr)
     sys.exit(1)
 
 # At least we're on the python version we need, move on.
 
 # BEFORE importing distutils, remove MANIFEST. distutils doesn't properly
 # update it when the contents of directories change.
 if os.path.exists('MANIFEST'): os.remove('MANIFEST')
 
 from distutils.core import setup
 
 # Our own imports
 from setupbase import target_update
 
 from setupbase import (
     setup_args,
     find_packages,
     find_package_data,
     check_package_data_first,
     find_entry_points,
     build_scripts_entrypt,
     find_data_files,
     git_prebuild,
     install_symlinked,
     install_lib_symlink,
     install_scripts_for_symlink,
     unsymlink,
 )
 
 isfile = os.path.isfile
 pjoin = os.path.join
 
 #-------------------------------------------------------------------------------
 # Handle OS specific things
 #-------------------------------------------------------------------------------
 
 if os.name in ('nt','dos'):
     os_name = 'windows'
 else:
     os_name = os.name
 
 # Under Windows, 'sdist' has not been supported.  Now that the docs build with
 # Sphinx it might work, but let's not turn it on until someone confirms that it
 # actually works.
 if os_name == 'windows' and 'sdist' in sys.argv:
     print('The sdist command is not available under Windows.  Exiting.')
     sys.exit(1)
 
 
 #-------------------------------------------------------------------------------
 # Things related to the IPython documentation
 #-------------------------------------------------------------------------------
 
 # update the manuals when building a source dist
 if len(sys.argv) >= 2 and sys.argv[1] in ('sdist','bdist_rpm'):
 
     # List of things to be updated. Each entry is a triplet of args for
     # target_update()
     to_update = [
                  ('docs/man/ipython.1.gz',
                   ['docs/man/ipython.1'],
                   'cd docs/man && gzip -9c ipython.1 > ipython.1.gz'),
                  ]
 
 
     [ target_update(*t) for t in to_update ]
 
 #---------------------------------------------------------------------------
 # Find all the packages, package data, and data_files
 #---------------------------------------------------------------------------
 
 packages = find_packages()
 package_data = find_package_data()
 
 data_files = find_data_files()
 
 setup_args['packages'] = packages
 setup_args['package_data'] = package_data
 setup_args['data_files'] = data_files
 
 #---------------------------------------------------------------------------
 # custom distutils commands
 #---------------------------------------------------------------------------
 # imports here, so they are after setuptools import if there was one
 from distutils.command.sdist import sdist
 
 setup_args['cmdclass'] = {
     'build_py': \
             check_package_data_first(git_prebuild('IPython')),
     'sdist' : git_prebuild('IPython', sdist),
     'symlink': install_symlinked,
     'install_lib_symlink': install_lib_symlink,
     'install_scripts_sym': install_scripts_for_symlink,
     'unsymlink': unsymlink,
 }
 
 
 #---------------------------------------------------------------------------
 # Handle scripts, dependencies, and setuptools specific things
 #---------------------------------------------------------------------------
 
 # For some commands, use setuptools.  Note that we do NOT list install here!
 # If you want a setuptools-enhanced install, just run 'setupegg.py install'
 needs_setuptools = {'develop', 'release', 'bdist_egg', 'bdist_rpm',
            'bdist', 'bdist_dumb', 'bdist_wininst', 'bdist_wheel',
            'egg_info', 'easy_install', 'upload', 'install_egg_info',
           }
 
 if len(needs_setuptools.intersection(sys.argv)) > 0:
     import setuptools
 
 # This dict is used for passing extra arguments that are setuptools
 # specific to setup
 setuptools_extra_args = {}
 
 # setuptools requirements
 
 extras_require = dict(
     parallel = ['ipyparallel'],
     qtconsole = ['qtconsole'],
     doc = ['Sphinx>=1.3'],
     test = ['nose>=0.10.1', 'requests', 'testpath', 'pygments', 'nbformat', 'ipykernel', 'numpy>=1.14'],
     terminal = [],
     kernel = ['ipykernel'],
     nbformat = ['nbformat'],
     notebook = ['notebook', 'ipywidgets'],
     nbconvert = ['nbconvert'],
 )
 
 install_requires = [
     'setuptools>=18.5',
     'jedi>=0.16',
     'decorator',
     'pickleshare',
     'traitlets>=4.2',
     'prompt_toolkit>=2.0.0,<3.1.0,!=3.0.0,!=3.0.1',
     'pygments',
     'backcall',
     'stack_data',
 ]
 
 # Platform-specific dependencies:
 # This is the correct way to specify these,
 # but requires pip >= 6. pip < 6 ignores these.
 
 extras_require.update({
     ':sys_platform != "win32"': ['pexpect'],
     ':sys_platform == "darwin"': ['appnope'],
     ':sys_platform == "win32"': ['colorama'],
 })
 # FIXME: re-specify above platform dependencies for pip < 6
 # These would result in non-portable bdists.
 if not any(arg.startswith('bdist') for arg in sys.argv):
     if sys.platform == 'darwin':
         install_requires.extend(['appnope'])
 
     if not sys.platform.startswith('win'):
         install_requires.append('pexpect')
 
     # workaround pypa/setuptools#147, where setuptools misspells
     # platform_python_implementation as python_implementation
     if 'setuptools' in sys.modules:
         for key in list(extras_require):
             if 'platform_python_implementation' in key:
                 new_key = key.replace('platform_python_implementation', 'python_implementation')
                 extras_require[new_key] = extras_require.pop(key)
 
 everything = set()
 for key, deps in extras_require.items():
     if ':' not in key:
         everything.update(deps)
-extras_require['all'] = everything
+extras_require['all'] = list(sorted(everything))
 
 if 'setuptools' in sys.modules:
     setuptools_extra_args['python_requires'] = '>=3.6'
     setuptools_extra_args['zip_safe'] = False
     setuptools_extra_args['entry_points'] = {
         'console_scripts': find_entry_points(),
         'pygments.lexers': [
             'ipythonconsole = IPython.lib.lexers:IPythonConsoleLexer',
             'ipython = IPython.lib.lexers:IPythonLexer',
             'ipython3 = IPython.lib.lexers:IPython3Lexer',
         ],
     }
     setup_args['extras_require'] = extras_require
     setup_args['install_requires'] = install_requires
 
 else:
     # scripts has to be a non-empty list, or install_scripts isn't called
     setup_args['scripts'] = [e.split('=')[0].strip() for e in find_entry_points()]
 
     setup_args['cmdclass']['build_scripts'] = build_scripts_entrypt
 
 #---------------------------------------------------------------------------
 # Do the actual setup now
 #---------------------------------------------------------------------------
 
 setup_args.update(setuptools_extra_args)
 
 
 
 def main():
     setup(**setup_args)
 
 if __name__ == '__main__':
     main()

tools/build_release

@@ -1,26 +1,22 b''
 #!/usr/bin/env python3
 """IPython release build script.
 """
 import os
+import sys
 from shutil import rmtree
 
 from toollib import sh, pjoin, get_ipdir, cd, sdists, buildwheels
 
 def build_release():
 
     # Get main ipython dir, this will raise if it doesn't pass some checks
     ipdir = get_ipdir()
     cd(ipdir)
 
-    # Cleanup
-    for d in ['build', 'dist', pjoin('docs', 'build'), pjoin('docs', 'dist'),
-              pjoin('docs', 'source', 'api', 'generated')]:
-        if os.path.isdir(d):
-            rmtree(d)
-
     # Build source and binary distros
     sh(sdists)
     buildwheels()
+    sh(' '.join([sys.executable, 'tools/retar.py', 'dist/*.gz']))
 
 if __name__ == '__main__':
     build_release()

tools/make_tarball.py

@@ -1,22 +1,21 b''
 #!/usr/bin/env python
 """Simple script to create a tarball with proper git info.
 """
 
 import subprocess
-import os
 
 from toollib import cd, sh
 
 tag = subprocess.check_output('git describe --tags', shell=True).decode('utf8', 'replace').strip()
 base_name = 'ipython-%s' % tag
 tar_name = '%s.tgz' % base_name
 
 # git archive is weird:  Even if I give it a specific path, it still won't
 # archive the whole tree.  It seems the only way to get the whole tree is to cd
 # to the top of the tree.  There are long threads (since 2007) on the git list
 # about this and it still doesn't work in a sensible way...
 
 cd('..')
 git_tpl = 'git archive --format=tar --prefix={0}/ HEAD | gzip > {1}'
 sh(git_tpl.format(base_name, tar_name))
 sh('mv {0} tools/'.format(tar_name))

tools/release

@@ -1,96 +1,93 b''
 #!/usr/bin/env python3
 """IPython release script.
 
 This should ONLY be run at real release time.
 """
 from __future__ import print_function
 
 import os
 from glob import glob
 from subprocess import call
 import sys
 
 from toollib import (get_ipdir, pjoin, cd, execfile, sh, archive,
         sdists, archive_user, archive_dir, buildwheels)
 from gh_api import post_download
 
 # Get main ipython dir, this will raise if it doesn't pass some checks
 ipdir = get_ipdir()
 tooldir = pjoin(ipdir, 'tools')
 distdir = pjoin(ipdir, 'dist')
 
 # Where I keep static backups of each release
 ipbackupdir = os.path.expanduser('~/ipython/backup')
 if not os.path.exists(ipbackupdir):
     os.makedirs(ipbackupdir)
 
 # Start in main IPython dir
 cd(ipdir)
 
 # Load release info
 version = None
 execfile(pjoin('IPython','core','release.py'), globals())
 
 # Build site addresses for file uploads
 release_site = '%s/release/%s' % (archive, version)
 backup_site =  '%s/backup/' % archive
 
 # Start actual release process
 print()
 print('Releasing IPython')
 print('=================')
 print()
 print('Version:', version)
 print()
 print('Source IPython directory:', ipdir)
 print()
 
 # Perform local backup, go to tools dir to run it.
 cd(tooldir)
 
 if 'upload' in sys.argv:
     cd(distdir)
 
     # do not upload OS specific files like .DS_Store
     to_upload = glob('*.whl')+glob('*.tar.gz')
     for fname in to_upload:
         # TODO: update to GitHub releases API
         continue
         print('uploading %s to GitHub' % fname)
         desc = "IPython %s source distribution" % version
         post_download("ipython/ipython", fname, description=desc)
 
     # Make target dir if it doesn't exist
     print('1. Uploading IPython to archive.ipython.org')
     sh('ssh %s "mkdir -p %s/release/%s" ' % (archive_user, archive_dir, version))
     sh('scp *.tar.gz *.whl %s' % release_site)
 
     print('2. Uploading backup files...')
     cd(ipbackupdir)
     sh('scp `ls -1tr *tgz | tail -1` %s' % backup_site)
 
     print('3. Uploading to PyPI using twine')
     cd(distdir)
     call(['twine', 'upload'] + to_upload)
 
 else:
     # Build, but don't upload
 
     # Make backup tarball
     sh('./make_tarball.py')
     sh('mv ipython-*.tgz %s' % ipbackupdir)
 
     # Build release files
-    sh('./build_release %s' % ipdir)
+    sh('./build_release')
 
     cd(ipdir)
 
-    # Upload all files
-    sh(sdists)
-
     buildwheels()
     print("`./release upload` to upload source distribution on PyPI and ipython archive")
     sys.exit(0)
 
 
 

tools/release_helper.sh

@@ -1,178 +1,199 b''
 # Simple tool to help for release
 # when releasing with bash, simple source it to get asked questions. 
 
 # misc check before starting
 
 python -c 'import keyring'
 python -c 'import twine'
 python -c 'import sphinx'
 python -c 'import sphinx_rtd_theme'
 python -c 'import nose'
 
 
 BLACK=$(tput setaf 1)
 RED=$(tput setaf 1)
 GREEN=$(tput setaf 2)
 YELLOW=$(tput setaf 3)
 BLUE=$(tput setaf 4)
 MAGENTA=$(tput setaf 5)
 CYAN=$(tput setaf 6)
 WHITE=$(tput setaf 7)
 NOR=$(tput sgr0)
 
 
 echo -n "PREV_RELEASE (X.y.z) [$PREV_RELEASE]: "
 read input
 PREV_RELEASE=${input:-$PREV_RELEASE}
 echo -n "MILESTONE (X.y) [$MILESTONE]: "
 read input
 MILESTONE=${input:-$MILESTONE}
 echo -n "VERSION (X.y.z) [$VERSION]:"
 read input
 VERSION=${input:-$VERSION}
 echo -n "BRANCH (master|X.y) [$BRANCH]:"
 read input
 BRANCH=${input:-$BRANCH}
 
 ask_section(){
     echo
     echo $BLUE"$1"$NOR 
     echo -n $GREEN"Press Enter to continue, S to skip: "$NOR
     read -n1 value
     echo 
     if [ -z $value ] || [ $value = 'y' ]  ; then
         return 0
     fi
     return 1
 }
 
 
 
 echo 
 if ask_section "Updating what's new with informations from docs/source/whatsnew/pr"
 then
     python tools/update_whatsnew.py
 
     echo
     echo $BLUE"please move the contents of "docs/source/whatsnew/development.rst" to version-X.rst"$NOR
     echo $GREEN"Press enter to continue"$NOR
     read
 fi
 
 if ask_section "Gen Stats, and authors"
 then
 
     echo 
     echo $BLUE"here are all the authors that contributed to this release:"$NOR
     git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
 
     echo 
     echo $BLUE"If you see any duplicates cancel (Ctrl-C), then edit .mailmap."
     echo $GREEN"Press enter to continue:"$NOR
     read
 
     echo $BLUE"generating stats"$NOR
     python tools/github_stats.py --milestone $MILESTONE > stats.rst
 
     echo $BLUE"stats.rst files generated."$NOR
     echo $GREEN"Please merge it with the right file (github-stats-X.rst) and commit."$NOR
     echo $GREEN"press enter to continue."$NOR
     read
 
 fi
 
 echo "Cleaning repository"
 git clean -xfdi
 
 echo $GREEN"please update version number in ${RED}IPython/core/release.py${NOR} , Do not commit yet – we'll do it later."$NOR
 
 echo $GREEN"Press enter to continue"$NOR
 read
 
 if ask_section "Build the documentation ?"
 then
     make html -C docs
     echo 
     echo $GREEN"Check the docs, press enter to continue"$NOR
     read
 
 fi
 
-echo
-echo $BLUE"Attempting to build package..."$NOR
-
-tools/build_release
-rm dist/*
-
 if ask_section "Should we commit, tag, push... etc ? "
 then
    echo
    echo $BLUE"Let's commit : git commit -am \"release $VERSION\" -S"
    echo $GREEN"Press enter to commit"$NOR
    read
    git commit -am "release $VERSION" -S
    
    echo
    echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
    echo $GREEN"Make sure you can push"$NOR
    echo $GREEN"Press enter to continue"$NOR
    read
    git push origin $BRANCH
    
    echo
    echo "Let's tag : git tag -am \"release $VERSION\" \"$VERSION\" -s"
    echo $GREEN"Press enter to tag commit"$NOR
    read
    git tag -am "release $VERSION" "$VERSION" -s
    
    echo
    echo $BLUE"And push the tag: git push origin \$VERSION ?"$NOR
    echo $GREEN"Press enter to continue"$NOR
    read
    git push origin $VERSION
    
    
    echo $GREEN"please update version number and back to .dev in ${RED}IPython/core/release.py"
    echo ${BLUE}"Do not commit yet – we'll do it later."$NOR
    
    echo $GREEN"Press enter to continue"$NOR
    read
    
    echo
    echo "Let's commit : "$BLUE"git commit -am \"back to dev\""$NOR
    echo $GREEN"Press enter to commit"$NOR
    read
    git commit -am "back to dev"
 
    echo
    echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
    echo $GREEN"Press enter to continue"$NOR
    read
    git push origin $BRANCH
 
    
    echo
    echo $BLUE"let's : git checkout $VERSION"$NOR
    echo $GREEN"Press enter to continue"$NOR
    read
    git checkout $VERSION
 fi
 
 if ask_section "Should we build and release ?"
 then
+    
+    echo $BLUE"going to set SOURCE_DATE_EPOCH"$NOR
+    echo $BLUE'export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)'$NOR
+    echo $GREEN"Press enter to continue"$NOR
+    read
+
+    export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)
+
+    echo $BLUE"SOURCE_DATE_EPOCH set to $SOURCE_DATE_EPOCH"$NOR
+    echo $GREEN"Press enter to continue"$NOR
+    read
+
+
+
+    echo
+    echo $BLUE"Attempting to build package..."$NOR
+
+    tools/release
+
+
+    echo $RED'$ shasum -a 256 dist/*'
+    shasum -a 256 dist/*
+    echo $NOR
+
+    echo $BLUE"We are going to rebuild, node the hash above, and compare them to the rebuild"$NOR
+    echo $GREEN"Press enter to continue"$NOR
+    read
 
     echo
     echo $BLUE"Attempting to build package..."$NOR
 
     tools/release
 
-    echo $RED
-    echo '$ shasum -a 256 dist/*'
+    echo $RED"Check the shasum for SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"
+    echo $RED'$ shasum -a 256 dist/*'
     shasum -a 256 dist/*
     echo $NOR
 
     if ask_section "upload packages ?"
     then 
        tools/release upload
     fi
 fi