##// END OF EJS Templates
Update the release process to attempt reproducible builds....
Matthias Bussonnier -
Show More
@@ -0,0 +1,60 b''
1 """
2 Un-targz and retargz a targz file to ensure reproducible build.
3
4 usage:
5
6 $ export SOURCE_DATE_EPOCH=$(date +%s)
7 ...
8 $ python retar.py <tarfile.gz>
9
10 The process of creating an sdist can be non-reproducible:
11 - directory created during the process get a mtime of the creation date;
12 - gziping files embed the timestamp of fo zip creation.
13
14 This will untar-retar; ensuring that all mtime > SOURCE_DATE_EPOCH will be set
15 equal to SOURCE_DATE_EPOCH.
16
17 """
18
19 import tarfile
20 import sys
21 import os
22 import gzip
23 import io
24
25 if len(sys.argv) > 2:
26 raise ValueError("Too many arguments")
27
28
29 timestamp = int(os.environ["SOURCE_DATE_EPOCH"])
30
31 old_buf = io.BytesIO()
32 with open(sys.argv[1], "rb") as f:
33 old_buf.write(f.read())
34 old_buf.seek(0)
35 old = tarfile.open(fileobj=old_buf, mode="r:gz")
36
37 buf = io.BytesIO()
38 new = tarfile.open(fileobj=buf, mode="w", format=tarfile.GNU_FORMAT)
39 for i, m in enumerate(old):
40 data = None
41 # mutation does not work, copy
42 if m.name.endswith('.DS_Store'):
43 continue
44 m2 = tarfile.TarInfo(m.name)
45 m2.mtime = min(timestamp, m.mtime)
46 m2.size = m.size
47 m2.type = m.type
48 m2.linkname = m.linkname
49 if m.isdir():
50 data = old.extractfile(m)
51 new.addfile(m2, data)
52 else:
53 new.addfile(m2)
54 new.close()
55 old.close()
56
57 buf.seek(0)
58 with open(sys.argv[1], "wb") as f:
59 with gzip.GzipFile('', "wb", fileobj=f, mtime=timestamp) as gzf:
60 gzf.write(buf.read())
@@ -1,303 +1,320 b''
1 1 .. _core_developer_guide:
2 2
3 3 =================================
4 4 Guide for IPython core Developers
5 5 =================================
6 6
7 7 This guide documents the development of IPython itself. Alternatively,
8 8 developers of third party tools and libraries that use IPython should see the
9 9 :doc:`../development/index`.
10 10
11 11
12 12 For instructions on how to make a developer install see :ref:`devinstall`.
13 13
14 14 Backporting Pull requests
15 15 =========================
16 16
17 17 All pull requests should usually be made against ``master``, if a Pull Request
18 18 need to be backported to an earlier release; then it should be tagged with the
19 19 correct ``milestone``.
20 20
21 21 If you tag a pull request with a milestone **before** merging the pull request,
22 22 and the base ref is ``master``, then our backport bot should automatically create
23 23 a corresponding pull-request that backport on the correct branch.
24 24
25 25 If you have write access to the IPython repository you can also just mention the
26 26 **backport bot** to do the work for you. The bot is evolving so instructions may
27 27 be different. At the time of this writing you can use::
28 28
29 29 @meeseeksdev[bot] backport [to] <branchname>
30 30
31 31 The bot will attempt to backport the current pull-request and issue a PR if
32 32 possible.
33 33
34 34 .. note::
35 35
36 36 The ``@`` and ``[bot]`` when mentioning the bot should be optional and can
37 37 be omitted.
38 38
39 39 If the pull request cannot be automatically backported, the bot should tell you
40 40 so on the PR and apply a "Need manual backport" tag to the origin PR.
41 41
42 42 .. _release_process:
43 43
44 44 IPython release process
45 45 =======================
46 46
47 47 This document contains the process that is used to create an IPython release.
48 48
49 49 Conveniently, the ``release`` script in the ``tools`` directory of the ``IPython``
50 50 repository automates most of the release process. This document serves as a
51 51 handy reminder and checklist for the release manager.
52 52
53 53 During the release process, you might need the extra following dependencies:
54 54
55 55 - ``keyring`` to access your GitHub authentication tokens
56 56 - ``graphviz`` to generate some graphs in the documentation
57 57 - ``ghpro`` to generate the stats
58 58
59 59 Make sure you have all the required dependencies to run the tests as well.
60 60
61 61 You can try to ``source tools/release_helper.sh`` when releasing via bash, it
62 62 should guide you through most of the process.
63 63
64 64
65 65 1. Set Environment variables
66 66 ----------------------------
67 67
68 68 Set environment variables to document previous release tag, current
69 69 release milestone, current release version, and git tag.
70 70
71 71 These variables may be used later to copy/paste as answers to the script
72 72 questions instead of typing the appropriate command when the time comes. These
73 73 variables are not used by the scripts directly; therefore, there is no need to
74 74 ``export`` them. The format for bash is as follows, but note that these values
75 75 are just an example valid only for the 5.0 release; you'll need to update them
76 76 for the release you are actually making::
77 77
78 78 PREV_RELEASE=4.2.1
79 79 MILESTONE=5.0
80 80 VERSION=5.0.0
81 81 BRANCH=master
82 82
83 For `reproducibility of builds <https://reproducible-builds.org/specs/source-date-epoch/>`_,
84 we recommend setting ``SOURCE_DATE_EPOCH`` prior to running the build; record the used value
85 of ``SOURCE_DATE_EPOCH`` as it may not be available from build artifact. You
86 should be able to use ``date +%s`` to get a formatted timestamp::
87
88 SOURCE_DATE_EPOCH=$(date +%s)
89
83 90
84 91 2. Create GitHub stats and finish release note
85 92 ----------------------------------------------
86 93
87 94 .. note::
88 95
89 96 This step is optional if making a Beta or RC release.
90 97
91 98 .. note::
92 99
93 100 Before generating the GitHub stats, verify that all closed issues and pull
94 101 requests have `appropriate milestones
95 102 <https://github.com/ipython/ipython/wiki/Dev:-GitHub-workflow#milestones>`_.
96 103 `This search
97 104 <https://github.com/ipython/ipython/issues?q=is%3Aclosed+no%3Amilestone+is%3Aissue>`_
98 105 should return no results before creating the GitHub stats.
99 106
100 107 If a major release:
101 108
102 109 - merge any pull request notes into what's new::
103 110
104 111 python tools/update_whatsnew.py
105 112
106 113 - update ``docs/source/whatsnew/development.rst``, to ensure it covers
107 114 the major release features
108 115
109 116 - move the contents of ``development.rst`` to ``versionX.rst`` where ``X`` is
110 117 the numerical release version
111 118
112 119 - generate summary of GitHub contributions, which can be done with::
113 120
114 121 python tools/github_stats.py --milestone $MILESTONE > stats.rst
115 122
116 123 which may need some manual cleanup of ``stats.rst``. Add the cleaned
117 124 ``stats.rst`` results to ``docs/source/whatsnew/github-stats-X.rst``
118 125 where ``X`` is the numerical release version (don't forget to add it to
119 126 the git repository as well). If creating a major release, make a new
120 127 ``github-stats-X.rst`` file; if creating a minor release, the content
121 128 from ``stats.rst`` may simply be added to the top of an existing
122 129 ``github-stats-X.rst`` file.
123 130
124 131 - Edit ``docs/source/whatsnew/index.rst`` to list the new ``github-stats-X``
125 132 file you just created.
126 133
127 134 - You do not need to temporarily remove the first entry called
128 135 ``development``, nor re-add it after the release, it will automatically be
129 136 hidden when releasing a stable version of IPython (if ``_version_extra``
130 137 in ``release.py`` is an empty string.
131 138
132 139 Make sure that the stats file has a header or it won't be rendered in
133 140 the final documentation.
134 141
135 142 To find duplicates and update `.mailmap`, use::
136 143
137 144 git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
138 145
139 146 If a minor release you might need to do some of the above points manually, and
140 147 forward port the changes.
141 148
142 149 3. Make sure the repository is clean
143 150 ------------------------------------
144 151
145 152 of any file that could be problematic.
146 153 Remove all non-tracked files with:
147 154
148 155 .. code::
149 156
150 157 git clean -xfdi
151 158
152 159 This will ask for confirmation before removing all untracked files. Make
153 160 sure the ``dist/`` folder is clean to avoid any stale builds from
154 161 previous build attempts.
155 162
156 163
157 164 4. Update the release version number
158 165 ------------------------------------
159 166
160 167 Edit ``IPython/core/release.py`` to have the current version.
161 168
162 169 in particular, update version number and ``_version_extra`` content in
163 170 ``IPython/core/release.py``.
164 171
165 172 Step 5 will validate your changes automatically, but you might still want to
166 173 make sure the version number matches pep440.
167 174
168 175 In particular, ``rc`` and ``beta`` are not separated by ``.`` or the ``sdist``
169 176 and ``bdist`` will appear as different releases. For example, a valid version
170 177 number for a release candidate (rc) release is: ``1.3rc1``. Notice that there
171 178 is no separator between the '3' and the 'r'. Check the environment variable
172 179 ``$VERSION`` as well.
173 180
174 181 You will likely just have to modify/comment/uncomment one of the lines setting
175 182 ``_version_extra``
176 183
177 184
178 185 5. Run the `tools/build_release` script
179 186 ---------------------------------------
180 187
181 188 Running ``tools/build_release`` does all the file checking and building that
182 189 the real release script will do. This makes test installations, checks that
183 190 the build procedure runs OK, and tests other steps in the release process.
184 191
185 192 The ``build_release`` script will in particular verify that the version number
186 193 match PEP 440, in order to avoid surprise at the time of build upload.
187 194
188 195 We encourage creating a test build of the docs as well.
189 196
190 197 6. Create and push the new tag
191 198 ------------------------------
192 199
193 200 Commit the changes to release.py::
194 201
195 202 git commit -am "release $VERSION" -S
196 203 git push origin $BRANCH
197 204
198 205 (omit the ``-S`` if you are no signing the package)
199 206
200 207 Create and push the tag::
201 208
202 209 git tag -am "release $VERSION" "$VERSION" -s
203 210 git push origin $VERSION
204 211
205 212 (omit the ``-s`` if you are no signing the package)
206 213
207 214 Update release.py back to ``x.y-dev`` or ``x.y-maint`` commit and push::
208 215
209 216 git commit -am "back to development" -S
210 217 git push origin $BRANCH
211 218
212 219 (omit the ``-S`` if you are no signing the package)
213 220
214 221 Now checkout the tag we just made::
215 222
216 223 git checkout $VERSION
217 224
218 225 7. Run the release script
219 226 -------------------------
220 227
221 228 Run the ``release`` script, this step requires having a current wheel, Python
222 229 >=3.4 and Python 2.7.::
223 230
224 231 ./tools/release
225 232
226 233 This makes the tarballs and wheels, and puts them under the ``dist/``
227 234 folder. Be sure to test the ``wheels`` and the ``sdist`` locally before
228 235 uploading them to PyPI. We do not use an universal wheel as each wheel
229 236 installs an ``ipython2`` or ``ipython3`` script, depending on the version of
230 237 Python it is built for. Using an universal wheel would prevent this.
231 238
239 Check the shasum of files with::
240
241 shasum -a 256 dist/*
242
243 and takes notes of them you might need them to update the conda-forge recipes.
244 Rerun the command and check the hash have not changed::
245
246 ./tools/release
247 shasum -a 256 dist/*
248
232 249 Use the following to actually upload the result of the build::
233 250
234 251 ./tools/release upload
235 252
236 253 It should posts them to ``archive.ipython.org`` and to PyPI.
237 254
238 255 PyPI/Warehouse will automatically hide previous releases. If you are uploading
239 256 a non-stable version, make sure to log-in to PyPI and un-hide previous version.
240 257
241 258
242 259 8. Draft a short release announcement
243 260 -------------------------------------
244 261
245 262 The announcement should include:
246 263
247 264 - release highlights
248 265 - a link to the html version of the *What's new* section of the documentation
249 266 - a link to upgrade or installation tips (if necessary)
250 267
251 268 Post the announcement to the mailing list and or blog, and link from Twitter.
252 269
253 270 .. note::
254 271
255 272 If you are doing a RC or Beta, you can likely skip the next steps.
256 273
257 274 9. Update milestones on GitHub
258 275 -------------------------------
259 276
260 277 These steps will bring milestones up to date:
261 278
262 279 - close the just released milestone
263 280 - open a new milestone for the next release (x, y+1), if the milestone doesn't
264 281 exist already
265 282
266 283 10. Update the IPython website
267 284 ------------------------------
268 285
269 286 The IPython website should document the new release:
270 287
271 288 - add release announcement (news, announcements)
272 289 - update current version and download links
273 290 - update links on the documentation page (especially if a major release)
274 291
275 292 11. Update readthedocs
276 293 ----------------------
277 294
278 295 Make sure to update readthedocs and set the latest tag as stable, as well as
279 296 checking that previous release is still building under its own tag.
280 297
281 298 12. Update the Conda-Forge feedstock
282 299 ------------------------------------
283 300
284 301 Follow the instructions on `the repository <https://github.com/conda-forge/ipython-feedstock>`_
285 302
286 303 13. Celebrate!
287 304 --------------
288 305
289 306 Celebrate the release and please thank the contributors for their work. Great
290 307 job!
291 308
292 309
293 310
294 311 Old Documentation
295 312 =================
296 313
297 314 Out of date documentation is still available and have been kept for archival purposes.
298 315
299 316 .. note::
300 317
301 318 Developers documentation used to be on the IPython wiki, but are now out of
302 319 date. The wiki is though still available for historical reasons: `Old IPython
303 320 GitHub Wiki. <https://github.com/ipython/ipython/wiki/Dev:-Index>`_
@@ -1,261 +1,261 b''
1 1 #!/usr/bin/env python3
2 2 # -*- coding: utf-8 -*-
3 3 """Setup script for IPython.
4 4
5 5 Under Posix environments it works like a typical setup.py script.
6 6 Under Windows, the command sdist is not supported, since IPython
7 7 requires utilities which are not available under Windows."""
8 8
9 9 #-----------------------------------------------------------------------------
10 10 # Copyright (c) 2008-2011, IPython Development Team.
11 11 # Copyright (c) 2001-2007, Fernando Perez <fernando.perez@colorado.edu>
12 12 # Copyright (c) 2001, Janko Hauser <jhauser@zscout.de>
13 13 # Copyright (c) 2001, Nathaniel Gray <n8gray@caltech.edu>
14 14 #
15 15 # Distributed under the terms of the Modified BSD License.
16 16 #
17 17 # The full license is in the file COPYING.rst, distributed with this software.
18 18 #-----------------------------------------------------------------------------
19 19
20 20 import os
21 21 import sys
22 22
23 23 # **Python version check**
24 24 #
25 25 # This check is also made in IPython/__init__, don't forget to update both when
26 26 # changing Python version requirements.
27 27 if sys.version_info < (3, 6):
28 28 pip_message = 'This may be due to an out of date pip. Make sure you have pip >= 9.0.1.'
29 29 try:
30 30 import pip
31 31 pip_version = tuple([int(x) for x in pip.__version__.split('.')[:3]])
32 32 if pip_version < (9, 0, 1) :
33 33 pip_message = 'Your pip version is out of date, please install pip >= 9.0.1. '\
34 34 'pip {} detected.'.format(pip.__version__)
35 35 else:
36 36 # pip is new enough - it must be something else
37 37 pip_message = ''
38 38 except Exception:
39 39 pass
40 40
41 41
42 42 error = """
43 43 IPython 7.10+ supports Python 3.6 and above, following NEP 29.
44 44 When using Python 2.7, please install IPython 5.x LTS Long Term Support version.
45 45 Python 3.3 and 3.4 were supported up to IPython 6.x.
46 46 Python 3.5 was supported with IPython 7.0 to 7.9.
47 47
48 48 See IPython `README.rst` file for more information:
49 49
50 50 https://github.com/ipython/ipython/blob/master/README.rst
51 51
52 52 Python {py} detected.
53 53 {pip}
54 54 """.format(py=sys.version_info, pip=pip_message )
55 55
56 56 print(error, file=sys.stderr)
57 57 sys.exit(1)
58 58
59 59 # At least we're on the python version we need, move on.
60 60
61 61 # BEFORE importing distutils, remove MANIFEST. distutils doesn't properly
62 62 # update it when the contents of directories change.
63 63 if os.path.exists('MANIFEST'): os.remove('MANIFEST')
64 64
65 65 from distutils.core import setup
66 66
67 67 # Our own imports
68 68 from setupbase import target_update
69 69
70 70 from setupbase import (
71 71 setup_args,
72 72 find_packages,
73 73 find_package_data,
74 74 check_package_data_first,
75 75 find_entry_points,
76 76 build_scripts_entrypt,
77 77 find_data_files,
78 78 git_prebuild,
79 79 install_symlinked,
80 80 install_lib_symlink,
81 81 install_scripts_for_symlink,
82 82 unsymlink,
83 83 )
84 84
85 85 isfile = os.path.isfile
86 86 pjoin = os.path.join
87 87
88 88 #-------------------------------------------------------------------------------
89 89 # Handle OS specific things
90 90 #-------------------------------------------------------------------------------
91 91
92 92 if os.name in ('nt','dos'):
93 93 os_name = 'windows'
94 94 else:
95 95 os_name = os.name
96 96
97 97 # Under Windows, 'sdist' has not been supported. Now that the docs build with
98 98 # Sphinx it might work, but let's not turn it on until someone confirms that it
99 99 # actually works.
100 100 if os_name == 'windows' and 'sdist' in sys.argv:
101 101 print('The sdist command is not available under Windows. Exiting.')
102 102 sys.exit(1)
103 103
104 104
105 105 #-------------------------------------------------------------------------------
106 106 # Things related to the IPython documentation
107 107 #-------------------------------------------------------------------------------
108 108
109 109 # update the manuals when building a source dist
110 110 if len(sys.argv) >= 2 and sys.argv[1] in ('sdist','bdist_rpm'):
111 111
112 112 # List of things to be updated. Each entry is a triplet of args for
113 113 # target_update()
114 114 to_update = [
115 115 ('docs/man/ipython.1.gz',
116 116 ['docs/man/ipython.1'],
117 117 'cd docs/man && gzip -9c ipython.1 > ipython.1.gz'),
118 118 ]
119 119
120 120
121 121 [ target_update(*t) for t in to_update ]
122 122
123 123 #---------------------------------------------------------------------------
124 124 # Find all the packages, package data, and data_files
125 125 #---------------------------------------------------------------------------
126 126
127 127 packages = find_packages()
128 128 package_data = find_package_data()
129 129
130 130 data_files = find_data_files()
131 131
132 132 setup_args['packages'] = packages
133 133 setup_args['package_data'] = package_data
134 134 setup_args['data_files'] = data_files
135 135
136 136 #---------------------------------------------------------------------------
137 137 # custom distutils commands
138 138 #---------------------------------------------------------------------------
139 139 # imports here, so they are after setuptools import if there was one
140 140 from distutils.command.sdist import sdist
141 141
142 142 setup_args['cmdclass'] = {
143 143 'build_py': \
144 144 check_package_data_first(git_prebuild('IPython')),
145 145 'sdist' : git_prebuild('IPython', sdist),
146 146 'symlink': install_symlinked,
147 147 'install_lib_symlink': install_lib_symlink,
148 148 'install_scripts_sym': install_scripts_for_symlink,
149 149 'unsymlink': unsymlink,
150 150 }
151 151
152 152
153 153 #---------------------------------------------------------------------------
154 154 # Handle scripts, dependencies, and setuptools specific things
155 155 #---------------------------------------------------------------------------
156 156
157 157 # For some commands, use setuptools. Note that we do NOT list install here!
158 158 # If you want a setuptools-enhanced install, just run 'setupegg.py install'
159 159 needs_setuptools = {'develop', 'release', 'bdist_egg', 'bdist_rpm',
160 160 'bdist', 'bdist_dumb', 'bdist_wininst', 'bdist_wheel',
161 161 'egg_info', 'easy_install', 'upload', 'install_egg_info',
162 162 }
163 163
164 164 if len(needs_setuptools.intersection(sys.argv)) > 0:
165 165 import setuptools
166 166
167 167 # This dict is used for passing extra arguments that are setuptools
168 168 # specific to setup
169 169 setuptools_extra_args = {}
170 170
171 171 # setuptools requirements
172 172
173 173 extras_require = dict(
174 174 parallel = ['ipyparallel'],
175 175 qtconsole = ['qtconsole'],
176 176 doc = ['Sphinx>=1.3'],
177 177 test = ['nose>=0.10.1', 'requests', 'testpath', 'pygments', 'nbformat', 'ipykernel', 'numpy>=1.14'],
178 178 terminal = [],
179 179 kernel = ['ipykernel'],
180 180 nbformat = ['nbformat'],
181 181 notebook = ['notebook', 'ipywidgets'],
182 182 nbconvert = ['nbconvert'],
183 183 )
184 184
185 185 install_requires = [
186 186 'setuptools>=18.5',
187 187 'jedi>=0.16',
188 188 'decorator',
189 189 'pickleshare',
190 190 'traitlets>=4.2',
191 191 'prompt_toolkit>=2.0.0,<3.1.0,!=3.0.0,!=3.0.1',
192 192 'pygments',
193 193 'backcall',
194 194 'stack_data',
195 195 ]
196 196
197 197 # Platform-specific dependencies:
198 198 # This is the correct way to specify these,
199 199 # but requires pip >= 6. pip < 6 ignores these.
200 200
201 201 extras_require.update({
202 202 ':sys_platform != "win32"': ['pexpect'],
203 203 ':sys_platform == "darwin"': ['appnope'],
204 204 ':sys_platform == "win32"': ['colorama'],
205 205 })
206 206 # FIXME: re-specify above platform dependencies for pip < 6
207 207 # These would result in non-portable bdists.
208 208 if not any(arg.startswith('bdist') for arg in sys.argv):
209 209 if sys.platform == 'darwin':
210 210 install_requires.extend(['appnope'])
211 211
212 212 if not sys.platform.startswith('win'):
213 213 install_requires.append('pexpect')
214 214
215 215 # workaround pypa/setuptools#147, where setuptools misspells
216 216 # platform_python_implementation as python_implementation
217 217 if 'setuptools' in sys.modules:
218 218 for key in list(extras_require):
219 219 if 'platform_python_implementation' in key:
220 220 new_key = key.replace('platform_python_implementation', 'python_implementation')
221 221 extras_require[new_key] = extras_require.pop(key)
222 222
223 223 everything = set()
224 224 for key, deps in extras_require.items():
225 225 if ':' not in key:
226 226 everything.update(deps)
227 extras_require['all'] = everything
227 extras_require['all'] = list(sorted(everything))
228 228
229 229 if 'setuptools' in sys.modules:
230 230 setuptools_extra_args['python_requires'] = '>=3.6'
231 231 setuptools_extra_args['zip_safe'] = False
232 232 setuptools_extra_args['entry_points'] = {
233 233 'console_scripts': find_entry_points(),
234 234 'pygments.lexers': [
235 235 'ipythonconsole = IPython.lib.lexers:IPythonConsoleLexer',
236 236 'ipython = IPython.lib.lexers:IPythonLexer',
237 237 'ipython3 = IPython.lib.lexers:IPython3Lexer',
238 238 ],
239 239 }
240 240 setup_args['extras_require'] = extras_require
241 241 setup_args['install_requires'] = install_requires
242 242
243 243 else:
244 244 # scripts has to be a non-empty list, or install_scripts isn't called
245 245 setup_args['scripts'] = [e.split('=')[0].strip() for e in find_entry_points()]
246 246
247 247 setup_args['cmdclass']['build_scripts'] = build_scripts_entrypt
248 248
249 249 #---------------------------------------------------------------------------
250 250 # Do the actual setup now
251 251 #---------------------------------------------------------------------------
252 252
253 253 setup_args.update(setuptools_extra_args)
254 254
255 255
256 256
257 257 def main():
258 258 setup(**setup_args)
259 259
260 260 if __name__ == '__main__':
261 261 main()
@@ -1,26 +1,22 b''
1 1 #!/usr/bin/env python3
2 2 """IPython release build script.
3 3 """
4 4 import os
5 import sys
5 6 from shutil import rmtree
6 7
7 8 from toollib import sh, pjoin, get_ipdir, cd, sdists, buildwheels
8 9
9 10 def build_release():
10 11
11 12 # Get main ipython dir, this will raise if it doesn't pass some checks
12 13 ipdir = get_ipdir()
13 14 cd(ipdir)
14 15
15 # Cleanup
16 for d in ['build', 'dist', pjoin('docs', 'build'), pjoin('docs', 'dist'),
17 pjoin('docs', 'source', 'api', 'generated')]:
18 if os.path.isdir(d):
19 rmtree(d)
20
21 16 # Build source and binary distros
22 17 sh(sdists)
23 18 buildwheels()
19 sh(' '.join([sys.executable, 'tools/retar.py', 'dist/*.gz']))
24 20
25 21 if __name__ == '__main__':
26 22 build_release()
@@ -1,22 +1,21 b''
1 1 #!/usr/bin/env python
2 2 """Simple script to create a tarball with proper git info.
3 3 """
4 4
5 5 import subprocess
6 import os
7 6
8 7 from toollib import cd, sh
9 8
10 9 tag = subprocess.check_output('git describe --tags', shell=True).decode('utf8', 'replace').strip()
11 10 base_name = 'ipython-%s' % tag
12 11 tar_name = '%s.tgz' % base_name
13 12
14 13 # git archive is weird: Even if I give it a specific path, it still won't
15 14 # archive the whole tree. It seems the only way to get the whole tree is to cd
16 15 # to the top of the tree. There are long threads (since 2007) on the git list
17 16 # about this and it still doesn't work in a sensible way...
18 17
19 18 cd('..')
20 19 git_tpl = 'git archive --format=tar --prefix={0}/ HEAD | gzip > {1}'
21 20 sh(git_tpl.format(base_name, tar_name))
22 21 sh('mv {0} tools/'.format(tar_name))
@@ -1,96 +1,93 b''
1 1 #!/usr/bin/env python3
2 2 """IPython release script.
3 3
4 4 This should ONLY be run at real release time.
5 5 """
6 6 from __future__ import print_function
7 7
8 8 import os
9 9 from glob import glob
10 10 from subprocess import call
11 11 import sys
12 12
13 13 from toollib import (get_ipdir, pjoin, cd, execfile, sh, archive,
14 14 sdists, archive_user, archive_dir, buildwheels)
15 15 from gh_api import post_download
16 16
17 17 # Get main ipython dir, this will raise if it doesn't pass some checks
18 18 ipdir = get_ipdir()
19 19 tooldir = pjoin(ipdir, 'tools')
20 20 distdir = pjoin(ipdir, 'dist')
21 21
22 22 # Where I keep static backups of each release
23 23 ipbackupdir = os.path.expanduser('~/ipython/backup')
24 24 if not os.path.exists(ipbackupdir):
25 25 os.makedirs(ipbackupdir)
26 26
27 27 # Start in main IPython dir
28 28 cd(ipdir)
29 29
30 30 # Load release info
31 31 version = None
32 32 execfile(pjoin('IPython','core','release.py'), globals())
33 33
34 34 # Build site addresses for file uploads
35 35 release_site = '%s/release/%s' % (archive, version)
36 36 backup_site = '%s/backup/' % archive
37 37
38 38 # Start actual release process
39 39 print()
40 40 print('Releasing IPython')
41 41 print('=================')
42 42 print()
43 43 print('Version:', version)
44 44 print()
45 45 print('Source IPython directory:', ipdir)
46 46 print()
47 47
48 48 # Perform local backup, go to tools dir to run it.
49 49 cd(tooldir)
50 50
51 51 if 'upload' in sys.argv:
52 52 cd(distdir)
53 53
54 54 # do not upload OS specific files like .DS_Store
55 55 to_upload = glob('*.whl')+glob('*.tar.gz')
56 56 for fname in to_upload:
57 57 # TODO: update to GitHub releases API
58 58 continue
59 59 print('uploading %s to GitHub' % fname)
60 60 desc = "IPython %s source distribution" % version
61 61 post_download("ipython/ipython", fname, description=desc)
62 62
63 63 # Make target dir if it doesn't exist
64 64 print('1. Uploading IPython to archive.ipython.org')
65 65 sh('ssh %s "mkdir -p %s/release/%s" ' % (archive_user, archive_dir, version))
66 66 sh('scp *.tar.gz *.whl %s' % release_site)
67 67
68 68 print('2. Uploading backup files...')
69 69 cd(ipbackupdir)
70 70 sh('scp `ls -1tr *tgz | tail -1` %s' % backup_site)
71 71
72 72 print('3. Uploading to PyPI using twine')
73 73 cd(distdir)
74 74 call(['twine', 'upload'] + to_upload)
75 75
76 76 else:
77 77 # Build, but don't upload
78 78
79 79 # Make backup tarball
80 80 sh('./make_tarball.py')
81 81 sh('mv ipython-*.tgz %s' % ipbackupdir)
82 82
83 83 # Build release files
84 sh('./build_release %s' % ipdir)
84 sh('./build_release')
85 85
86 86 cd(ipdir)
87 87
88 # Upload all files
89 sh(sdists)
90
91 88 buildwheels()
92 89 print("`./release upload` to upload source distribution on PyPI and ipython archive")
93 90 sys.exit(0)
94 91
95 92
96 93
@@ -1,178 +1,199 b''
1 1 # Simple tool to help for release
2 2 # when releasing with bash, simple source it to get asked questions.
3 3
4 4 # misc check before starting
5 5
6 6 python -c 'import keyring'
7 7 python -c 'import twine'
8 8 python -c 'import sphinx'
9 9 python -c 'import sphinx_rtd_theme'
10 10 python -c 'import nose'
11 11
12 12
13 13 BLACK=$(tput setaf 1)
14 14 RED=$(tput setaf 1)
15 15 GREEN=$(tput setaf 2)
16 16 YELLOW=$(tput setaf 3)
17 17 BLUE=$(tput setaf 4)
18 18 MAGENTA=$(tput setaf 5)
19 19 CYAN=$(tput setaf 6)
20 20 WHITE=$(tput setaf 7)
21 21 NOR=$(tput sgr0)
22 22
23 23
24 24 echo -n "PREV_RELEASE (X.y.z) [$PREV_RELEASE]: "
25 25 read input
26 26 PREV_RELEASE=${input:-$PREV_RELEASE}
27 27 echo -n "MILESTONE (X.y) [$MILESTONE]: "
28 28 read input
29 29 MILESTONE=${input:-$MILESTONE}
30 30 echo -n "VERSION (X.y.z) [$VERSION]:"
31 31 read input
32 32 VERSION=${input:-$VERSION}
33 33 echo -n "BRANCH (master|X.y) [$BRANCH]:"
34 34 read input
35 35 BRANCH=${input:-$BRANCH}
36 36
37 37 ask_section(){
38 38 echo
39 39 echo $BLUE"$1"$NOR
40 40 echo -n $GREEN"Press Enter to continue, S to skip: "$NOR
41 41 read -n1 value
42 42 echo
43 43 if [ -z $value ] || [ $value = 'y' ] ; then
44 44 return 0
45 45 fi
46 46 return 1
47 47 }
48 48
49 49
50 50
51 51 echo
52 52 if ask_section "Updating what's new with informations from docs/source/whatsnew/pr"
53 53 then
54 54 python tools/update_whatsnew.py
55 55
56 56 echo
57 57 echo $BLUE"please move the contents of "docs/source/whatsnew/development.rst" to version-X.rst"$NOR
58 58 echo $GREEN"Press enter to continue"$NOR
59 59 read
60 60 fi
61 61
62 62 if ask_section "Gen Stats, and authors"
63 63 then
64 64
65 65 echo
66 66 echo $BLUE"here are all the authors that contributed to this release:"$NOR
67 67 git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
68 68
69 69 echo
70 70 echo $BLUE"If you see any duplicates cancel (Ctrl-C), then edit .mailmap."
71 71 echo $GREEN"Press enter to continue:"$NOR
72 72 read
73 73
74 74 echo $BLUE"generating stats"$NOR
75 75 python tools/github_stats.py --milestone $MILESTONE > stats.rst
76 76
77 77 echo $BLUE"stats.rst files generated."$NOR
78 78 echo $GREEN"Please merge it with the right file (github-stats-X.rst) and commit."$NOR
79 79 echo $GREEN"press enter to continue."$NOR
80 80 read
81 81
82 82 fi
83 83
84 84 echo "Cleaning repository"
85 85 git clean -xfdi
86 86
87 87 echo $GREEN"please update version number in ${RED}IPython/core/release.py${NOR} , Do not commit yet – we'll do it later."$NOR
88 88
89 89 echo $GREEN"Press enter to continue"$NOR
90 90 read
91 91
92 92 if ask_section "Build the documentation ?"
93 93 then
94 94 make html -C docs
95 95 echo
96 96 echo $GREEN"Check the docs, press enter to continue"$NOR
97 97 read
98 98
99 99 fi
100 100
101 echo
102 echo $BLUE"Attempting to build package..."$NOR
103
104 tools/build_release
105 rm dist/*
106
107 101 if ask_section "Should we commit, tag, push... etc ? "
108 102 then
109 103 echo
110 104 echo $BLUE"Let's commit : git commit -am \"release $VERSION\" -S"
111 105 echo $GREEN"Press enter to commit"$NOR
112 106 read
113 107 git commit -am "release $VERSION" -S
114 108
115 109 echo
116 110 echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
117 111 echo $GREEN"Make sure you can push"$NOR
118 112 echo $GREEN"Press enter to continue"$NOR
119 113 read
120 114 git push origin $BRANCH
121 115
122 116 echo
123 117 echo "Let's tag : git tag -am \"release $VERSION\" \"$VERSION\" -s"
124 118 echo $GREEN"Press enter to tag commit"$NOR
125 119 read
126 120 git tag -am "release $VERSION" "$VERSION" -s
127 121
128 122 echo
129 123 echo $BLUE"And push the tag: git push origin \$VERSION ?"$NOR
130 124 echo $GREEN"Press enter to continue"$NOR
131 125 read
132 126 git push origin $VERSION
133 127
134 128
135 129 echo $GREEN"please update version number and back to .dev in ${RED}IPython/core/release.py"
136 130 echo ${BLUE}"Do not commit yet – we'll do it later."$NOR
137 131
138 132 echo $GREEN"Press enter to continue"$NOR
139 133 read
140 134
141 135 echo
142 136 echo "Let's commit : "$BLUE"git commit -am \"back to dev\""$NOR
143 137 echo $GREEN"Press enter to commit"$NOR
144 138 read
145 139 git commit -am "back to dev"
146 140
147 141 echo
148 142 echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
149 143 echo $GREEN"Press enter to continue"$NOR
150 144 read
151 145 git push origin $BRANCH
152 146
153 147
154 148 echo
155 149 echo $BLUE"let's : git checkout $VERSION"$NOR
156 150 echo $GREEN"Press enter to continue"$NOR
157 151 read
158 152 git checkout $VERSION
159 153 fi
160 154
161 155 if ask_section "Should we build and release ?"
162 156 then
163 157
158 echo $BLUE"going to set SOURCE_DATE_EPOCH"$NOR
159 echo $BLUE'export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)'$NOR
160 echo $GREEN"Press enter to continue"$NOR
161 read
162
163 export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)
164
165 echo $BLUE"SOURCE_DATE_EPOCH set to $SOURCE_DATE_EPOCH"$NOR
166 echo $GREEN"Press enter to continue"$NOR
167 read
168
169
170
171 echo
172 echo $BLUE"Attempting to build package..."$NOR
173
174 tools/release
175
176
177 echo $RED'$ shasum -a 256 dist/*'
178 shasum -a 256 dist/*
179 echo $NOR
180
181 echo $BLUE"We are going to rebuild, node the hash above, and compare them to the rebuild"$NOR
182 echo $GREEN"Press enter to continue"$NOR
183 read
184
164 185 echo
165 186 echo $BLUE"Attempting to build package..."$NOR
166 187
167 188 tools/release
168 189
169 echo $RED
170 echo '$ shasum -a 256 dist/*'
190 echo $RED"Check the shasum for SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"
191 echo $RED'$ shasum -a 256 dist/*'
171 192 shasum -a 256 dist/*
172 193 echo $NOR
173 194
174 195 if ask_section "upload packages ?"
175 196 then
176 197 tools/release upload
177 198 fi
178 199 fi
General Comments 0
You need to be logged in to leave comments. Login now