Commit

r3497:d3230053 public

Description:

release: Merge default into stable for release preparation

Commit status:

[Not Reviewed]

References:

merge stable

Comments:

0 Commit comments 0 Inline Comments

Unresolved TODOs:

There are no unresolved TODOs

Author

-
Add another comment

r3497:d32300537fe6 - - The requested changes are too big and content was truncated. 807 files changed. Show full diff

@@ -0,0 +1,42
1 .. _restore-deleted-repositories:
2
3 Restoring Deleted Repositories
4 ==============================
5
6 By default when repository or whole repository group is deleted an archived copy
7 of filesystem repositories are kept. You can see them as special entries in the
8 repository storage such as::
9
10 drwxrwxr-x 3 rcdev rcdev 4096 Dec 4 2017 rm__20171204_105727_400795__ce-import
11 drwxrwxr-x 6 rcdev rcdev 4096 Nov 21 2017 rm__20180221_152430_675047__svn-repo
12 drwxr-xr-x 7 rcdev rcdev 4096 Mar 28 2018 rm__20180328_143124_617576__test-git
13 drwxr-xr-x 7 rcdev rcdev 4096 Mar 28 2018 rm__20180328_144954_317729__test-git-install-hooks
14
15
16 Data from those repositories can be restored by simply removing the
17 `rm_YYYYDDMM_HHMMSS_DDDDDD__` prefix and additionally only in case of Mercurial
18 repositories remove the `.hg` store prefix.::
19
20 rm__.hg => .hg
21
22
23 For Git or SVN repositories this operation is not required.
24
25 After removing the prefix repository can be brought by opening
26 :menuselection:`Admin --> Settings --> Remap and Rescan` and running `Rescan Filesystem`
27
28 This will create a new DB entry restoring the data previously removed.
29 To restore OLD database entries this should be done by restoring from a Database backup.
30
31 RhodeCode also keeps the repository group structure, this is marked by entries that
32 in addition have GROUP in the prefix, eg::
33
34 drwxr-xr-x 2 rcdev rcdev 4096 Jan 18 16:13 rm__20181130_120650_977082_GROUP_Test1
35 drwxr-xr-x 2 rcdev rcdev 4096 Jan 18 16:13 rm__20181130_120659_922952_GROUP_Docs
36
37
38
39 .. note::
40
41 RhodeCode Tools have a special cleanup tool for the archived repositories. Please
42 see :ref:`clean-up-cmds`
@@ -0,0 +1,37
1 .. _store-methods-ref:
2
3 store methods
4 =============
5
6 file_store_add (EE only)
7 ------------------------
8
9 .. py:function:: file_store_add(apiuser, filename, content)
10
11 Upload API for the file_store
12
13 Example usage from CLI::
14 rhodecode-api --instance-name=enterprise-1 upload_file "{"content": "$(cat image.jpg | base64)", "filename":"image.jpg"}"
15
16 This command takes the following options:
17
18 :param apiuser: This is filled automatically from the |authtoken|.
19 :type apiuser: AuthUser
20 :param filename: name of the file uploaded
21 :type filename: str
22 :param content: base64 encoded content of the uploaded file
23 :type content: str
24
25 Example output:
26
27 .. code-block:: bash
28
29 id : <id_given_in_input>
30 result: {
31 "access_path": "/_file_store/download/84d156f7-8323-4ad3-9fce-4a8e88e1deaf-0.jpg",
32 "access_path_fqn": "http://server.domain.com/_file_store/download/84d156f7-8323-4ad3-9fce-4a8e88e1deaf-0.jpg",
33 "store_fid": "84d156f7-8323-4ad3-9fce-4a8e88e1deaf-0.jpg"
34 }
35 error : null
36
37
@@ -0,0 +1,88
1 .. _auth-saml-bulk-enroll-users-ref:
2
3
4 Bulk enroll multiple existing users
5 -----------------------------------
6
7
8 RhodeCode Supports standard SAML 2.0 SSO for the web-application part.
9 Below is an example how to enroll list of all or some users to use SAML authentication.
10 This method simply enables SAML authentication for many users at once.
11
12
13 From the server RhodeCode Enterprise is running run ishell on the instance which we
14 want to apply the SAML migration::
15
16 rccontrol ishell enterprise-1
17
18 Follow these steps to enable SAML authentication for multiple users.
19
20
21 1) Create a user_id => attribute mapping
22
23
24 `saml2user` is a mapping of external ID from SAML provider such as OneLogin, DuoSecurity, Google.
25 This mapping consists of local rhodecode user_id mapped to set of required attributes needed to bind SAML
26 account to internal rhodecode user.
27 For example, 123 is local rhodecode user_id, and '48253211' is OneLogin ID.
28 For other providers you'd have to figure out what would be the user-id, sometimes it's the email, i.e for Google
29 The most important this id needs to be unique for each user.
30
31 .. code-block:: python
32
33 In [1]: saml2user = {
34 ...: # OneLogin, uses externalID available to read from in the UI
35 ...: 123: {'id: '48253211'},
36 ...: # for Google/DuoSecurity email is also an option for unique ID
37 ...: 124: {'id: 'email@domain.com'},
38 ...: }
39
40
41 2) Import the plugin you want to run migration for.
42
43 From available options pick only one and run the `import` statement
44
45 .. code-block:: python
46
47 # for Duo Security
48 In [2]: from rc_auth_plugins.auth_duo_security import RhodeCodeAuthPlugin
49 # for OneLogin
50 In [2]: from rc_auth_plugins.auth_onelogin import RhodeCodeAuthPlugin
51 # generic SAML plugin
52 In [2]: from rc_auth_plugins.auth_saml import RhodeCodeAuthPlugin
53
54 3) Run the migration based on saml2user mapping.
55
56 Enter in the ishell prompt
57
58 .. code-block:: python
59
60 In [3]: for user in User.get_all():
61 ...: existing_identity = ExternalIdentity().query().filter(ExternalIdentity.local_user_id == user.user_id).scalar()
62 ...: attrs = saml2user.get(user.user_id)
63 ...: provider = RhodeCodeAuthPlugin.uid
64 ...: if existing_identity:
65 ...: print('Identity for user `{}` already exists, skipping'.format(user.username))
66 ...: continue
67 ...: if attrs:
68 ...: external_id = attrs['id']
69 ...: new_external_identity = ExternalIdentity()
70 ...: new_external_identity.external_id = external_id
71 ...: new_external_identity.external_username = '{}-saml-{}'.format(user.username, user.user_id)
72 ...: new_external_identity.provider_name = provider
73 ...: new_external_identity.local_user_id = user_id
74 ...: new_external_identity.access_token = ''
75 ...: new_external_identity.token_secret = ''
76 ...: new_external_identity.alt_token = ''
77 ...: Session().add(ex_identity)
78 ...: Session().commit()
79 ...: print('Set user `{}` external identity bound to ExternalID:{}'.format(user.username, external_id))
80
81 .. note::
82
83 saml2user can be really big and hard to maintain in ishell. It's also possible
84 to load it as a JSON file prepared before and stored on disk. To do so run::
85
86 import json
87 saml2user = json.loads(open('/path/to/saml2user.json','rb').read())
88
@@ -0,0 +1,148
1 |RCE| 4.16.0 |RNS|
2 ------------------
3
4 Release Date
5 ^^^^^^^^^^^^
6
7 - 2019-02-15
8
9
10 New Features
11 ^^^^^^^^^^^^
12
13
14 - Full-text search: added support for ElasticSearch 6.X (ES6)
15 - Full-text search: Expose a quick way to search within repository groups using ES6.
16 - Full-text search: Add quick links to broaden/narrow search scope to repositories or
17 repository groups from global search.
18 - Full-text search: ES6 backend adds new highlighter, and search markers for better UX when searching.
19 - Full-text search: ES6 backend has enabled advanced `query string syntax`
20 adding more search and filtering capabilities.
21 - Full-text search: ES6 engine will now show added information where available such as line numbers file size.
22 - Files: added option to use highlight marker to show keywords inside file source. This
23 is used now for ES6 backend extended highlighting capabilities
24 - Artifacts (beta): EE edition exposes new feature called storage_api this allows storing
25 binary files outside of Version Control System, but in the scope of a repository or group.
26 This will soon become an Artifacts functionality available in EE edition.
27 - Authentication: introduced `User restriction` and `Scope restriction` for RhodeCode authentication plugins.
28 Admins can limit usage of RhodeCode plugins to super-admins user types, and usage in Web, or VCS protocol only.
29 This is mostly to help to migrate users to SAML, keeping the super-admins to manage instances via local-logins,
30 and secondly to force usage of AuthenticationTokens instead of re-using same credentials for
31 WEB and VCS authentication.
32 - API: added basic upload API for the storage_api. It's possible to store files using internal
33 API. This is a start for attachments upload in RhodeCode.
34 - API: added store_exception_api for remote exception storage. This is used by a new
35 indexer that will report any problems back into the RhodeCode instance in case of indexing problems.
36 - API: added function to fetch comments for a repository.
37 - Quick search: improve the styling of search input and results.
38 - Pull requests: allowed to select all forks and parent forks of target repository in creation UI.
39 This is a common workflow supported by GitHub etc.
40
41
42 General
43 ^^^^^^^
44
45 - Users/Repositories/Repository groups: expose IDs of those objects in advanced views.
46 Useful for API calls or usage in ishell.
47 - UI: moved repo group select next to the name as it's very relevant to each other.
48 - Pull requests: increase the stability of concurrent pull requests created.
49 - Pull requests: introduced operation state for pull requests to prevent from
50 locks during merge/update operations in concurrent busy environments.
51 - Pull requests: ensure that merge response provide more details about failed operations.
52 - UI / Files: expose downloads options onto files view similar as in summary page.
53 - Repositories: show hooks version and update link in the advanced section of repository page.
54 - Events: trigger 'review_status_change' in all cases when reviewers are changed
55 influencing review status.
56 - Files: display submodules in a sorted way, equal to how Directories are sorted.
57 - API: fetching all pull-requests now sorts the results and exposed a flag to show/hide
58 the merge result state for faster result fetching.
59 - API: merge_pull_request expose detailed merge message in the merge operation
60 next to numeric merge response code.
61 - API: added possibility to specify owner to create_pull_request API.
62 - SSH: Added ability to disable server-side SSH key generation to enforce users
63 generated SSH keys only outside of the server.
64 - Integrations: allow PUT method for WebHook integration.
65 - Dependencies: bumped git to 2.19.2 release.
66 - Dependencies: dropped pygments-markdown-lexer as it's natively supported by pygments now.
67 - Dependencies: bumped pyramid to 1.10.1
68 - Dependencies: bumped pastedeploy to 2.0.1
69 - Dependencies: bumped pastescript to 3.0.0
70 - Dependencies: bumped pathlib2 to 2.3.3
71 - Dependencies: bumped webob to 1.8.4
72 - Dependencies: bumped iso8601 to 0.1.12
73 - Dependencies: bumped more-itertools to 5.0.0
74 - Dependencies: bumped psutil to 5.4.8
75 - Dependencies: bumped pyasn1 to 0.4.5
76 - Dependencies: bumped pygments to 2.3.1
77 - Dependencies: bumped pyramid-debugtoolbar to 4.5.0
78 - Dependencies: bumped subprocess32 to 3.5.3
79 - Dependencies: bumped supervisor to 3.3.5
80 - Dependencies: bumped dogpile.cache to 0.7.1
81 - Dependencies: bumped simplejson to 3.16.0
82 - Dependencies: bumped gevent to 1.4.0
83 - Dependencies: bumped configparser to 3.5.1
84
85
86 Security
87 ^^^^^^^^
88
89 - Fork page: don't expose fork origin link if we don't have permission to access this repository.
90 Additionally don't pre-select such repository in pull request ref selector.
91 - Security: fix possible XSS in the issue tracker URL.
92 - Security: sanitize plaintext renderer with bleach, preventing XSS in rendered html.
93 - Audit logs: added audit logs for API permission calls.
94
95
96 Performance
97 ^^^^^^^^^^^
98
99 - Summary page: don't load repo size when showing expanded information about repository.
100 Size calculation needs to be triggered manually.
101 - Git: use rev-list for fetching last commit data in case of single commit history.
102 In some cases, it is much faster than previously used git log command.
103
104
105 Fixes
106 ^^^^^
107
108 - Installer: fixed 32bit package builds broken in previous releases.
109 - Git: use iterative fetch to prevent errors about too many arguments on
110 synchronizing very large repositories.
111 - Git: pass in the SSL dir that is exposed from wire for remote GIT commands.
112 - LDAP+Groups: improve logging, and fix the case when extracting group name from LDAP
113 returned nothing. We should warn about that, but not FAIL on login.
114 - Default reviewers: fixed submodule support in picking reviewers from annotation for files.
115 - Hooks: handle non-ascii characters in hooks new pull-requests open template.
116 - Diffs: fixed missing limited diff container display on over-size limit diffs.
117 - Diffs: fixed 500 error in case of some very uncommon diffs containing only Unicode characters.
118 - Repositories: handle VCS backend unavailable correctly in advanced settings for the repository.
119 - Remap & rescan: prevent empty/damaged repositories to break the remap operation.
120 - Visual: fixed show revision/commit length settings.
121 - Mercurial submodules: only show submodule in the path that it belongs too.
122 Before even submodules from root node were shown in subdirectories.
123 - UI/Files: fixed icons in file tree search.
124 - WebHook integration: quote URL variables to prevent URL errors with special chars
125 like # in the title.
126 - API: pull-requests, fixed invocation of merge as another user.
127 - VCS: limit fd leaks on subprocessio calls.
128 - VCS: expose SSL certificate path over the wire to the vcsserver, this solves some
129 remote SSL import problems reported.
130
131
132 Upgrade notes
133 ^^^^^^^^^^^^^
134
135 This release brings the new Full-text search capabilities using ElasticSearch 6.
136 If you use Elastic Search backend a backward compatibility mode is enabled and
137 ElasticSearch backend defaults to previously used ElasticSearch 2.
138
139 To use new features a full index rebuild is required, in addition ```--es-version=6``` flag
140 needs to be used with indexer and ```search.es_version = 6``` should be set in rhodecode.ini
141
142 Additionally new mapping format is available for the indexer that has additional capabilities
143 for include/exclude rules. Old format should work as well, but we encourage to
144 generate a new mapping.ini file using rhodecode-index command, and migrate your repositories
145 to the new format.
146
147 Please refer to the :ref:`indexing-ref` documentation for more details.
148
@@ -0,0 +1,107
1 # -*- coding: utf-8 -*-
2
3 # Copyright (C) 2010-2019 RhodeCode GmbH
4 #
5 # This program is free software: you can redistribute it and/or modify
6 # it under the terms of the GNU Affero General Public License, version 3
7 # (only), as published by the Free Software Foundation.
8 #
9 # This program is distributed in the hope that it will be useful,
10 # but WITHOUT ANY WARRANTY; without even the implied warranty of
11 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 # GNU General Public License for more details.
13 #
14 # You should have received a copy of the GNU Affero General Public License
15 # along with this program. If not, see <http://www.gnu.org/licenses/>.
16 #
17 # This program is dual-licensed. If you wish to learn more about the
18 # RhodeCode Enterprise Edition, including its added features, Support services,
19 # and proprietary license terms, please see https://rhodecode.com/licenses/
20
21
22 import pytest
23
24 from rhodecode.model.db import User, ChangesetComment
25 from rhodecode.model.meta import Session
26 from rhodecode.model.comment import CommentsModel
27 from rhodecode.api.tests.utils import (
28 build_data, api_call, assert_error, assert_call_ok)
29
30
31 @pytest.fixture()
32 def make_repo_comments_factory(request):
33
34 def maker(repo):
35 user = User.get_first_super_admin()
36 commit = repo.scm_instance()[0]
37
38 commit_id = commit.raw_id
39 file_0 = commit.affected_files[0]
40 comments = []
41
42 # general
43 CommentsModel().create(
44 text='General Comment', repo=repo, user=user, commit_id=commit_id,
45 comment_type=ChangesetComment.COMMENT_TYPE_NOTE, send_email=False)
46
47 # inline
48 CommentsModel().create(
49 text='Inline Comment', repo=repo, user=user, commit_id=commit_id,
50 f_path=file_0, line_no='n1',
51 comment_type=ChangesetComment.COMMENT_TYPE_NOTE, send_email=False)
52
53 # todo
54 CommentsModel().create(
55 text='INLINE TODO Comment', repo=repo, user=user, commit_id=commit_id,
56 f_path=file_0, line_no='n1',
57 comment_type=ChangesetComment.COMMENT_TYPE_TODO, send_email=False)
58
59 @request.addfinalizer
60 def cleanup():
61 for comment in comments:
62 Session().delete(comment)
63 return maker
64
65
66 @pytest.mark.usefixtures("testuser_api", "app")
67 class TestGetRepo(object):
68
69 @pytest.mark.parametrize('filters, expected_count', [
70 ({}, 3),
71 ({'comment_type': ChangesetComment.COMMENT_TYPE_NOTE}, 2),
72 ({'comment_type': ChangesetComment.COMMENT_TYPE_TODO}, 1),
73 ({'commit_id': 'FILLED DYNAMIC'}, 3),
74 ])
75 def test_api_get_repo_comments(self, backend, user_util,
76 make_repo_comments_factory, filters, expected_count):
77 commits = [{'message': 'A'}, {'message': 'B'}]
78 repo = backend.create_repo(commits=commits)
79 make_repo_comments_factory(repo)
80
81 api_call_params = {'repoid': repo.repo_name,}
82 api_call_params.update(filters)
83
84 if 'commit_id' in api_call_params:
85 commit = repo.scm_instance()[0]
86 commit_id = commit.raw_id
87 api_call_params['commit_id'] = commit_id
88
89 id_, params = build_data(self.apikey, 'get_repo_comments', **api_call_params)
90 response = api_call(self.app, params)
91 result = assert_call_ok(id_, given=response.body)
92
93 assert len(result) == expected_count
94
95 def test_api_get_repo_comments_wrong_comment_typ(self, backend_hg):
96
97 repo = backend_hg.create_repo()
98 make_repo_comments_factory(repo)
99
100 api_call_params = {'repoid': repo.repo_name,}
101 api_call_params.update({'comment_type': 'bogus'})
102
103 expected = 'comment_type must be one of `{}` got {}'.format(
104 ChangesetComment.COMMENT_TYPES, 'bogus')
105 id_, params = build_data(self.apikey, 'get_repo_comments', **api_call_params)
106 response = api_call(self.app, params)
107 assert_error(id_, expected, given=response.body)
@@ -0,0 +1,59
1 # -*- coding: utf-8 -*-
2
3 # Copyright (C) 2010-2019 RhodeCode GmbH
4 #
5 # This program is free software: you can redistribute it and/or modify
6 # it under the terms of the GNU Affero General Public License, version 3
7 # (only), as published by the Free Software Foundation.
8 #
9 # This program is distributed in the hope that it will be useful,
10 # but WITHOUT ANY WARRANTY; without even the implied warranty of
11 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 # GNU General Public License for more details.
13 #
14 # You should have received a copy of the GNU Affero General Public License
15 # along with this program. If not, see <http://www.gnu.org/licenses/>.
16 #
17 # This program is dual-licensed. If you wish to learn more about the
18 # RhodeCode Enterprise Edition, including its added features, Support services,
19 # and proprietary license terms, please see https://rhodecode.com/licenses/
20
21
22 import pytest
23
24 from rhodecode.api.tests.utils import build_data, api_call, assert_ok, assert_error
25
26
27 @pytest.mark.usefixtures("testuser_api", "app")
28 class TestStoreException(object):
29
30 def test_store_exception_invalid_json(self):
31 id_, params = build_data(self.apikey, 'store_exception',
32 exc_data_json='XXX,{')
33 response = api_call(self.app, params)
34
35 expected = 'Failed to parse JSON data from exc_data_json field. ' \
36 'Please make sure it contains a valid JSON.'
37 assert_error(id_, expected, given=response.body)
38
39 def test_store_exception_missing_json_params_json(self):
40 id_, params = build_data(self.apikey, 'store_exception',
41 exc_data_json='{"foo":"bar"}')
42 response = api_call(self.app, params)
43
44 expected = "Missing exc_traceback, or exc_type_name in " \
45 "exc_data_json field. Missing: 'exc_traceback'"
46 assert_error(id_, expected, given=response.body)
47
48 def test_store_exception(self):
49 id_, params = build_data(
50 self.apikey, 'store_exception',
51 exc_data_json='{"exc_traceback": "invalid", "exc_type_name":"ValueError"}')
52 response = api_call(self.app, params)
53 exc_id = response.json['result']['exc_id']
54
55 expected = {
56 'exc_id': exc_id,
57 'exc_url': 'http://example.com/_admin/settings/exceptions/{}'.format(exc_id)
58 }
59 assert_ok(id_, expected, given=response.body)
@@ -0,0 +1,49
1 # -*- coding: utf-8 -*-
2
3 # Copyright (C) 2016-2019 RhodeCode GmbH
4 #
5 # This program is free software: you can redistribute it and/or modify
6 # it under the terms of the GNU Affero General Public License, version 3
7 # (only), as published by the Free Software Foundation.
8 #
9 # This program is distributed in the hope that it will be useful,
10 # but WITHOUT ANY WARRANTY; without even the implied warranty of
11 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 # GNU General Public License for more details.
13 #
14 # You should have received a copy of the GNU Affero General Public License
15 # along with this program. If not, see <http://www.gnu.org/licenses/>.
16 #
17 # This program is dual-licensed. If you wish to learn more about the
18 # RhodeCode Enterprise Edition, including its added features, Support services,
19 # and proprietary license terms, please see https://rhodecode.com/licenses/
20 import os
21 from rhodecode.apps.file_store import config_keys
22 from rhodecode.config.middleware import _bool_setting, _string_setting
23
24
25 def _sanitize_settings_and_apply_defaults(settings):
26 """
27 Set defaults, convert to python types and validate settings.
28 """
29 _bool_setting(settings, config_keys.enabled, 'true')
30
31 _string_setting(settings, config_keys.backend, 'local')
32
33 default_store = os.path.join(os.path.dirname(settings['__file__']), 'upload_store')
34 _string_setting(settings, config_keys.store_path, default_store)
35
36
37 def includeme(config):
38 settings = config.registry.settings
39 _sanitize_settings_and_apply_defaults(settings)
40
41 config.add_route(
42 name='upload_file',
43 pattern='/_file_store/upload')
44 config.add_route(
45 name='download_file',
46 pattern='/_file_store/download/{fid}')
47
48 # Scan module for configuration decorators.
49 config.scan('.views', ignore='.tests')
@@ -0,0 +1,27
1 # -*- coding: utf-8 -*-
2
3 # Copyright (C) 2016-2019 RhodeCode GmbH
4 #
5 # This program is free software: you can redistribute it and/or modify
6 # it under the terms of the GNU Affero General Public License, version 3
7 # (only), as published by the Free Software Foundation.
8 #
9 # This program is distributed in the hope that it will be useful,
10 # but WITHOUT ANY WARRANTY; without even the implied warranty of
11 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 # GNU General Public License for more details.
13 #
14 # You should have received a copy of the GNU Affero General Public License
15 # along with this program. If not, see <http://www.gnu.org/licenses/>.
16 #
17 # This program is dual-licensed. If you wish to learn more about the
18 # RhodeCode Enterprise Edition, including its added features, Support services,
19 # and proprietary license terms, please see https://rhodecode.com/licenses/
20
21
22 # Definition of setting keys used to configure this module. Defined here to
23 # avoid repetition of keys throughout the module.
24
25 enabled = 'file_store.enabled'
26 backend = 'file_store.backend'
27 store_path = 'file_store.storage_path'