release: Merge default into stable for release preparation

docs/admin/restore-deleted-repositories.rst

@@ -0,0 +1,42 b''
+.. _restore-deleted-repositories:
+
+Restoring Deleted Repositories
+==============================
+
+By default when repository or whole repository group is deleted an archived copy
+of filesystem repositories are kept. You can see them as special entries in the
+repository storage such as::
+
+    drwxrwxr-x   3 rcdev rcdev  4096 Dec  4  2017 rm__20171204_105727_400795__ce-import
+    drwxrwxr-x   6 rcdev rcdev  4096 Nov 21  2017 rm__20180221_152430_675047__svn-repo
+    drwxr-xr-x   7 rcdev rcdev  4096 Mar 28  2018 rm__20180328_143124_617576__test-git
+    drwxr-xr-x   7 rcdev rcdev  4096 Mar 28  2018 rm__20180328_144954_317729__test-git-install-hooks
+
+
+Data from those repositories can be restored by simply removing the
+`rm_YYYYDDMM_HHMMSS_DDDDDD__` prefix and additionally only in case of Mercurial
+repositories remove the `.hg` store prefix.::
+
+    rm__.hg => .hg
+
+
+For Git or SVN repositories this operation is not required.
+
+After removing the prefix repository can be brought by opening
+:menuselection:`Admin --> Settings --> Remap and Rescan` and running `Rescan Filesystem`
+
+This will create a new DB entry restoring the data previously removed.
+To restore OLD database entries this should be done by restoring from a Database backup.
+
+RhodeCode also keeps the repository group structure, this is marked by entries that
+in addition have GROUP in the prefix, eg::
+
+    drwxr-xr-x   2 rcdev rcdev  4096 Jan 18 16:13 rm__20181130_120650_977082_GROUP_Test1
+    drwxr-xr-x   2 rcdev rcdev  4096 Jan 18 16:13 rm__20181130_120659_922952_GROUP_Docs
+
+
+
+.. note::
+
+    RhodeCode Tools have a special cleanup tool for the archived repositories. Please
+    see :ref:`clean-up-cmds`

docs/api/methods/store-methods.rst

@@ -0,0 +1,37 b''
+.. _store-methods-ref:
+
+store methods
+=============
+
+file_store_add (EE only)
+------------------------
+
+.. py:function:: file_store_add(apiuser, filename, content)
+
+   Upload API for the file_store
+
+   Example usage from CLI::
+       rhodecode-api --instance-name=enterprise-1 upload_file "{"content": "$(cat image.jpg | base64)", "filename":"image.jpg"}"
+
+   This command takes the following options:
+
+   :param apiuser: This is filled automatically from the |authtoken|.
+   :type apiuser: AuthUser
+   :param filename: name of the file uploaded
+   :type filename: str
+   :param content: base64 encoded content of the uploaded file
+   :type content: str
+
+   Example output:
+
+   .. code-block:: bash
+
+     id : <id_given_in_input>
+     result: {
+       "access_path": "/_file_store/download/84d156f7-8323-4ad3-9fce-4a8e88e1deaf-0.jpg",
+       "access_path_fqn": "http://server.domain.com/_file_store/download/84d156f7-8323-4ad3-9fce-4a8e88e1deaf-0.jpg",
+       "store_fid": "84d156f7-8323-4ad3-9fce-4a8e88e1deaf-0.jpg"
+     }
+     error :  null
+
+

docs/auth/auth-saml-bulk-enroll-users.rst

@@ -0,0 +1,88 b''
+.. _auth-saml-bulk-enroll-users-ref:
+
+
+Bulk enroll multiple existing users
+-----------------------------------
+
+
+RhodeCode Supports standard SAML 2.0 SSO for the web-application part.
+Below is an example how to enroll list of all or some users to use SAML authentication.
+This method simply enables SAML authentication for many users at once.
+
+
+From the server RhodeCode Enterprise is running run ishell on the instance which we
+want to apply the SAML migration::
+
+    rccontrol ishell enterprise-1
+
+Follow these steps to enable SAML authentication for multiple users.
+
+
+1) Create a user_id => attribute mapping
+
+
+`saml2user` is a mapping of external ID from SAML provider such as OneLogin, DuoSecurity, Google.
+This mapping consists of local rhodecode user_id mapped to set of required attributes needed to bind SAML
+account to internal rhodecode user.
+For example, 123 is local rhodecode user_id, and '48253211' is OneLogin ID.
+For other providers you'd have to figure out what would be the user-id, sometimes it's the email, i.e for Google
+The most important this id needs to be unique for each user.
+
+.. code-block:: python
+
+    In [1]: saml2user = {
+       ...: # OneLogin, uses externalID available to read from in the UI
+       ...: 123: {'id: '48253211'},
+       ...: # for Google/DuoSecurity email is also an option for unique ID
+       ...: 124: {'id: '[e-mail]'},
+       ...: }
+
+
+2) Import the plugin you want to run migration for.
+
+From available options pick only one and run the `import` statement
+
+.. code-block:: python
+
+    # for Duo Security
+    In [2]: from rc_auth_plugins.auth_duo_security import RhodeCodeAuthPlugin
+    # for OneLogin
+    In [2]: from rc_auth_plugins.auth_onelogin import RhodeCodeAuthPlugin
+    # generic SAML plugin
+    In [2]: from rc_auth_plugins.auth_saml import RhodeCodeAuthPlugin
+
+3) Run the migration based on saml2user mapping.
+
+Enter in the ishell prompt
+
+.. code-block:: python
+
+    In [3]: for user in User.get_all():
+       ...:     existing_identity = ExternalIdentity().query().filter(ExternalIdentity.local_user_id == user.user_id).scalar()
+       ...:     attrs = saml2user.get(user.user_id)
+       ...:     provider = RhodeCodeAuthPlugin.uid
+       ...:     if existing_identity:
+       ...:         print('Identity for user `{}` already exists, skipping'.format(user.username))
+       ...:         continue
+       ...:     if attrs:
+       ...:         external_id = attrs['id']
+       ...:         new_external_identity = ExternalIdentity()
+       ...:         new_external_identity.external_id = external_id
+       ...:         new_external_identity.external_username = '{}-saml-{}'.format(user.username, user.user_id)
+       ...:         new_external_identity.provider_name = provider
+       ...:         new_external_identity.local_user_id = user_id
+       ...:         new_external_identity.access_token = ''
+       ...:         new_external_identity.token_secret = ''
+       ...:         new_external_identity.alt_token = ''
+       ...:         Session().add(ex_identity)
+       ...:         Session().commit()
+       ...:         print('Set user `{}` external identity bound to ExternalID:{}'.format(user.username, external_id))
+
+.. note::
+
+    saml2user can be really big and hard to maintain in ishell. It's also possible
+    to load it as a JSON file prepared before and stored on disk. To do so run::
+
+        import json
+        saml2user = json.loads(open('/path/to/saml2user.json','rb').read())
+

docs/release-notes/release-notes-4.16.0.rst

@@ -0,0 +1,148 b''
+|RCE| 4.16.0 |RNS|
+------------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2019-02-15
+
+
+New Features
+^^^^^^^^^^^^
+
+
+- Full-text search: added support for ElasticSearch 6.X (ES6)
+- Full-text search: Expose a quick way to search within repository groups using ES6.
+- Full-text search: Add quick links to broaden/narrow search scope to repositories or
+  repository groups from global search.
+- Full-text search: ES6 backend adds new highlighter, and search markers for better UX when searching.
+- Full-text search: ES6 backend has enabled advanced `query string syntax`
+  adding more search and filtering capabilities.
+- Full-text search: ES6 engine will now show added information where available such as line numbers file size.
+- Files: added option to use highlight marker to show keywords inside file source. This
+  is used now for ES6 backend extended highlighting capabilities
+- Artifacts (beta): EE edition exposes new feature called storage_api this allows storing
+  binary files outside of Version Control System, but in the scope of a repository or group.
+  This will soon become an Artifacts functionality available in EE edition.
+- Authentication: introduced `User restriction` and `Scope restriction` for RhodeCode authentication plugins.
+  Admins can limit usage of RhodeCode plugins to super-admins user types, and usage in Web, or VCS protocol only.
+  This is mostly to help to migrate users to SAML, keeping the super-admins to manage instances via local-logins,
+  and secondly to force usage of AuthenticationTokens instead of re-using same credentials for
+  WEB and VCS authentication.
+- API: added basic upload API for the storage_api. It's possible to store files using internal
+  API. This is a start for attachments upload in RhodeCode.
+- API: added store_exception_api for remote exception storage. This is used by a new
+  indexer that will report any problems back into the RhodeCode instance in case of indexing problems.
+- API: added function to fetch comments for a repository.
+- Quick search: improve the styling of search input and results.
+- Pull requests: allowed to select all forks and parent forks of target repository in creation UI.
+  This is a common workflow supported by GitHub etc.
+
+
+General
+^^^^^^^
+
+- Users/Repositories/Repository groups: expose IDs of those objects in advanced views.
+  Useful for API calls or usage in ishell.
+- UI: moved repo group select next to the name as it's very relevant to each other.
+- Pull requests: increase the stability of concurrent pull requests created.
+- Pull requests: introduced operation state for pull requests to prevent from
+  locks during merge/update operations in concurrent busy environments.
+- Pull requests: ensure that merge response provide more details about failed operations.
+- UI / Files: expose downloads options onto files view similar as in summary page.
+- Repositories: show hooks version and update link in the advanced section of repository page.
+- Events: trigger 'review_status_change' in all cases when reviewers are changed
+  influencing review status.
+- Files: display submodules in a sorted way, equal to how Directories are sorted.
+- API: fetching all pull-requests now sorts the results and exposed a flag to show/hide
+  the merge result state for faster result fetching.
+- API: merge_pull_request expose detailed merge message in the merge operation
+  next to numeric merge response code.
+- API: added possibility to specify owner to create_pull_request API.
+- SSH: Added ability to disable server-side SSH key generation to enforce users
+  generated SSH keys only outside of the server.
+- Integrations: allow PUT method for WebHook integration.
+- Dependencies: bumped git to 2.19.2 release.
+- Dependencies: dropped pygments-markdown-lexer as it's natively supported by pygments now.
+- Dependencies: bumped pyramid to 1.10.1
+- Dependencies: bumped pastedeploy to 2.0.1
+- Dependencies: bumped pastescript to 3.0.0
+- Dependencies: bumped pathlib2 to 2.3.3
+- Dependencies: bumped webob to 1.8.4
+- Dependencies: bumped iso8601 to 0.1.12
+- Dependencies: bumped more-itertools to 5.0.0
+- Dependencies: bumped psutil to 5.4.8
+- Dependencies: bumped pyasn1 to 0.4.5
+- Dependencies: bumped pygments to 2.3.1
+- Dependencies: bumped pyramid-debugtoolbar to 4.5.0
+- Dependencies: bumped subprocess32 to 3.5.3
+- Dependencies: bumped supervisor to 3.3.5
+- Dependencies: bumped dogpile.cache to 0.7.1
+- Dependencies: bumped simplejson to 3.16.0
+- Dependencies: bumped gevent to 1.4.0
+- Dependencies: bumped configparser to 3.5.1
+
+
+Security
+^^^^^^^^
+
+- Fork page: don't expose fork origin link if we don't have permission to access this repository.
+  Additionally don't pre-select such repository in pull request ref selector.
+- Security: fix possible XSS in the issue tracker URL.
+- Security: sanitize plaintext renderer with bleach, preventing XSS in rendered html.
+- Audit logs: added audit logs for API permission calls.
+
+
+Performance
+^^^^^^^^^^^
+
+- Summary page: don't load repo size when showing expanded information about repository.
+  Size calculation needs to be triggered manually.
+- Git: use rev-list for fetching last commit data in case of single commit history.
+  In some cases, it is much faster than previously used git log command.
+
+
+Fixes
+^^^^^
+
+- Installer: fixed 32bit package builds broken in previous releases.
+- Git: use iterative fetch to prevent errors about too many arguments on
+  synchronizing very large repositories.
+- Git: pass in the SSL dir that is exposed from wire for remote GIT commands.
+- LDAP+Groups: improve logging, and fix the case when extracting group name from LDAP
+  returned nothing. We should warn about that, but not FAIL on login.
+- Default reviewers: fixed submodule support in picking reviewers from annotation for files.
+- Hooks: handle non-ascii characters in hooks new pull-requests open template.
+- Diffs: fixed missing limited diff container display on over-size limit diffs.
+- Diffs: fixed 500 error in case of some very uncommon diffs containing only Unicode characters.
+- Repositories: handle VCS backend unavailable correctly in advanced settings for the repository.
+- Remap & rescan: prevent empty/damaged repositories to break the remap operation.
+- Visual: fixed show revision/commit length settings.
+- Mercurial submodules: only show submodule in the path that it belongs too.
+  Before even submodules from root node were shown in subdirectories.
+- UI/Files: fixed icons in file tree search.
+- WebHook integration: quote URL variables to prevent URL errors with special chars
+  like # in the title.
+- API: pull-requests, fixed invocation of merge as another user.
+- VCS: limit fd leaks on subprocessio calls.
+- VCS: expose SSL certificate path over the wire to the vcsserver, this solves some
+  remote SSL import problems reported.
+
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+This release brings the new Full-text search capabilities using ElasticSearch 6.
+If you use Elastic Search backend a backward compatibility mode is enabled and
+ElasticSearch backend defaults to previously used ElasticSearch 2.
+
+To use new features a full index rebuild is required, in addition ```--es-version=6``` flag
+needs to be used with indexer and ```search.es_version = 6``` should be set in rhodecode.ini
+
+Additionally new mapping format is available for the indexer that has additional capabilities
+for include/exclude rules. Old format should work as well, but we encourage to
+generate a new mapping.ini file using rhodecode-index command, and migrate your repositories
+to the new format.
+
+Please refer to the :ref:`indexing-ref` documentation for more details.
+

rhodecode/api/tests/test_get_repo_comments.py

@@ -0,0 +1,107 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+import pytest
+
+from rhodecode.model.db import User, ChangesetComment
+from rhodecode.model.meta import Session
+from rhodecode.model.comment import CommentsModel
+from rhodecode.api.tests.utils import (
+    build_data, api_call, assert_error, assert_call_ok)
+
+
[e-mail]()
+def make_repo_comments_factory(request):
+
+    def maker(repo):
+        user = User.get_first_super_admin()
+        commit = repo.scm_instance()[0]
+
+        commit_id = commit.raw_id
+        file_0 = commit.affected_files[0]
+        comments = []
+
+        # general
+        CommentsModel().create(
+            text='General Comment', repo=repo, user=user, commit_id=commit_id,
+            comment_type=ChangesetComment.COMMENT_TYPE_NOTE, send_email=False)
+
+        # inline
+        CommentsModel().create(
+            text='Inline Comment', repo=repo, user=user, commit_id=commit_id,
+            f_path=file_0, line_no='n1',
+            comment_type=ChangesetComment.COMMENT_TYPE_NOTE, send_email=False)
+
+        # todo
+        CommentsModel().create(
+            text='INLINE TODO Comment', repo=repo, user=user, commit_id=commit_id,
+            f_path=file_0, line_no='n1',
+            comment_type=ChangesetComment.COMMENT_TYPE_TODO, send_email=False)
+
+        @request.addfinalizer
+        def cleanup():
+            for comment in comments:
+                Session().delete(comment)
+    return maker
+
+
[e-mail]("testuser_api", "app")
+class TestGetRepo(object):
+
+    @pytest.mark.parametrize('filters, expected_count', [
+        ({}, 3),
+        ({'comment_type': ChangesetComment.COMMENT_TYPE_NOTE}, 2),
+        ({'comment_type': ChangesetComment.COMMENT_TYPE_TODO}, 1),
+        ({'commit_id': 'FILLED DYNAMIC'}, 3),
+    ])
+    def test_api_get_repo_comments(self, backend, user_util,
+                                   make_repo_comments_factory, filters, expected_count):
+        commits = [{'message': 'A'}, {'message': 'B'}]
+        repo = backend.create_repo(commits=commits)
+        make_repo_comments_factory(repo)
+
+        api_call_params = {'repoid': repo.repo_name,}
+        api_call_params.update(filters)
+
+        if 'commit_id' in api_call_params:
+            commit = repo.scm_instance()[0]
+            commit_id = commit.raw_id
+            api_call_params['commit_id'] = commit_id
+
+        id_, params = build_data(self.apikey, 'get_repo_comments', **api_call_params)
+        response = api_call(self.app, params)
+        result = assert_call_ok(id_, given=response.body)
+
+        assert len(result) == expected_count
+
+    def test_api_get_repo_comments_wrong_comment_typ(self, backend_hg):
+
+        repo = backend_hg.create_repo()
+        make_repo_comments_factory(repo)
+
+        api_call_params = {'repoid': repo.repo_name,}
+        api_call_params.update({'comment_type': 'bogus'})
+
+        expected = 'comment_type must be one of `{}` got {}'.format(
+                    ChangesetComment.COMMENT_TYPES, 'bogus')
+        id_, params = build_data(self.apikey, 'get_repo_comments', **api_call_params)
+        response = api_call(self.app, params)
+        assert_error(id_, expected, given=response.body)

rhodecode/api/tests/test_store_exception.py

@@ -0,0 +1,59 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+import pytest
+
+from rhodecode.api.tests.utils import build_data, api_call, assert_ok, assert_error
+
+
[e-mail]("testuser_api", "app")
+class TestStoreException(object):
+
+    def test_store_exception_invalid_json(self):
+        id_, params = build_data(self.apikey, 'store_exception',
+                                 exc_data_json='XXX,{')
+        response = api_call(self.app, params)
+
+        expected = 'Failed to parse JSON data from exc_data_json field. ' \
+                   'Please make sure it contains a valid JSON.'
+        assert_error(id_, expected, given=response.body)
+
+    def test_store_exception_missing_json_params_json(self):
+        id_, params = build_data(self.apikey, 'store_exception',
+                                 exc_data_json='{"foo":"bar"}')
+        response = api_call(self.app, params)
+
+        expected = "Missing exc_traceback, or exc_type_name in " \
+                   "exc_data_json field. Missing: 'exc_traceback'"
+        assert_error(id_, expected, given=response.body)
+
+    def test_store_exception(self):
+        id_, params = build_data(
+            self.apikey, 'store_exception',
+            exc_data_json='{"exc_traceback": "invalid", "exc_type_name":"ValueError"}')
+        response = api_call(self.app, params)
+        exc_id = response.json['result']['exc_id']
+
+        expected = {
+            'exc_id': exc_id,
+            'exc_url': 'http://example.com/_admin/settings/exceptions/{}'.format(exc_id)
+        }
+        assert_ok(id_, expected, given=response.body)

rhodecode/apps/file_store/__init__.py

@@ -0,0 +1,49 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import os
+from rhodecode.apps.file_store import config_keys
+from rhodecode.config.middleware import _bool_setting, _string_setting
+
+
+def _sanitize_settings_and_apply_defaults(settings):
+    """
+    Set defaults, convert to python types and validate settings.
+    """
+    _bool_setting(settings, config_keys.enabled, 'true')
+
+    _string_setting(settings, config_keys.backend, 'local')
+
+    default_store = os.path.join(os.path.dirname(settings['__file__']), 'upload_store')
+    _string_setting(settings, config_keys.store_path, default_store)
+
+
+def includeme(config):
+    settings = config.registry.settings
+    _sanitize_settings_and_apply_defaults(settings)
+
+    config.add_route(
+        name='upload_file',
+        pattern='/_file_store/upload')
+    config.add_route(
+        name='download_file',
+        pattern='/_file_store/download/{fid}')
+
+    # Scan module for configuration decorators.
+    config.scan('.views', ignore='.tests')

rhodecode/apps/file_store/config_keys.py

@@ -0,0 +1,27 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+# Definition of setting keys used to configure this module. Defined here to
+# avoid repetition of keys throughout the module.
+
+enabled = 'file_store.enabled'
+backend = 'file_store.backend'
+store_path = 'file_store.storage_path'

rhodecode/apps/file_store/exceptions.py

@@ -0,0 +1,31 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+class FileNotAllowedException(Exception):
+    """
+    Thrown if file does not have an allowed extension.
+    """
+
+
+class FileOverSizeException(Exception):
+    """
+    Thrown if file is over the set limit.
+    """

rhodecode/apps/file_store/extensions.py

@@ -0,0 +1,66 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+ANY = []
+TEXT_EXT = ['txt', 'md', 'rst', 'log']
+DOCUMENTS_EXT = ['pdf', 'rtf', 'odf', 'ods', 'gnumeric', 'abw', 'doc', 'docx', 'xls', 'xlsx']
+IMAGES_EXT = ['jpg', 'jpe', 'jpeg', 'png', 'gif', 'svg', 'bmp', 'tiff']
+AUDIO_EXT = ['wav', 'mp3', 'aac', 'ogg', 'oga', 'flac']
+VIDEO_EXT = ['mpeg', '3gp', 'avi', 'divx', 'dvr', 'flv', 'mp4', 'wmv']
+DATA_EXT = ['csv', 'ini', 'json', 'plist', 'xml', 'yaml', 'yml']
+SCRIPTS_EXT = ['js', 'php', 'pl', 'py', 'rb', 'sh', 'go', 'c', 'h']
+ARCHIVES_EXT = ['gz', 'bz2', 'zip', 'tar', 'tgz', 'txz', '7z']
+EXECUTABLES_EXT = ['so', 'exe', 'dll']
+
+
+DEFAULT = DOCUMENTS_EXT + TEXT_EXT + IMAGES_EXT + DATA_EXT
+
+GROUPS = dict((
+    ('any', ANY),
+    ('text', TEXT_EXT),
+    ('documents', DOCUMENTS_EXT),
+    ('images', IMAGES_EXT),
+    ('audio', AUDIO_EXT),
+    ('video', VIDEO_EXT),
+    ('data', DATA_EXT),
+    ('scripts', SCRIPTS_EXT),
+    ('archives', ARCHIVES_EXT),
+    ('executables', EXECUTABLES_EXT),
+    ('default', DEFAULT),
+))
+
+
+def resolve_extensions(extensions, groups=None):
+    """
+    Calculate allowed extensions based on a list of extensions provided, and optional
+    groups of extensions from the available lists.
+
+    :param extensions: a list of extensions e.g ['py', 'txt']
+    :param groups: additionally groups to extend the extensions.
+    """
+    groups = groups or []
+    valid_exts = set([x.lower() for x in extensions])
+
+    for group in groups:
+        if group in GROUPS:
+            valid_exts.update(GROUPS[group])
+
+    return valid_exts

rhodecode/apps/file_store/local_store.py

@@ -0,0 +1,211 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import os
+import time
+import shutil
+import hashlib
+
+from rhodecode.lib.ext_json import json
+from rhodecode.apps.file_store import utils
+from rhodecode.apps.file_store.extensions import resolve_extensions
+from rhodecode.apps.file_store.exceptions import FileNotAllowedException
+
+METADATA_VER = 'v1'
+
+
+class LocalFileStorage(object):
+
+    @classmethod
+    def resolve_name(cls, name, directory):
+        """
+        Resolves a unique name and the correct path. If a filename
+        for that path already exists then a numeric prefix with values > 0 will be
+        added, for example test.jpg -> test-1.jpg etc. initially file would have 0 prefix.
+
+        :param name: base name of file
+        :param directory: absolute directory path
+        """
+
+        basename, ext = os.path.splitext(name)
+        counter = 0
+        while True:
+            name = '%s-%d%s' % (basename, counter, ext)
+
+            # sub_store prefix to optimize disk usage, e.g some_path/ab/final_file
+            sub_store = cls._sub_store_from_filename(basename)
+            sub_store_path = os.path.join(directory, sub_store)
+            if not os.path.exists(sub_store_path):
+                os.makedirs(sub_store_path)
+
+            path = os.path.join(sub_store_path, name)
+            if not os.path.exists(path):
+                return name, path
+            counter += 1
+
+    @classmethod
+    def _sub_store_from_filename(cls, filename):
+        return filename[:2]
+
+    @classmethod
+    def calculate_path_hash(cls, file_path):
+        """
+        Efficient calculation of file_path sha256 sum
+
+        :param file_path:
+        :return: sha256sum
+        """
+        digest = hashlib.sha256()
+        with open(file_path, 'rb') as f:
+            for chunk in iter(lambda: f.read(1024 * 100), b""):
+                digest.update(chunk)
+
+        return digest.hexdigest()
+
+    def __init__(self, base_path, extension_groups=None):
+
+        """
+        Local file storage
+
+        :param base_path: the absolute base path where uploads are stored
+        :param extension_groups: extensions string
+        """
+
+        extension_groups = extension_groups or ['any']
+        self.base_path = base_path
+        self.extensions = resolve_extensions([], groups=extension_groups)
+
+    def store_path(self, filename):
+        """
+        Returns absolute file path of the filename, joined to the
+        base_path.
+
+        :param filename: base name of file
+        """
+        sub_store = self._sub_store_from_filename(filename)
+        return os.path.join(self.base_path, sub_store, filename)
+
+    def delete(self, filename):
+        """
+        Deletes the filename. Filename is resolved with the
+        absolute path based on base_path. If file does not exist,
+        returns **False**, otherwise **True**
+
+        :param filename: base name of file
+        """
+        if self.exists(filename):
+            os.remove(self.store_path(filename))
+            return True
+        return False
+
+    def exists(self, filename):
+        """
+        Checks if file exists. Resolves filename's absolute
+        path based on base_path.
+
+        :param filename: base name of file
+        """
+        return os.path.exists(self.store_path(filename))
+
+    def filename_allowed(self, filename, extensions=None):
+        """Checks if a filename has an allowed extension
+
+        :param filename: base name of file
+        :param extensions: iterable of extensions (or self.extensions)
+        """
+        _, ext = os.path.splitext(filename)
+        return self.extension_allowed(ext, extensions)
+
+    def extension_allowed(self, ext, extensions=None):
+        """
+        Checks if an extension is permitted. Both e.g. ".jpg" and
+        "jpg" can be passed in. Extension lookup is case-insensitive.
+
+        :param ext: extension to check
+        :param extensions: iterable of extensions to validate against (or self.extensions)
+        """
+
+        extensions = extensions or self.extensions
+        if not extensions:
+            return True
+        if ext.startswith('.'):
+            ext = ext[1:]
+        return ext.lower() in extensions
+
+    def save_file(self, file_obj, filename, directory=None, extensions=None,
+                  extra_metadata=None, **kwargs):
+        """
+        Saves a file object to the uploads location.
+        Returns the resolved filename, i.e. the directory +
+        the (randomized/incremented) base name.
+
+        :param file_obj: **cgi.FieldStorage** object (or similar)
+        :param filename: original filename
+        :param directory: relative path of sub-directory
+        :param extensions: iterable of allowed extensions, if not default
+        :param extra_metadata: extra JSON metadata to store next to the file with .meta suffix
+        """
+
+        extensions = extensions or self.extensions
+
+        if not self.filename_allowed(filename, extensions):
+            raise FileNotAllowedException()
+
+        if directory:
+            dest_directory = os.path.join(self.base_path, directory)
+        else:
+            dest_directory = self.base_path
+
+        if not os.path.exists(dest_directory):
+            os.makedirs(dest_directory)
+
+        filename = utils.uid_filename(filename)
+
+        # resolve also produces special sub-dir for file optimized store
+        filename, path = self.resolve_name(filename, dest_directory)
+        stored_file_dir = os.path.dirname(path)
+
+        file_obj.seek(0)
+
+        with open(path, "wb") as dest:
+            shutil.copyfileobj(file_obj, dest)
+
+        metadata = {}
+        if extra_metadata:
+            metadata = extra_metadata
+
+        size = os.stat(path).st_size
+        file_hash = self.calculate_path_hash(path)
+
+        metadata.update(
+            {"filename": filename,
+             "size": size,
+             "time": time.time(),
+             "sha256": file_hash,
+             "meta_ver": METADATA_VER})
+
+        filename_meta = filename + '.meta'
+        with open(os.path.join(stored_file_dir, filename_meta), "wb") as dest_meta:
+            dest_meta.write(json.dumps(metadata))
+
+        if directory:
+            filename = os.path.join(directory, filename)
+
+        return filename, metadata

rhodecode/apps/file_store/tests/__init__.py

@@ -0,0 +1,20 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+

rhodecode/apps/file_store/tests/test_upload_file.py

@@ -0,0 +1,111 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import os
+import pytest
+
+from rhodecode.lib.ext_json import json
+from rhodecode.tests import TestController
+from rhodecode.apps.file_store import utils, config_keys
+
+
+def route_path(name, params=None, **kwargs):
+    import urllib
+
+    base_url = {
+        'upload_file': '/_file_store/upload',
+        'download_file': '/_file_store/download/{fid}',
+
+    }[name].format(**kwargs)
+
+    if params:
+        base_url = '{}?{}'.format(base_url, urllib.urlencode(params))
+    return base_url
+
+
+class TestFileStoreViews(TestController):
+
+    @pytest.mark.parametrize("fid, content, exists", [
+        ('abcde-0.jpg', "xxxxx", True),
+        ('abcde-0.exe', "1234567", True),
+        ('abcde-0.jpg', "xxxxx", False),
+    ])
+    def test_get_files_from_store(self, fid, content, exists, tmpdir):
+        self.log_user()
+        store_path = self.app._pyramid_settings[config_keys.store_path]
+
+        if exists:
+            status = 200
+            store = utils.get_file_storage({config_keys.store_path: store_path})
+            filesystem_file = os.path.join(str(tmpdir), fid)
+            with open(filesystem_file, 'wb') as f:
+                f.write(content)
+
+            with open(filesystem_file, 'rb') as f:
+                fid, metadata = store.save_file(f, fid, extra_metadata={'filename': fid})
+
+        else:
+            status = 404
+
+        response = self.app.get(route_path('download_file', fid=fid), status=status)
+
+        if exists:
+            assert response.text == content
+            file_store_path = os.path.dirname(store.resolve_name(fid, store_path)[1])
+            metadata_file = os.path.join(file_store_path, fid + '.meta')
+            assert os.path.exists(metadata_file)
+            with open(metadata_file, 'rb') as f:
+                json_data = json.loads(f.read())
+
+            assert json_data
+            assert 'size' in json_data
+
+    def test_upload_files_without_content_to_store(self):
+        self.log_user()
+        response = self.app.post(
+            route_path('upload_file'),
+            params={'csrf_token': self.csrf_token},
+            status=200)
+
+        assert response.json == {
+            u'error': u'store_file data field is missing',
+            u'access_path': None,
+            u'store_fid': None}
+
+    def test_upload_files_bogus_content_to_store(self):
+        self.log_user()
+        response = self.app.post(
+            route_path('upload_file'),
+            params={'csrf_token': self.csrf_token, 'store_file': 'bogus'},
+            status=200)
+
+        assert response.json == {
+            u'error': u'filename cannot be read from the data field',
+            u'access_path': None,
+            u'store_fid': None}
+
+    def test_upload_content_to_store(self):
+        self.log_user()
+        response = self.app.post(
+            route_path('upload_file'),
+            upload_files=[('store_file', 'myfile.txt', 'SOME CONTENT')],
+            params={'csrf_token': self.csrf_token},
+            status=200)
+
+        assert response.json['store_fid']

rhodecode/apps/file_store/utils.py

@@ -0,0 +1,47 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+import os
+import uuid
+
+
+def get_file_storage(settings):
+    from rhodecode.apps.file_store.local_store import LocalFileStorage
+    from rhodecode.apps.file_store import config_keys
+    store_path = settings.get(config_keys.store_path)
+    return LocalFileStorage(base_path=store_path)
+
+
+def uid_filename(filename, randomized=True):
+    """
+    Generates a randomized or stable (uuid) filename,
+    preserving the original extension.
+
+    :param filename: the original filename
+    :param randomized: define if filename should be stable (sha1 based) or randomized
+    """
+    _, ext = os.path.splitext(filename)
+    if randomized:
+        uid = uuid.uuid4()
+    else:
+        hash_key = '{}.{}'.format(filename, 'store')
+        uid = uuid.uuid5(uuid.NAMESPACE_URL, hash_key)
+    return str(uid) + ext.lower()

rhodecode/apps/file_store/views.py

@@ -0,0 +1,115 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import logging
+
+from pyramid.view import view_config
+from pyramid.response import FileResponse
+from pyramid.httpexceptions import HTTPFound, HTTPNotFound
+
+from rhodecode.apps._base import BaseAppView
+from rhodecode.apps.file_store import utils
+from rhodecode.apps.file_store.exceptions import (
+    FileNotAllowedException, FileOverSizeException)
+
+from rhodecode.lib import helpers as h
+from rhodecode.lib import audit_logger
+from rhodecode.lib.auth import (CSRFRequired, NotAnonymous)
+from rhodecode.model.db import Session, FileStore
+
+log = logging.getLogger(__name__)
+
+
+class FileStoreView(BaseAppView):
+    upload_key = 'store_file'
+
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+        self.storage = utils.get_file_storage(self.request.registry.settings)
+        return c
+
+    @NotAnonymous()
+    @CSRFRequired()
+    @view_config(route_name='upload_file', request_method='POST', renderer='json_ext')
+    def upload_file(self):
+        self.load_default_context()
+        file_obj = self.request.POST.get(self.upload_key)
+
+        if file_obj is None:
+            return {'store_fid': None,
+                    'access_path': None,
+                    'error': '{} data field is missing'.format(self.upload_key)}
+
+        if not hasattr(file_obj, 'filename'):
+            return {'store_fid': None,
+                    'access_path': None,
+                    'error': 'filename cannot be read from the data field'}
+
+        filename = file_obj.filename
+
+        metadata = {
+            'user_uploaded': {'username': self._rhodecode_user.username,
+                              'user_id': self._rhodecode_user.user_id,
+                              'ip': self._rhodecode_user.ip_addr}}
+        try:
+            store_fid, metadata = self.storage.save_file(
+                file_obj.file, filename, extra_metadata=metadata)
+        except FileNotAllowedException:
+            return {'store_fid': None,
+                    'access_path': None,
+                    'error': 'File {} is not allowed.'.format(filename)}
+
+        except FileOverSizeException:
+            return {'store_fid': None,
+                    'access_path': None,
+                    'error': 'File {} is exceeding allowed limit.'.format(filename)}
+
+        try:
+            entry = FileStore.create(
+                file_uid=store_fid, filename=metadata["filename"],
+                file_hash=metadata["sha256"], file_size=metadata["size"],
+                file_description='upload attachment',
+                check_acl=False, user_id=self._rhodecode_user.user_id
+            )
+            Session().add(entry)
+            Session().commit()
+            log.debug('Stored upload in DB as %s', entry)
+        except Exception:
+            log.exception('Failed to store file %s', filename)
+            return {'store_fid': None,
+                    'access_path': None,
+                    'error': 'File {} failed to store in DB.'.format(filename)}
+
+        return {'store_fid': store_fid,
+                'access_path': h.route_path('download_file', fid=store_fid)}
+
+    @view_config(route_name='download_file')
+    def download_file(self):
+        self.load_default_context()
+        file_uid = self.request.matchdict['fid']
+        log.debug('Requesting FID:%s from store %s', file_uid, self.storage)
+
+        if not self.storage.exists(file_uid):
+            log.debug('File with FID:%s not found in the store', file_uid)
+            raise HTTPNotFound()
+
+        FileStore.bump_access_counter(file_uid)
+
+        file_path = self.storage.store_path(file_uid)
+        return FileResponse(file_path)

rhodecode/config/rcextensions/examples/validate_pushed_files_name_and_size.py

@@ -0,0 +1,110 b''
+# Example to validate pushed files names and size using some sort of rules
+
+
+
+@has_kwargs({
+    'server_url': 'url of instance that triggered this hook',
+    'config': 'path to .ini config used',
+    'scm': 'type of version control "git", "hg", "svn"',
+    'username': 'username of actor who triggered this event',
+    'ip': 'ip address of actor who triggered this hook',
+    'action': '',
+    'repository': 'repository name',
+    'repo_store_path': 'full path to where repositories are stored',
+    'commit_ids': 'pre transaction metadata for commit ids',
+    'hook_type': '',
+    'user_agent': 'Client user agent, e.g git or mercurial CLI version',
+})
+def _pre_push_hook(*args, **kwargs):
+    """
+    Post push hook
+    To stop version control from storing the transaction and send a message to user
+    use non-zero HookResponse with a message, e.g return HookResponse(1, 'Not allowed')
+
+    This message will be shown back to client during PUSH operation
+
+    Commit ids might look like that::
+
+    [{u'hg_env|git_env': ...,
+      u'multiple_heads': [],
+      u'name': u'default',
+      u'new_rev': u'd0befe0692e722e01d5677f27a104631cf798b69',
+      u'old_rev': u'd0befe0692e722e01d5677f27a104631cf798b69',
+      u'ref': u'',
+      u'total_commits': 2,
+      u'type': u'branch'}]
+    """
+    import fnmatch
+    from .helpers import extra_fields, extract_pre_files
+    from .utils import str2bool, aslist
+    from rhodecode.lib.helpers import format_byte_size_binary
+
+    # returns list of dicts with key-val fetched from extra fields
+    repo_extra_fields = extra_fields.run(**kwargs)
+
+    # optionally use 'extra fields' to control the logic per repo
+    # e.g store a list of patterns to be forbidden e.g `*.exe, *.dump`
+    forbid_files = repo_extra_fields.get('forbid_files_glob', {}).get('field_value')
+    forbid_files = aslist(forbid_files)
+
+    # optionally get bytes limit for a single file, e.g 1024 for 1KB
+    forbid_size_over = repo_extra_fields.get('forbid_size_over', {}).get('field_value')
+    forbid_size_over = int(forbid_size_over or 0)
+
+    def validate_file_name_and_size(file_data, forbidden_files=None, size_limit=None):
+        """
+        This function validates commited files against some sort of rules.
+        It should return a valid boolean, and a reason for failure
+
+        file_data =[
+            'raw_diff', 'old_revision', 'stats', 'original_filename', 'is_limited_diff',
+            'chunks', 'new_revision', 'operation', 'exceeds_limit', 'filename'
+            ]
+        file_data['ops'] = {
+        # is file binary
+        'binary': False,
+
+        # lines
+        'added': 32,
+        'deleted': 0
+
+        'ops': {3: 'modified file'},
+        'new_mode': '100644',
+        'old_mode': None
+        }
+        """
+        file_name = file_data['filename']
+        operation = file_data['operation']  # can be A(dded), M(odified), D(eleted)
+
+        # check files names
+        if forbidden_files:
+            reason = 'File {} is forbidden to be pushed'.format(file_name)
+            for forbidden_pattern in forbid_files:
+                # here we can also filter for operation, e.g if check for only ADDED files
+                # if operation == 'A':
+                if fnmatch.fnmatch(file_name, forbidden_pattern):
+                    return False, reason
+
+        # validate A(dded) files and size
+        if size_limit and operation == 'A':
+            size = len(file_data['raw_diff'])
+
+            reason = 'File {} size of {} bytes exceeds limit {}'.format(
+                file_name, format_byte_size_binary(size),
+                format_byte_size_binary(size_limit))
+            if size > size_limit:
+                return False, reason
+
+        return True, ''
+
+    if forbid_files or forbid_size_over:
+        # returns list of dicts with key-val fetched from extra fields
+        file_list = extract_pre_files.run(**kwargs)
+
+        for file_data in file_list:
+            file_valid, reason = validate_file_name_and_size(
+                file_data, forbid_files, forbid_size_over)
+            if not file_valid:
+                return HookResponse(1, reason)
+
+    return HookResponse(0, '')

rhodecode/config/rcextensions/helpers/extract_pre_files.py

@@ -0,0 +1,96 b''
+# -*- coding: utf-8 -*-
+# Copyright (C) 2016-2019 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+"""
+us in hooks::
+
+    from .helpers import extract_pre_files
+    # returns list of dicts with key-val fetched from extra fields
+    file_list = extract_pre_files.run(**kwargs)
+
+"""
+import re
+import collections
+import json
+
+from rhodecode.lib import diffs
+from rhodecode.lib.vcs.backends.hg.diff import MercurialDiff
+from rhodecode.lib.vcs.backends.git.diff import GitDiff
+
+
+def get_hg_files(repo, refs):
+    files = []
+    return files
+
+
+def get_git_files(repo, refs):
+    files = []
+
+    for data in refs:
+        # we should now extract commit data
+        old_rev = data['old_rev']
+        new_rev = data['new_rev']
+
+        if '00000000' in old_rev:
+            # new branch, we don't need to extract nothing
+            return files
+
+        git_env = dict(data['git_env'])
+
+        cmd = [
+            'diff', old_rev, new_rev
+        ]
+
+        stdout, stderr = repo.run_git_command(cmd, extra_env=git_env)
+        vcs_diff = GitDiff(stdout)
+
+        diff_processor = diffs.DiffProcessor(vcs_diff, format='newdiff')
+        # this is list of dicts with diff information
+        # _parsed[0].keys()
+        # ['raw_diff', 'old_revision', 'stats', 'original_filename',
+        #  'is_limited_diff', 'chunks', 'new_revision', 'operation',
+        #  'exceeds_limit', 'filename']
+        files = _parsed = diff_processor.prepare()
+
+    return files
+
+
+def run(*args, **kwargs):
+    from rhodecode.model.db import Repository
+
+    vcs_type = kwargs['scm']
+    # use temp name then the main one propagated
+    repo_name = kwargs.pop('REPOSITORY', None) or kwargs['repository']
+
+    repo = Repository.get_by_repo_name(repo_name)
+    vcs_repo = repo.scm_instance(cache=False)
+
+    files = []
+
+    if vcs_type == 'git':
+        for rev_data in kwargs['commit_ids']:
+            new_environ = dict((k, v) for k, v in rev_data['git_env'])
+        files = get_git_files(vcs_repo, kwargs['commit_ids'])
+
+    if vcs_type == 'hg':
+        for rev_data in kwargs['commit_ids']:
+            new_environ = dict((k, v) for k, v in rev_data['hg_env'])
+        files = get_hg_files(vcs_repo, kwargs['commit_ids'])
+
+    return files

.bumpversion.cfg

@@ -1,5 +1,5 b''
 [bumpversion]
-current_version = 4.15.2
+current_version = 4.16.0
 message = release: Bump version {current_version} to {new_version}
 
 [bumpversion:file:rhodecode/VERSION]

.release.cfg

@@ -5,25 +5,20 b' done = false'
 done = true
 
 [task:rc_tools_pinned]
-done = true
 
 [task:fixes_on_stable]
-done = true
 
 [task:pip2nix_generated]
-done = true
 
 [task:changelog_updated]
-done = true
 
 [task:generate_api_docs]
-done = true
+
+[task:updated_translation]
 
 [release]
-state = prepared
-version = 4.15.2
-
-[task:updated_translation]
+state = in_progress
+version = 4.16.0
 
 [task:generate_js_routes]
 

LICENSE.txt

@@ -12,8 +12,6 b' permission notice:'
        file:licenses/msgpack_license.txt
    Copyright (c) 2009 - tornado
        file:licenses/tornado_license.txt
-   Copyright (c) 2015 - pygments-markdown-lexer
-       file:licenses/pygments_markdown_lexer_license.txt
    Copyright 2006 - diff_match_patch
        file:licenses/diff_match_patch_license.txt
 

Makefile

@@ -1,5 +1,5 b''
 
-.PHONY: clean docs docs-clean docs-cleanup test test-clean test-only test-only-postgres test-only-mysql web-build
+.PHONY: clean docs docs-clean docs-cleanup test test-clean test-only test-only-postgres test-only-mysql web-build generate-pkgs
 
 NODE_PATH=./node_modules
 WEBPACK=./node_binaries/webpack
@@ -8,7 +8,7 b' GRUNT=./node_binaries/grunt'
 
 clean:
 	make test-clean
-	find . -type f \( -iname '*.c' -o -iname '*.pyc' -o -iname '*.so' \) -exec rm '{}' ';'
+	find . -type f \( -iname '*.c' -o -iname '*.pyc' -o -iname '*.so' -o -iname '*.orig' \) -exec rm '{}' ';'
 
 test:
 	make test-clean
@@ -51,3 +51,5 b' docs-cleanup:'
 web-build:
 	NODE_PATH=$(NODE_PATH) $(GRUNT)
 
+generate-pkgs:
+	nix-shell pkgs/shell-generate.nix --command "pip2nix generate --licenses"

configs/development.ini

@@ -53,7 +53,7 b' asyncore_use_poll = true'
 ## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 #use = egg:gunicorn#main
-## Sets the number of process workers. More workers means more concurent connections
+## Sets the number of process workers. More workers means more concurrent connections
 ## RhodeCode can handle at the same time. Each additional worker also it increases
 ## memory usage as each has it's own set of caches.
 ## Recommended value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers, but no more
@@ -133,10 +133,10 b' rhodecode.api.url = /_admin/api'
 ## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
-## return gzipped responses from Rhodecode (static files/application)
+## return gzipped responses from RhodeCode (static files/application)
 gzip_responses = false
 
-## autogenerate javascript routes file on startup
+## auto-generate javascript routes file on startup
 generate_js_files = false
 
 ## System global default language.
@@ -153,7 +153,7 b' startup.import_repos = false'
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
-## URL at which the application is running. This is used for bootstraping
+## URL at which the application is running. This is used for Bootstrapping
 ## requests in context when no web request is available. Used in ishell, or
 ## SSH calls. Set this for events to receive proper url for SSH calls.
 app.base_url = http://rhodecode.local
@@ -203,7 +203,7 b' gist_alias_url ='
 ## used for access.
 ## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
-## Additionally @TOKEN syntaxt can be used to bound the view to specific
+## Additionally @TOKEN syntax can be used to bound the view to specific
 ## authentication token. Such view would be only accessible when used together
 ## with this authentication token
 ##
@@ -227,14 +227,14 b' default_encoding = UTF-8'
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
-## multiple instances of rhodecode, make sure it's globally unique for
-## all running rhodecode instances. Leave empty if you don't use it
+## multiple instances of RhodeCode, make sure it's globally unique for
+## all running RhodeCode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
-## authentication settings. Then you can enable e.g. the internal rhodecode auth
+## authentication settings. Then you can enable e.g. the internal RhodeCode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
@@ -250,7 +250,7 b' instance_id ='
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
-## This will be served instead of default 401 on bad authnetication
+## This will be served instead of default 401 on bad authentication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
@@ -284,6 +284,13 b' labs_settings_active = true'
 ## This is used to store exception from RhodeCode in shared directory
 #exception_tracker.store_path =
 
+## File store configuration. This is used to store and serve uploaded files
+file_store.enabled = true
+## Storage backend, available options are: local
+file_store.backend = local
+## path to store the uploaded binaries
+file_store.storage_path = %(here)s/data/file_store
+
 
 ####################################
 ###        CELERY CONFIG        ####
@@ -325,6 +332,7 b' rc_cache.cache_perms.expiration_time = 3'
 #rc_cache.cache_perms.arguments.host = localhost
 #rc_cache.cache_perms.arguments.port = 6379
 #rc_cache.cache_perms.arguments.db = 0
+## more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_perms.arguments.distributed_lock = true
 
 ## `cache_repo` cache settings for FileTree, Readme, RSS FEEDS
@@ -340,6 +348,7 b' rc_cache.cache_repo.expiration_time = 25'
 #rc_cache.cache_repo.arguments.host = localhost
 #rc_cache.cache_repo.arguments.port = 6379
 #rc_cache.cache_repo.arguments.db = 1
+## more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_repo.arguments.distributed_lock = true
 
 ## cache settings for SQL queries, this needs to use memory type backend
@@ -424,7 +433,7 b' channelstream.server = 127.0.0.1:9800'
 ## location of the channelstream server from outside world
 ## use ws:// for http or wss:// for https. This address needs to be handled
 ## by external HTTP server such as Nginx or Apache
-## see nginx/apache configuration examples in our docs
+## see Nginx/Apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = secret
 channelstream.history.location = %(here)s/channelstream_history
@@ -441,14 +450,14 b' channelstream.proxy_path = /_channelstre'
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
-## appenlight integration enabled
+## Appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
-# used for JS client
+## used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
@@ -473,7 +482,7 b' appenlight.logging.level = WARNING'
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
-## list of additonal keywords that should be grabbed from environ object
+## list of additional keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
@@ -533,7 +542,7 b' sqlalchemy.db1.convert_unicode = true'
 vcs.server.enable = true
 vcs.server = localhost:9900
 
-## Web server connectivity protocol, responsible for web based VCS operatations
+## Web server connectivity protocol, responsible for web based VCS operations
 ## Available protocols are:
 ## `http` - use http-rpc backend (default)
 vcs.server.protocol = http
@@ -582,7 +591,8 b' svn.proxy.config_file_path = %(here)s/mo'
 ## In most cases it should be set to `/`.
 svn.proxy.location_root = /
 ## Command to reload the mod dav svn configuration on change.
-## Example: `/etc/init.d/apache2 reload`
+## Example: `/etc/init.d/apache2 reload` or /home/USER/apache_reload.sh
+## Make sure user who runs RhodeCode process is allowed to reload Apache
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 ## If the timeout expires before the reload command finishes, the command will
 ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
@@ -593,7 +603,7 b' svn.proxy.location_root = /'
 ############################################################
 
 ## Defines if a custom authorized_keys file should be created and written on
-## any change user ssh keys. Setting this to false also disables posibility
+## any change user ssh keys. Setting this to false also disables possibility
 ## of adding SSH keys by users from web interface. Super admins can still
 ## manage SSH Keys.
 ssh.generate_authorized_keyfile = false
@@ -601,13 +611,13 b' ssh.generate_authorized_keyfile = false'
 ## Options for ssh, default is `no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding`
 # ssh.authorized_keys_ssh_opts =
 
-## Path to the authrozied_keys file where the generate entries are placed.
+## Path to the authorized_keys file where the generate entries are placed.
 ## It is possible to have multiple key files specified in `sshd_config` e.g.
 ## AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/authorized_keys_rhodecode
 ssh.authorized_keys_file_path = ~/.ssh/authorized_keys_rhodecode
 
 ## Command to execute the SSH wrapper. The binary is available in the
-## rhodecode installation directory.
+## RhodeCode installation directory.
 ## e.g ~/.rccontrol/community-1/profile/bin/rc-ssh-wrapper
 ssh.wrapper_cmd = ~/.rccontrol/community-1/rc-ssh-wrapper
 
@@ -615,7 +625,7 b' ssh.wrapper_cmd = ~/.rccontrol/community'
 ssh.wrapper_cmd_allow_shell = false
 
 ## Enables logging, and detailed output send back to the client during SSH
-## operations. Usefull for debugging, shouldn't be used in production.
+## operations. Useful for debugging, shouldn't be used in production.
 ssh.enable_debug_logging = true
 
 ## Paths to binary executable, by default they are the names, but we can
@@ -624,6 +634,10 b' ssh.executable.hg = ~/.rccontrol/vcsserv'
 ssh.executable.git = ~/.rccontrol/vcsserver-1/profile/bin/git
 ssh.executable.svn = ~/.rccontrol/vcsserver-1/profile/bin/svnserve
 
+## Enables SSH key generator web interface. Disabling this still allows users
+## to add their own keys.
+ssh.enable_ui_key_generator = true
+
 
 ## Dummy marker to add new entries after.
 ## Add any custom entries below. Please don't remove.

configs/production.ini

@@ -53,7 +53,7 b' port = 5000'
 ## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 use = egg:gunicorn#main
-## Sets the number of process workers. More workers means more concurent connections
+## Sets the number of process workers. More workers means more concurrent connections
 ## RhodeCode can handle at the same time. Each additional worker also it increases
 ## memory usage as each has it's own set of caches.
 ## Recommended value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers, but no more
@@ -108,10 +108,10 b' use = egg:rhodecode-enterprise-ce'
 ## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
-## return gzipped responses from Rhodecode (static files/application)
+## return gzipped responses from RhodeCode (static files/application)
 gzip_responses = false
 
-## autogenerate javascript routes file on startup
+## auto-generate javascript routes file on startup
 generate_js_files = false
 
 ## System global default language.
@@ -128,7 +128,7 b' startup.import_repos = false'
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
-## URL at which the application is running. This is used for bootstraping
+## URL at which the application is running. This is used for Bootstrapping
 ## requests in context when no web request is available. Used in ishell, or
 ## SSH calls. Set this for events to receive proper url for SSH calls.
 app.base_url = http://rhodecode.local
@@ -178,7 +178,7 b' gist_alias_url ='
 ## used for access.
 ## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
-## Additionally @TOKEN syntaxt can be used to bound the view to specific
+## Additionally @TOKEN syntax can be used to bound the view to specific
 ## authentication token. Such view would be only accessible when used together
 ## with this authentication token
 ##
@@ -202,14 +202,14 b' default_encoding = UTF-8'
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
-## multiple instances of rhodecode, make sure it's globally unique for
-## all running rhodecode instances. Leave empty if you don't use it
+## multiple instances of RhodeCode, make sure it's globally unique for
+## all running RhodeCode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
-## authentication settings. Then you can enable e.g. the internal rhodecode auth
+## authentication settings. Then you can enable e.g. the internal RhodeCode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
@@ -225,7 +225,7 b' instance_id ='
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
-## This will be served instead of default 401 on bad authnetication
+## This will be served instead of default 401 on bad authentication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
@@ -259,6 +259,13 b' labs_settings_active = true'
 ## This is used to store exception from RhodeCode in shared directory
 #exception_tracker.store_path =
 
+## File store configuration. This is used to store and serve uploaded files
+file_store.enabled = true
+## Storage backend, available options are: local
+file_store.backend = local
+## path to store the uploaded binaries
+file_store.storage_path = %(here)s/data/file_store
+
 
 ####################################
 ###        CELERY CONFIG        ####
@@ -300,6 +307,7 b' rc_cache.cache_perms.expiration_time = 3'
 #rc_cache.cache_perms.arguments.host = localhost
 #rc_cache.cache_perms.arguments.port = 6379
 #rc_cache.cache_perms.arguments.db = 0
+## more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_perms.arguments.distributed_lock = true
 
 ## `cache_repo` cache settings for FileTree, Readme, RSS FEEDS
@@ -315,6 +323,7 b' rc_cache.cache_repo.expiration_time = 25'
 #rc_cache.cache_repo.arguments.host = localhost
 #rc_cache.cache_repo.arguments.port = 6379
 #rc_cache.cache_repo.arguments.db = 1
+## more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_repo.arguments.distributed_lock = true
 
 ## cache settings for SQL queries, this needs to use memory type backend
@@ -399,7 +408,7 b' channelstream.server = 127.0.0.1:9800'
 ## location of the channelstream server from outside world
 ## use ws:// for http or wss:// for https. This address needs to be handled
 ## by external HTTP server such as Nginx or Apache
-## see nginx/apache configuration examples in our docs
+## see Nginx/Apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = secret
 channelstream.history.location = %(here)s/channelstream_history
@@ -416,14 +425,14 b' channelstream.proxy_path = /_channelstre'
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
-## appenlight integration enabled
+## Appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
-# used for JS client
+## used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
@@ -448,7 +457,7 b' appenlight.logging.level = WARNING'
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
-## list of additonal keywords that should be grabbed from environ object
+## list of additional keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
@@ -506,7 +515,7 b' sqlalchemy.db1.convert_unicode = true'
 vcs.server.enable = true
 vcs.server = localhost:9900
 
-## Web server connectivity protocol, responsible for web based VCS operatations
+## Web server connectivity protocol, responsible for web based VCS operations
 ## Available protocols are:
 ## `http` - use http-rpc backend (default)
 vcs.server.protocol = http
@@ -555,7 +564,8 b' svn.proxy.config_file_path = %(here)s/mo'
 ## In most cases it should be set to `/`.
 svn.proxy.location_root = /
 ## Command to reload the mod dav svn configuration on change.
-## Example: `/etc/init.d/apache2 reload`
+## Example: `/etc/init.d/apache2 reload` or /home/USER/apache_reload.sh
+## Make sure user who runs RhodeCode process is allowed to reload Apache
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 ## If the timeout expires before the reload command finishes, the command will
 ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
@@ -566,7 +576,7 b' svn.proxy.location_root = /'
 ############################################################
 
 ## Defines if a custom authorized_keys file should be created and written on
-## any change user ssh keys. Setting this to false also disables posibility
+## any change user ssh keys. Setting this to false also disables possibility
 ## of adding SSH keys by users from web interface. Super admins can still
 ## manage SSH Keys.
 ssh.generate_authorized_keyfile = false
@@ -574,13 +584,13 b' ssh.generate_authorized_keyfile = false'
 ## Options for ssh, default is `no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding`
 # ssh.authorized_keys_ssh_opts =
 
-## Path to the authrozied_keys file where the generate entries are placed.
+## Path to the authorized_keys file where the generate entries are placed.
 ## It is possible to have multiple key files specified in `sshd_config` e.g.
 ## AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/authorized_keys_rhodecode
 ssh.authorized_keys_file_path = ~/.ssh/authorized_keys_rhodecode
 
 ## Command to execute the SSH wrapper. The binary is available in the
-## rhodecode installation directory.
+## RhodeCode installation directory.
 ## e.g ~/.rccontrol/community-1/profile/bin/rc-ssh-wrapper
 ssh.wrapper_cmd = ~/.rccontrol/community-1/rc-ssh-wrapper
 
@@ -588,7 +598,7 b' ssh.wrapper_cmd = ~/.rccontrol/community'
 ssh.wrapper_cmd_allow_shell = false
 
 ## Enables logging, and detailed output send back to the client during SSH
-## operations. Usefull for debugging, shouldn't be used in production.
+## operations. Useful for debugging, shouldn't be used in production.
 ssh.enable_debug_logging = false
 
 ## Paths to binary executable, by default they are the names, but we can
@@ -597,6 +607,10 b' ssh.executable.hg = ~/.rccontrol/vcsserv'
 ssh.executable.git = ~/.rccontrol/vcsserver-1/profile/bin/git
 ssh.executable.svn = ~/.rccontrol/vcsserver-1/profile/bin/svnserve
 
+## Enables SSH key generator web interface. Disabling this still allows users
+## to add their own keys.
+ssh.enable_ui_key_generator = true
+
 
 ## Dummy marker to add new entries after.
 ## Add any custom entries below. Please don't remove.

default.nix

@@ -19,14 +19,15 b''
 #  }
 
 args@
-{ pythonPackages ? "python27Packages"
+{ system ? builtins.currentSystem
+, pythonPackages ? "python27Packages"
 , pythonExternalOverrides ? self: super: {}
 , doCheck ? false
 , ...
 }:
 
 let
-  pkgs_ = (import <nixpkgs> {});
+  pkgs_ = args.pkgs or (import <nixpkgs> { inherit system; });
 in
 
 let
@@ -80,7 +81,8 b' let'
 
   nodeEnv = import ./pkgs/node-default.nix {
     inherit
-      pkgs;
+      pkgs
+      system;
   };
   nodeDependencies = nodeEnv.shell.nodeDependencies;
 

docs/admin/backup-restore.rst

@@ -116,7 +116,7 b' Full-text Search Backup'
 
 You may also have full text search set up, but the index can be rebuild from
 re-imported |repos| if necessary. You will most likely want to backup your
-:file:`mapping.ini` file if you've configured that. For more information, see
+:file:`search_mapping.ini` file if you've configured that. For more information, see
 the :ref:`indexing-ref` section.
 
 Restoration Steps
@@ -140,7 +140,7 b' Post Restoration Steps'
 Once you have restored your |RCE| instance to basic functionality, you can
 then work on restoring any specific setup changes you had made.
 
-* To recreate the |RCE| index, use the backed up :file:`mapping.ini` file if
+* To recreate the |RCE| index, use the backed up :file:`search_mapping.ini` file if
   you had made changes and rerun the indexer. See the
   :ref:`indexing-ref` section for details.
 * To reconfigure any extensions, copy the backed up extensions into the

docs/admin/config-files-overview.rst

@@ -23,9 +23,9 b' sections.'
         * :ref:`increase-gunicorn`
         * :ref:`x-frame`
 
-    \- **mapping.ini**
+    \- **search_mapping.ini**
         Default location:
-        :file:`/home/{user}/.rccontrol/{instance-id}/mapping.ini`
+        :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`
 
         This file is used to control the |RCE| indexer. It comes configured
         to index your instance. To change the default configuration, see

docs/admin/indexing.rst

@@ -3,35 +3,41 b''
 Full-text Search
 ----------------
 
-By default RhodeCode is configured to use `Whoosh`_ to index |repos| and
-provide full-text search.
+RhodeCode provides a full text search capabilities to search inside file content,
+commit message, and file paths. Indexing is not enabled by default and to use
+full text search building an index is a pre-requisite.
 
-|RCE| also provides support for `Elasticsearch`_ as a backend for scalable
-search. See :ref:`enable-elasticsearch` for details.
+By default RhodeCode is configured to use `Whoosh`_ to index |repos| and
+provide full-text search. `Whoosh`_ works well for a small amount of data and
+shouldn't be used in case of large code-bases and lots of repositories.
+
+|RCE| also provides support for `ElasticSearch 6`_ as a backend more for advanced
+and scalable search. See :ref:`enable-elasticsearch` for details.
 
 Indexing
 ^^^^^^^^
 
-To run the indexer you need to use an |authtoken| with admin rights to all
-|repos|.
+To run the indexer you need to have an |authtoken| with admin rights to all |repos|.
 
-To index new content added, you have the option to set the indexer up in a
+To index repositories stored in RhodeCode, you have the option to set the indexer up in a
 number of ways, for example:
 
-* Call the indexer via a cron job. We recommend running this nightly,
-  unless you need everything indexed immediately.
-* Set the indexer to infinitely loop and reindex as soon as it has run its
-  cycle.
+* Call the indexer via a cron job. We recommend running this once at night.
+  In case you need everything indexed immediately it's possible to index few
+  times during the day. Indexer has a special locking mechanism that won't allow
+  two instances of indexer running at once. It's safe to run it even every 1hr.
+* Set the indexer to infinitely loop and reindex as soon as it has run its previous cycle.
 * Hook the indexer up with your CI server to reindex after each push.
 
-The indexer works by indexing new commits added since the last run. If you
-wish to build a brand new index from scratch each time,
-use the ``force`` option in the configuration file.
+The indexer works by indexing new commits added since the last run, and comparing
+file changes to index only new or modified files.
+If you wish to build a brand new index from scratch each time, use the ``force``
+option in the configuration file, or run it with --force flag.
 
 .. important::
 
    You need to have |RCT| installed, see :ref:`install-tools`. Since |RCE|
-   3.5.0 they are installed by default.
+   3.5.0 they are installed by default and available with community/enterprise installations.
 
 To set up indexing, use the following steps:
 
@@ -45,6 +51,13 b' 4. :ref:`advanced-indexing`'
 Configure the ``.rhoderc`` File
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+.. note::
+
+    Optionally it's possible to use indexer without the ``.rhoderc``. Simply instead of
+    executing with `--instance-name=enterprise-1` execute providing the host and token
+    directly: `--api-host=http://127.0.0.1:10000 --api-key=<auth-token-goes-here>`
+
+
 |RCT| uses the :file:`/home/{user}/.rhoderc` file for connection details
 to |RCE| instances. If this file is not automatically created,
 you can configure it using the following example. You need to configure the
@@ -58,9 +71,9 b' details for each instance you want to in'
 
      - NAME: enterprise-1
      - STATUS: RUNNING
-     - TYPE: Momentum
-     - VERSION: 1.5.0
-     - URL: http://127.0.0.1:10000
+    - TYPE: Enterprise
+    - VERSION: 4.1.0
+    - URL: http://127.0.0.1:10003
 
 To get your API Token, on the |RCE| interface go to
 :menuselection:`username --> My Account --> Auth tokens`
@@ -72,29 +85,35 b' To get your API Token, on the |RCE| inte'
     [instance:enterprise-1]
     api_host = http://127.0.0.1:10000
     api_key = <auth token goes here>
-    repo_dir = /home/<username>/repos
+
 
 .. _run-index:
 
 Run the Indexer
 ^^^^^^^^^^^^^^^
 
-Run the indexer using the following command, and specify the instance you
-want to index:
+Run the indexer using the following command, and specify the instance you want to index:
 
 .. code-block:: bash
 
-   # From inside a virtualevv
-   (venv)$ rhodecode-index --instance-name=enterprise-1
-
-   # Using default installation
+   # Using default simples indexing of all repositories
    $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
        --instance-name=enterprise-1
 
-   # Using a custom mapping file
+   # Using a custom mapping file with indexing rules, and using elasticsearch 6 backend
    $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
        --instance-name=enterprise-1 \
-       --mapping=/home/user/.rccontrol/enterprise-1/mapping.ini
+       --mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini \
+       --es-version=6 --engine-location=http://elasticsearch-host:9200
+
+   # Using a custom mapping file and invocation without ``.rhoderc``
+   $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
+       --api-host=http://rhodecodecode.myserver.com --api-key=xxxxx \
+       --mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini
+
+   # From inside a virtualev on your local machine or CI server.
+   (venv)$ rhodecode-index --instance-name=enterprise-1
+
 
 .. note::
 
@@ -136,119 +155,173 b' 3. Save the file.'
     # using a specially configured mapping file
     */15 * * * * ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
        --instance-name=enterprise-4 \
-       --mapping=/home/user/.rccontrol/enterprise-4/mapping.ini
+       --mapping=/home/user/.rccontrol/enterprise-4/search_mapping.ini
 
 .. _advanced-indexing:
 
 Advanced Indexing
 ^^^^^^^^^^^^^^^^^
 
-|RCT| indexes based on the :file:`mapping.ini` file. To configure your index,
-you can specify different options in this file. The default location is:
+
+Force Re-Indexing single repository
++++++++++++++++++++++++++++++++++++
+
+Often it's required to re-index whole repository because of some repository changes,
+or to remove some indexed secrets, or files. There's a special `--repo-name=` flag
+for the indexer that limits execution to a single repository. For example to force-reindex
+single repository such call can be made::
+
+    rhodecode-index --instance-name=enterprise-1 --force --repo-name=rhodecode-vcsserver
+
+
+Removing repositories from index
+++++++++++++++++++++++++++++++++
 
-* :file:`/home/{user}/.rccontrol/{instance-id}/mapping.ini`, using default
-  |RCT|.
+The indexer automatically removes renamed repositories and builds index for new names.
+In the same way if a listed repository in mapping.ini is not reported existing by the
+server it's removed from the index.
+In case that you wish to remove indexed repository manually such call would allow that::
+
+    rhodecode-index --instance-name=enterprise-1 --remove-only --repo-name=rhodecode-vcsserver
+
+
+Using search_mapping.ini file for advanced index rules
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+By default rhodecode-index runs for all repositories, all files with parsing limits
+defined by the CLI default arguments. You can change those limits by calling with
+different flags such as `--max-filesize=2048kb` or `--repo-limit=10`
+
+For more advanced execution logic it's possible to use a configuration file that
+would define detailed rules which repositories and how should be indexed.
+
+|RCT| provides an example index configuration file called :file:`search_mapping.ini`.
+This file is created by default during installation and is located at:
+
+* :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`, using default |RCT|.
 * :file:`~/venv/lib/python2.7/site-packages/rhodecode_tools/templates/mapping.ini`,
   when using ``virtualenv``.
 
 .. note::
 
-    If you need to create the :file:`mapping.ini` file, use the |RCT|
-    ``rhodecode-index --create-mapping path/to/file`` API call. For details,
-    see the :ref:`tools-cli` section.
-
-The indexer runs in a random order to prevent a failing |repo| from stopping
-a build. To configure different indexing scenarios, set the following options
-inside the :file:`mapping.ini` and specify the altered file using the
-``--mapping`` option.
+    If you need to create the :file:`search_mapping.ini` file manually, use the |RCT|
+    ``rhodecode-index --create-mapping path/to/search_mapping.ini`` API call.
+    For details, see the :ref:`tools-cli` section.
 
-* ``index_files`` : Index the specified file types.
-* ``skip_files`` : Do not index the specified file types.
-* ``index_files_content`` : Index the content of the specified file types.
-* ``skip_files_content`` : Do not index the content of the specified files.
-* ``force`` : Create a fresh index on each run.
-* ``max_filesize`` : Files larger than the set size will not be indexed.
-* ``commit_parse_limit`` : Set the batch size when indexing commit messages.
-  Set to a lower number to lessen memory load.
-* ``repo_limit`` : Set the maximum number or |repos| indexed per run.
-* ``[INCLUDE]`` : Set |repos| you want indexed. This takes precedent over
-  ``[EXCLUDE]``.
-* ``[EXCLUDE]`` : Set |repos| you do not want indexed. Exclude can be used to
-  not index branches, forks, or log |repos|.
+To Run the indexer with mapping file provide it using `--mapping` flag::
 
-At the end of the file you can specify conditions for specific |repos| that
-will override the default values. To configure your indexer,
-use the following example :file:`mapping.ini` file.
+    rhodecode-index --instance-name=enterprise-1 --mapping=/my/path/search_mapping.ini
+
+
+Here's a detailed example of using :file:`search_mapping.ini` file.
 
 .. code-block:: ini
 
     [__DEFAULT__]
-    # default patterns for indexing files and content of files.
-    # Binary files are skipped by default.
+    ; Create index on commits data, and files data in this order. Available options
+    ; are `commits`, `files`
+    index_types = commits,files
+
+    ; Commit fetch limit. In what amount of chunks commits should be fetched
+    ; via api and parsed. This allows server to transfer smaller chunks and be less loaded
+    commit_fetch_limit = 1000
 
-    # Index python and markdown files
-    index_files = *.py, *.md
+    ; Commit process limit. Limit the number of commits indexer should fetch, and
+    ; store inside the full text search index. eg. if repo has 2000 commits, and
+    ; limit is 1000, on the first run it will process commits 0-1000 and on the
+    ; second 1000-2000 commits. Help reduce memory usage, default is 50000
+    ; (set -1 for unlimited)
+    commit_process_limit = 20000
 
-    # Do not index these file types
-    skip_files = *.svg, *.log, *.dump, *.txt
+    ; Limit of how many repositories each run can process, default is -1 (unlimited)
+    ; in case of 1000s of repositories it's better to execute in chunks to not overload
+    ; the server.
+    repo_limit = -1
 
-    # Index both file types and their content
-    index_files_content = *.cpp, *.ini, *.py
+    ; Default patterns for indexing files and content of files. Binary files
+    ; are skipped by default.
+
+    ; Add to index those comma separated files; globs syntax
+    ; e.g index_files = *.py, *.c, *.h, *.js
+    index_files = *,
+
+    ; Do not add to index those comma separated files, this excludes
+    ; both search by name and content; globs syntax
+    ; e.g index_files = *.key, *.sql, *.xml, *.pem, *.crt
+    skip_files = ,
 
-    # Index file names, but not file content
-    skip_files_content = *.svg,
+    ; Add to index content of those comma separated files; globs syntax
+    ; e.g index_files = *.h, *.obj
+    index_files_content = *,
 
-    # Force rebuilding an index from scratch. Each repository will be rebuild
-    # from scratch with a global flag. Use local flag to rebuild single repos
+    ; Do not add to index content of those comma separated files; globs syntax
+    ; Binary files are not indexed by default.
+    ; e.g index_files = *.min.js, *.xml, *.dump, *.log, *.dump
+    skip_files_content = ,
+
+    ; Force rebuilding an index from scratch. Each repository will be rebuild from
+    ; scratch with a global flag. Use --repo-name=NAME --force to rebuild single repo
     force = false
 
-    # Do not index files larger than 385KB
-    max_filesize = 385KB
+    ; maximum file size that indexer will use, files above that limit are not going
+    ; to have they content indexed.
+    ; Possible options are KB (kilobytes), MB (megabytes), eg 1MB or 1024KB
+    max_filesize = 10MB
 
-    # Limit commit indexing to 500 per batch
-    commit_parse_limit = 500
-
-    # Limit each index run to 25 repos
-    repo_limit = 25
 
-    # __INCLUDE__ is more important that __EXCLUDE__.
-
-    [__INCLUDE__]
-    # Include all repos with these names
+    [__INDEX_RULES__]
+    ; Ordered match rules for repositories. A list of all repositories will be fetched
+    ; using API and this list will be filtered using those rules.
+    ; Syntax for entry: `glob_pattern_OR_full_repo_name = 0 OR 1` where 0=exclude, 1=include
+    ; When this ordered list is traversed first match will return the include/exclude marker
+    ; For example:
+    ;    upstream/binary_repo = 0
+    ;    upstream/subrepo/xml_files = 0
+    ;    upstream/* = 1
+    ;    special-repo = 1
+    ;    * = 0
+    ; This will index all repositories under upstream/*, but skip upstream/binary_repo
+    ; and upstream/sub_repo/xml_files, last * = 0 means skip all other matches
 
-    docs/* = 1
-    lib/* = 1
-
-    [__EXCLUDE__]
-    # Do not include the following repo in index
 
-    dev-docs/* = 1
-    legacy-repos/* = 1
-    *-dev/* = 1
-
-    # Each repo that needs special indexing is a separate section below.
-    # In each section set the options to override the global configuration
-    # parameters above.
-    # If special settings are not configured, the global configuration values
-    # above are inherited. If no special repositories are
-    # defined here RhodeCode will use the API to ask for all repositories
+    ; == EXPLICIT REPOSITORY INDEXING ==
+    ; If defined this will skip using __INDEX_RULES__, and will not use API to fetch
+    ; list of repositories, it will explicitly take names defined with [NAME] format and
+    ; try to build the index, to build index just for repo_name_1 and special-repo use:
+    ;    [repo_name_1]
+    ;    [special-repo]
 
-    # For this repo use different settings
-    [special-repo]
-    commit_parse_limit = 20,
-    skip_files = *.idea, *.xml,
+    ; == PER REPOSITORY CONFIGURATION ==
+    ; This allows overriding the global configuration per repository.
+    ; example to set specific file limit, and skip certain files for repository special-repo
+    ; the CLI flags doesn't override the conf settings.
+    ;    [conf:special-repo]
+    ;    max_filesize = 5mb
+    ;    skip_files = *.xml, *.sql
+
 
-    # For another repo use different settings
-    [another-special-repo]
-    index_files = *,
-    max_filesize = 800MB
-    commit_parse_limit = 20000
+
+In case of 1000s of repositories it can be tricky to write the include/exclude rules at first.
+There's a special flag to test the mapping file rules and list repositories that would
+be indexed. Run the indexer with `--show-matched-repos` to list only the
+match repositories defined in .ini file rules::
+
+    rhodecode-index --instance-name=enterprise-1 --show-matched-repos --mapping=/my/path/search_mapping.ini
+
 
 .. _enable-elasticsearch:
 
-Enabling Elasticsearch
+Enabling ElasticSearch
 ^^^^^^^^^^^^^^^^^^^^^^
 
+ElasticSearch is available in EE edition only. It provides much scalable and more advanced
+search capabilities. While Whoosh is fine for upto 1-2GB of data, beyond that amount it
+starts slowing down, and can cause other problems.
+New ElasticSearch 6 also provides much more advanced query language.
+It allows advanced filtering by file paths, extensions, use OR statements, ranges etc.
+Please check query language examples in the search field for some advanced query language usage.
+
+
 1. Open the :file:`rhodecode.ini` file for the instance you wish to edit. The
    default location is
    :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
@@ -268,9 +341,19 b' and change it to:'
 .. code-block:: ini
 
     search.module = rc_elasticsearch
-    search.location = http://localhost:9200/
+    search.location = http://localhost:9200
+    ## specify Elastic Search version, 6 for latest or 2 for legacy
+    search.es_version = 6
+
+where ``search.location`` points to the ElasticSearch server
+by default running on port 9200.
 
-where ``search.location`` points to the elasticsearch server.
+Index invocation also needs change. Please provide --es-version= and
+--engine-location= parameters to define ElasticSearch server location and it's version.
+For example::
+
+    rhodecode-index --instace-name=enterprise-1 --es-version=6 --engine-location=http://localhost:9200
+
 
 .. _Whoosh: https://pypi.python.org/pypi/Whoosh/
-.. _Elasticsearch: https://www.elastic.co/
  No newline at end of file
+.. _ElasticSearch 6: https://www.elastic.co/

docs/admin/nginx-config-example.rst

@@ -9,6 +9,9 b' Use the following example to configure N'
     ## Rate limiter for certain pages to prevent brute force attacks
     limit_req_zone  $binary_remote_addr  zone=req_limit:10m   rate=1r/s;
 
+    ## cache zone
+    proxy_cache_path /etc/nginx/nginx_cache levels=1:2 use_temp_path=off keys_zone=cache_zone:10m inactive=720h max_size=10g;
+
     ## Custom log format
     log_format log_custom '$remote_addr - $remote_user [$time_local] '
                           '"$request" $status $body_bytes_sent '
@@ -141,6 +144,34 b' Use the following example to configure N'
             try_files $uri @rhodecode_http;
         }
 
+        ## Special Cache for file store, make sure you enable this intentionally as
+        ## it could bypass upload files permissions
+        # location /_file_store/download {
+        #
+        #    proxy_cache cache_zone;
+        #    # ignore Set-Cookie
+        #    proxy_ignore_headers Set-Cookie;
+        #    proxy_ignore_headers Cookie;
+        #
+        #    proxy_cache_key $host$uri$is_args$args;
+        #    proxy_cache_methods GET;
+        #
+        #    proxy_cache_bypass  $http_cache_control;
+        #    proxy_cache_valid 200 302 720h;
+        #
+        #    proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
+        #
+        #    # returns cache status in headers
+        #    add_header X-Proxy-Cache $upstream_cache_status;
+        #    add_header Cache-Control "public";
+        #
+        #    proxy_cache_lock on;
+        #    proxy_cache_lock_age 5m;
+        #
+        #    proxy_pass  http://rc;
+        #
+        # }
+
         location / {
             try_files $uri @rhodecode_http;
         }

docs/admin/repo-perm-steps.rst

@@ -35,7 +35,7 b' 2. On the |repo| group settings page you'
 
 * :guilabel:`Owner`: Lets you change the group owner. Useful when users are
   moving roles within an organisation.
-* :guilabel:`Group parent`: Lets you add the |repo| group as a sub-group
+* :guilabel:`Repository group`: Lets you add the |repo| group as a sub-group
   of a larger group, i.e. :guilabel:`QA-Repos >> QA-Repos-Berlin`
 * :guilabel:`Enable automatic locking`: For more information,
   see :ref:`repo-locking`

docs/admin/system-admin.rst

@@ -30,3 +30,5 b' The following are the most common system'
     enable-debug
     admin-tricks
     cleanup-cmds
+    restore-deleted-repositories
+

docs/admin/system-overview.rst

@@ -78,7 +78,7 b' Configuration Files'
 -------------------
 
 * :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
-* :file:`/home/{user}/.rccontrol/{instance-id}/mapping.ini`
+* :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`
 * :file:`/home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini`
 * :file:`/home/{user}/.rccontrol/supervisor/supervisord.ini`
 * :file:`/home/{user}/.rccontrol.ini`

docs/api/api.rst

@@ -196,7 +196,8 b' are not required in args.'
 .. --- API DEFS MARKER ---
 .. toctree::
 
-   methods/views
+   methods/repo-methods
+   methods/store-methods
    methods/license-methods
    methods/deprecated-methods
    methods/gist-methods

docs/api/methods/pull-request-methods.rst

@@ -83,7 +83,7 b' comment_pull_request'
 create_pull_request 
 -------------------
 
-.. py:function:: create_pull_request(apiuser, source_repo, target_repo, source_ref, target_ref, title=<Optional:''>, description=<Optional:''>, description_renderer=<Optional:''>, reviewers=<Optional:None>)
+.. py:function:: create_pull_request(apiuser, source_repo, target_repo, source_ref, target_ref, owner=<Optional:<OptionalAttr:apiuser>>, title=<Optional:''>, description=<Optional:''>, description_renderer=<Optional:''>, reviewers=<Optional:None>)
 
    Creates a new pull request.
 
@@ -104,6 +104,8 b' create_pull_request'
    :type source_ref: str
    :param target_ref: Set the target ref name.
    :type target_ref: str
+   :param owner: user_id or username
+   :type owner: Optional(str)
    :param title: Optionally Set the pull request title, it's generated otherwise
    :type title: str
    :param description: Set the pull request description.
@@ -248,7 +250,7 b' get_pull_request_comments'
 get_pull_requests 
 -----------------
 
-.. py:function:: get_pull_requests(apiuser, repoid, status=<Optional:'new'>)
+.. py:function:: get_pull_requests(apiuser, repoid, status=<Optional:'new'>, merge_state=<Optional:True>)
 
    Get all pull requests from the repository specified in `repoid`.
 
@@ -262,6 +264,9 b' get_pull_requests'
        * ``open``
        * ``closed``
    :type status: str
+   :param merge_state: Optional calculate merge state for each repository.
+       This could result in longer time to fetch the data
+   :type merge_state: bool
 
    Example output:
 
@@ -358,6 +363,7 b' merge_pull_request'
        "result": {
            "executed":         "<bool>",
            "failure_reason":   "<int>",
+           "merge_status_message":   "<str>",
            "merge_commit_id":  "<merge_commit_id>",
            "possible":         "<bool>",
            "merge_ref":        {

docs/api/methods/repo-methods.rst

@@ -394,6 +394,105 b' get_repo_changesets'
       of changed files.
 
 
+get_repo_comments 
+-----------------
+
+.. py:function:: get_repo_comments(apiuser, repoid, commit_id=<Optional:None>, comment_type=<Optional:None>, userid=<Optional:None>)
+
+   Get all comments for a repository
+
+   :param apiuser: This is filled automatically from the |authtoken|.
+   :type apiuser: AuthUser
+   :param repoid: Set the repository name or repository ID.
+   :type repoid: str or int
+   :param commit_id: Optionally filter the comments by the commit_id
+   :type commit_id: Optional(str), default: None
+   :param comment_type: Optionally filter the comments by the comment_type
+       one of: 'note', 'todo'
+   :type comment_type: Optional(str), default: None
+   :param userid: Optionally filter the comments by the author of comment
+   :type userid: Optional(str or int), Default: None
+
+   Example error output:
+
+   .. code-block:: bash
+
+       {
+           "id" : <id_given_in_input>,
+           "result" : [
+               {
+                 "comment_author": <USER_DETAILS>,
+                 "comment_created_on": "2017-02-01T14:38:16.309",
+                 "comment_f_path": "file.txt",
+                 "comment_id": 282,
+                 "comment_lineno": "n1",
+                 "comment_resolved_by": null,
+                 "comment_status": [],
+                 "comment_text": "This file needs a header",
+                 "comment_type": "todo"
+               }
+           ],
+           "error" :  null
+       }
+
+
+get_repo_file 
+-------------
+
+.. py:function:: get_repo_file(apiuser, repoid, commit_id, file_path, max_file_bytes=<Optional:None>, details=<Optional:'basic'>, cache=<Optional:True>)
+
+   Returns a single file from repository at given revision.
+
+   This command can only be run using an |authtoken| with admin rights,
+   or users with at least read rights to |repos|.
+
+   :param apiuser: This is filled automatically from the |authtoken|.
+   :type apiuser: AuthUser
+   :param repoid: The repository name or repository ID.
+   :type repoid: str or int
+   :param commit_id: The revision for which listing should be done.
+   :type commit_id: str
+   :param file_path: The path from which to start displaying.
+   :type file_path: str
+   :param details: Returns different set of information about nodes.
+       The valid options are ``minimal`` ``basic`` and ``full``.
+   :type details: Optional(str)
+   :param max_file_bytes: Only return file content under this file size bytes
+   :type max_file_bytes: Optional(int)
+   :param cache: Use internal caches for fetching files. If disabled fetching
+       files is slower but more memory efficient
+   :type cache: Optional(bool)
+   Example output:
+
+   .. code-block:: bash
+
+       id : <id_given_in_input>
+       result: {
+           "binary": false,
+           "extension": "py",
+           "lines": 35,
+           "content": "....",
+           "md5": "76318336366b0f17ee249e11b0c99c41",
+           "mimetype": "text/x-python",
+           "name": "python.py",
+           "size": 817,
+           "type": "file",
+       }
+       error:  null
+
+
+get_repo_fts_tree 
+-----------------
+
+.. py:function:: get_repo_fts_tree(apiuser, repoid, commit_id, root_path)
+
+   Returns a list of tree nodes for path at given revision. This api is built
+   strictly for usage in full text search building, and shouldn't be consumed
+
+   This command can only be run using an |authtoken| with admin rights,
+   or users with at least read rights to |repos|.
+
+
 get_repo_nodes 
 --------------
 
@@ -419,8 +518,8 b' get_repo_nodes'
        ``all`` (default), ``files`` and ``dirs``.
    :type ret_type: Optional(str)
    :param details: Returns extended information about nodes, such as
-       md5, binary, and or content.  The valid options are ``basic`` and
-       ``full``.
+           md5, binary, and or content.
+           The valid options are ``basic`` and ``full``.
    :type details: Optional(str)
    :param max_file_bytes: Only return file content under this file size bytes
    :type details: Optional(int)
@@ -432,10 +531,17 b' get_repo_nodes'
        id : <id_given_in_input>
        result: [
                  {
-                   "name" : "<name>"
-                   "type" : "<type>",
-                   "binary": "<true|false>" (only in extended mode)
-                   "md5"  : "<md5 of file content>" (only in extended mode)
+                         "binary": false,
+                         "content": "File line
+   Line2
+   ",
+                         "extension": "md",
+                         "lines": 2,
+                         "md5": "059fa5d29b19c0657e384749480f6422",
+                         "mimetype": "text/x-minidsrc",
+                         "name": "file.md",
+                         "size": 580,
+                         "type": "file"
                  },
                  ...
                ]

docs/api/methods/server-methods.rst

@@ -103,7 +103,7 b' get_method'
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param pattern: pattern to match method names against
-   :type older_then: Optional("*")
+   :type pattern: Optional("*")
 
    Example output:
 
@@ -232,3 +232,37 b' rescan_repos'
      }
 
 
+store_exception 
+---------------
+
+.. py:function:: store_exception(apiuser, exc_data_json, prefix=<Optional:'rhodecode'>)
+
+   Stores sent exception inside the built-in exception tracker in |RCE| server.
+
+   This command can only be run using an |authtoken| with admin rights to
+   the specified repository.
+
+   This command takes the following options:
+
+   :param apiuser: This is filled automatically from the |authtoken|.
+   :type apiuser: AuthUser
+
+   :param exc_data_json: JSON data with exception e.g
+       {"exc_traceback": "Value `1` is not allowed", "exc_type_name": "ValueError"}
+   :type exc_data_json: JSON data
+
+   :param prefix: prefix for error type, e.g 'rhodecode', 'vcsserver', 'rhodecode-tools'
+   :type prefix: Optional("rhodecode")
+
+   Example output:
+
+   .. code-block:: bash
+
+     id : <id_given_in_input>
+     "result": {
+       "exc_id": 139718459226384,
+       "exc_url": "http://localhost:8080/_admin/settings/exceptions/139718459226384"
+     }
+     error :  null
+
+

docs/auth/auth-saml-generic.rst

@@ -15,4 +15,5 b' Please check for reference two example p'
 
    auth-saml-duosecurity
    auth-saml-onelogin
+   auth-saml-bulk-enroll-users
 

docs/auth/ssh-connection.rst

@@ -73,6 +73,10 b' 2. Enable the SSH module on instance.'
         ssh.executable.git = ~/.rccontrol/vcsserver-1/profile/bin/git
         ssh.executable.svn = ~/.rccontrol/vcsserver-1/profile/bin/svnserve
 
+        ## Enables SSH key generator web interface. Disabling this still allows users
+        ## to add their own keys.
+        ssh.enable_ui_key_generator = true
+
 
 3. Set base_url for instance to enable proper event handling (Optional):
 

docs/contributing/dev-setup.rst

@@ -52,6 +52,7 b' run::'
 Followed by::
 
    nix-channel --update
+   nix-env -i nix-2.0.4
 
 
 Install required binaries
@@ -65,6 +66,18 b' run::'
     nix-env -i nix-prefetch-git
 
 
+Speed up JS build by installing PhantomJS
+-----------------------------------------
+
+PhantomJS will be downloaded each time nix-shell is invoked. To speed this by
+setting already downloaded version do this::
+
+    nix-env -i phantomjs-2.1.1
+
+    # and set nix bin path
+    export PATH=$PATH:~/.nix-profile/bin
+
+
 Clone the required repositories
 -------------------------------
 
@@ -76,8 +89,8 b' you have it installed before continuing.'
 To obtain the required sources, use the following commands::
 
     mkdir rhodecode-develop && cd rhodecode-develop
-    hg clone https://code.rhodecode.com/rhodecode-enterprise-ce
-    hg clone https://code.rhodecode.com/rhodecode-vcsserver
+    hg clone -u default https://code.rhodecode.com/rhodecode-enterprise-ce
+    hg clone -u default https://code.rhodecode.com/rhodecode-vcsserver
 
 .. note::
 
@@ -93,11 +106,15 b' need to install the following.'
 
 required libraries::
 
+    # svn related
     sudo apt-get install libapr1-dev libaprutil1-dev
     sudo apt-get install libsvn-dev
+    # libcurl required too
+    sudo apt-get install libcurl4-openssl-dev
+    # mysql/pg server for development, optional
     sudo apt-get install mysql-server libmysqlclient-dev
     sudo apt-get install postgresql postgresql-contrib libpq-dev
-    sudo apt-get install libcurl4-openssl-dev
+
 
 
 Enter the Development Shell
@@ -182,7 +199,7 b" To use the application's frontend and pr"
 you will need to compile the CSS and JavaScript with Grunt.
 This is easily done from within the nix-shell using the following command::
 
-    grunt
+    make web-build
 
 When developing new features you will need to recompile following any
 changes made to the CSS or JavaScript files when developing the code::

docs/integrations/integrations-rcextensions.rst

@@ -18,11 +18,11 b' Activating rcextensions'
 To activate rcextensions simply copy or rename the created template rcextensions
 into the path where the rhodecode.ini file is located::
 
-    pushd ~/rccontrol/enterprise-1/
+    pushd ~/.rccontrol/enterprise-1/
     or
-    pushd ~/rccontrol/community-1/
+    pushd ~/.rccontrol/community-1/
 
-    mv etc/rcextensions.tmpl rcextensions
+    mv profile/etc/rcextensions.tmpl rcextensions
 
 
 rcextensions are loaded when |RCE| starts. So a restart is required after activation or

docs/release-notes/release-notes-4.14.0.rst

@@ -104,7 +104,7 b' Upgrade notes'
 - In this release, we're shipping a new `rcextensions`. The changes made are
   backward incompatible. An update of `rcextensions` is required
   prior to using them again. Please check the new `rcextensions.tmpl` directory
-  located in `etc/rcextensions.tmpl` in your instance installation path.
+  located in `profile/etc/rcextensions.tmpl` in your instance installation path.
   Old code should be 100% portable by just copy&paste to the right function.
 
 - Mailing: We introduced a new mailing library. The older options should be compatible and

docs/release-notes/release-notes.rst

@@ -9,6 +9,7 b' Release Notes'
 .. toctree::
    :maxdepth: 1
 
+   release-notes-4.16.0.rst
    release-notes-4.15.2.rst
    release-notes-4.15.1.rst
    release-notes-4.15.0.rst

docs/tools/tools-cli.rst

@@ -516,7 +516,7 b' Example usage:'
     $ ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
         --instance-name=enterprise-4
 
-    # Run indexer based on mapping.ini file
+    # Run indexer based on search_mapping.ini file
     # This is using pre-350 virtualenv
     (venv)$ rhodecode-index --instance-name=enterprise-1
 
@@ -527,7 +527,7 b' Example usage:'
 
     # Create the indexing mapping file
     $ ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
-        --create-mapping mapping.ini --instance-name=enterprise-4
+        --create-mapping search_mapping.ini --instance-name=enterprise-4
 
 .. _tools-rhodecode-list-instance:
 

grunt_config.json

@@ -52,7 +52,7 b''
         "<%= dirs.js.src %>/plugins/jquery.auto-grow-input.js",
         "<%= dirs.js.src %>/plugins/jquery.autocomplete.js",
         "<%= dirs.js.src %>/plugins/jquery.debounce.js",
-        "<%= dirs.js.src %>/plugins/jquery.mark.js",
+        "<%= dirs.js.node_modules %>/mark.js/dist/jquery.mark.min.js",
         "<%= dirs.js.src %>/plugins/jquery.timeago.js",
         "<%= dirs.js.src %>/plugins/jquery.timeago-extension.js",
         "<%= dirs.js.src %>/select2/select2.js",

package.json

@@ -22,6 +22,7 b''
     "grunt-contrib-watch": "^0.6.1",
     "grunt-webpack": "^3.1.3",
     "jquery": "1.11.3",
+    "mark.js": "8.11.1",
     "jshint": "^2.9.1-rc3",
     "moment": "^2.18.1",
     "mousetrap": "^1.6.1",

pkgs/node-packages.nix

@@ -409,13 +409,13 b' let'
         sha512 = "mJ3QKWtCchL1vhU/kZlJnLPuQZnlDOdZsyP0bbLWPGdYsQDnSBvyTLhzwBA3QAMlzEL9V4JHygEmK6/OTEyytA==";
       };
     };
-    "@webcomponents/shadycss-1.6.0" = {
+    "@webcomponents/shadycss-1.7.1" = {
       name = "_at_webcomponents_slash_shadycss";
       packageName = "@webcomponents/shadycss";
-      version = "1.6.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/@webcomponents/shadycss/-/shadycss-1.6.0.tgz";
-        sha512 = "iURGZZU6BaiRJtGgjMn208QxPkY11QwT/VmuHNa4Yb+kJxU/WODe4C8b0LDOtnk4KJzJg50hCfwvPRAjePEzbA==";
+      version = "1.7.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/@webcomponents/shadycss/-/shadycss-1.7.1.tgz";
+        sha512 = "6SZqLajRPWL0rrKDZOGF8PCBq5B9JqgFmE5rX5psk6i8WrqiMkSCuO8+rnirzViTsU5CqnjQPFC3OvG4YJdMrQ==";
       };
     };
     "@webcomponents/webcomponentsjs-2.2.1" = {
@@ -499,13 +499,13 b' let'
         sha1 = "82ffb02b29e662ae53bdc20af15947706739c536";
       };
     };
-    "ajv-6.6.1" = {
+    "ajv-6.6.2" = {
       name = "ajv";
       packageName = "ajv";
-      version = "6.6.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/ajv/-/ajv-6.6.1.tgz";
-        sha512 = "ZoJjft5B+EJBjUyu9C9Hc0OZyPZSSlOF+plzouTrg6UlA8f+e/n8NIgBFG/9tppJtpPWfthHakK7juJdNDODww==";
+      version = "6.6.2";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/ajv/-/ajv-6.6.2.tgz";
+        sha512 = "FBHEW6Jf5TB9MGBgUUA9XHkTbjXYfAUjY43ACMfmdMRHniyoMHjHjzD50OK8LGDWQwp4rWEsIq5kEqq7rvIM1g==";
       };
     };
     "ajv-keywords-3.2.0" = {
@@ -815,15 +815,6 b' let'
         sha1 = "b6bbe0b0674b9d719708ca38de8c237cb526c3d1";
       };
     };
-    "async-1.0.0" = {
-      name = "async";
-      packageName = "async";
-      version = "1.0.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/async/-/async-1.0.0.tgz";
-        sha1 = "f8fc04ca3a13784ade9e1641af98578cfbd647a9";
-      };
-    };
     "async-2.6.1" = {
       name = "async";
       packageName = "async";
@@ -1445,6 +1436,15 b' let'
         sha512 = "+hN/Zh2D08Mx65pZ/4g5bsmNiZUuChDiQfTUQ7qJr4/kuopCr88xZsAXv6mBoZEsUI4OuGHlX59qE94K2mMW8Q==";
       };
     };
+    "big.js-5.2.2" = {
+      name = "big.js";
+      packageName = "big.js";
+      version = "5.2.2";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/big.js/-/big.js-5.2.2.tgz";
+        sha512 = "vyL2OymJxmarO8gxMr0mhChsO9QGwhynfuu4+MHTAW6czfq9humCB7rKpUjDd9YUiDPU4mzpyupFSvOClAwbmQ==";
+      };
+    };
     "binary-extensions-1.12.0" = {
       name = "binary-extensions";
       packageName = "binary-extensions";
@@ -1679,22 +1679,22 b' let'
         sha1 = "b534e7c734c4f81ec5fbe8aca2ad24354b962c6c";
       };
     };
-    "caniuse-db-1.0.30000912" = {
+    "caniuse-db-1.0.30000927" = {
       name = "caniuse-db";
       packageName = "caniuse-db";
-      version = "1.0.30000912";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/caniuse-db/-/caniuse-db-1.0.30000912.tgz";
-        sha512 = "uiepPdHcJ06Na9t15L5l+pp3NWQU4IETbmleghD6tqCqbIYqhHSu7nVfbK2gqPjfy+9jl/wHF1UQlyTszh9tJQ==";
-      };
-    };
-    "caniuse-lite-1.0.30000912" = {
+      version = "1.0.30000927";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/caniuse-db/-/caniuse-db-1.0.30000927.tgz";
+        sha512 = "CX/QvLA8oh7kQ9cHCCzFm0UZW4KwSyQSRJ5A1XtH42HaMJQ0yh+9fEVWagMqv9I1vSCtaqA5Mb8k0uKfv7jhDw==";
+      };
+    };
+    "caniuse-lite-1.0.30000927" = {
       name = "caniuse-lite";
       packageName = "caniuse-lite";
-      version = "1.0.30000912";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30000912.tgz";
-        sha512 = "M3zAtV36U+xw5mMROlTXpAHClmPAor6GPKAMD5Yi7glCB5sbMPFtnQ3rGpk4XqPdUrrTIaVYSJZxREZWNy8QJg==";
+      version = "1.0.30000927";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30000927.tgz";
+        sha512 = "ogq4NbUWf1uG/j66k0AmiO3GjqJAlQyF8n4w8a954cbCyFKmYGvRtgz6qkq2fWuduTXHibX7GyYL5Pg58Aks2g==";
       };
     };
     "caseless-0.12.0" = {
@@ -1733,13 +1733,13 b' let'
         sha1 = "a8115c55e4a702fe4d150abd3872822a7e09fc98";
       };
     };
-    "chalk-2.4.1" = {
+    "chalk-2.4.2" = {
       name = "chalk";
       packageName = "chalk";
-      version = "2.4.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/chalk/-/chalk-2.4.1.tgz";
-        sha512 = "ObN6h1v2fTJSmUXoS3nMQ92LbDK9be4TV+6G+omQlGJFdcUX5heKi1LZ1YnRMIgwTLEj3E24bT6tYni50rlCfQ==";
+      version = "2.4.2";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz";
+        sha512 = "Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==";
       };
     };
     "chokidar-2.0.4" = {
@@ -1958,15 +1958,6 b' let'
         sha1 = "2423fe6678ac0c5dae8852e5d0e5be08c997abcc";
       };
     };
-    "colors-1.0.3" = {
-      name = "colors";
-      packageName = "colors";
-      version = "1.0.3";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/colors/-/colors-1.0.3.tgz";
-        sha1 = "0433f44d809680fdeb60ed260f1b0c262e82a40b";
-      };
-    };
     "colors-1.1.2" = {
       name = "colors";
       packageName = "colors";
@@ -1976,13 +1967,13 b' let'
         sha1 = "168a4701756b6a7f51a12ce0c97bfa28c084ed63";
       };
     };
-    "colors-1.3.2" = {
+    "colors-1.3.3" = {
       name = "colors";
       packageName = "colors";
-      version = "1.3.2";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/colors/-/colors-1.3.2.tgz";
-        sha512 = "rhP0JSBGYvpcNQj4s5AdShMeE5ahMop96cTeDl/v9qQQm2fYClE2QXZRi8wLzc+GmXSxdIqqbOIAhyObEXDbfQ==";
+      version = "1.3.3";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/colors/-/colors-1.3.3.tgz";
+        sha512 = "mmGt/1pZqYRjMxB1axhTo16/snVZ5krrKkcmMeVKxzECMMXoCgnvTPp10QgHfcbQZw8Dq2jMNG6je4JlWU0gWg==";
       };
     };
     "combined-stream-1.0.7" = {
@@ -2102,13 +2093,13 b' let'
         sha512 = "Y+SQCF+0NoWQryez2zXn5J5knmr9z/9qSQt7fbL78u83rxmigOy8X5+BFn8CFSuX+nKT8gpYwJX68ekqtQt6ZA==";
       };
     };
-    "core-js-2.5.7" = {
+    "core-js-2.6.1" = {
       name = "core-js";
       packageName = "core-js";
-      version = "2.5.7";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/core-js/-/core-js-2.5.7.tgz";
-        sha512 = "RszJCAxg/PP6uzXVXL6BsxSXx/B05oJAQ2vkJRjyjrEcNVycaqOmNb5OTxZPE3xa5gwZduqza6L9JOCenh/Ecw==";
+      version = "2.6.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/core-js/-/core-js-2.6.1.tgz";
+        sha512 = "L72mmmEayPJBejKIWe2pYtGis5r0tQ5NaJekdhyXgeMQTpJoBsH0NL4ElY2LfSoV15xeQWKQ+XTTOZdyero5Xg==";
       };
     };
     "core-util-is-1.0.2" = {
@@ -2246,15 +2237,6 b' let'
         sha1 = "ddd52c587033f49e94b71fc55569f252e8ff5f85";
       };
     };
-    "cycle-1.0.3" = {
-      name = "cycle";
-      packageName = "cycle";
-      version = "1.0.3";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/cycle/-/cycle-1.0.3.tgz";
-        sha1 = "21e80b2be8580f98b468f379430662b046c34ad2";
-      };
-    };
     "cyclist-0.2.2" = {
       name = "cyclist";
       packageName = "cyclist";
@@ -2489,13 +2471,13 b' let'
         sha1 = "bd28773e2642881aec51544924299c5cd822185b";
       };
     };
-    "domelementtype-1.3.0" = {
+    "domelementtype-1.3.1" = {
       name = "domelementtype";
       packageName = "domelementtype";
-      version = "1.3.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/domelementtype/-/domelementtype-1.3.0.tgz";
-        sha1 = "b17aed82e8ab59e52dd9c19b1756e0fc187204c2";
+      version = "1.3.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/domelementtype/-/domelementtype-1.3.1.tgz";
+        sha512 = "BSKB+TSpMpFI/HOxCNr1O8aMOTZ8hT3pM3GQ0w/mWRmkhEDSFJkkyzz4XQsBV44BChwGkrDfMyjVD0eA2aFV3w==";
       };
     };
     "domhandler-2.1.0" = {
@@ -2552,13 +2534,13 b' let'
         sha1 = "3a83a904e54353287874c564b7549386849a98c9";
       };
     };
-    "electron-to-chromium-1.3.85" = {
+    "electron-to-chromium-1.3.98" = {
       name = "electron-to-chromium";
       packageName = "electron-to-chromium";
-      version = "1.3.85";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.3.85.tgz";
-        sha512 = "kWSDVVF9t3mft2OHVZy4K85X2beP6c6mFm3teFS/mLSDJpQwuFIWHrULCX+w6H1E55ZYmFRlT+ATAFRwhrYzsw==";
+      version = "1.3.98";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.3.98.tgz";
+        sha512 = "WIZdNuvE3dFr6kkPgv4d/cfswNZD6XbeLBM8baOIQTsnbf4xWrVEaLvp7oNnbnMWWXDqq7Tbv+H5JfciLTJm4Q==";
       };
     };
     "elliptic-6.4.1" = {
@@ -2633,13 +2615,13 b' let'
         sha512 = "MfrRBDWzIWifgq6tJj60gkAwtLNb6sQPlcFrSOflcP1aFmmruKQ2wRnze/8V6kgyz7H3FF8Npzv78mZ7XLLflg==";
       };
     };
-    "es-abstract-1.12.0" = {
+    "es-abstract-1.13.0" = {
       name = "es-abstract";
       packageName = "es-abstract";
-      version = "1.12.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/es-abstract/-/es-abstract-1.12.0.tgz";
-        sha512 = "C8Fx/0jFmV5IPoMOFPA9P9G5NtqW+4cOPit3MIuvR2t7Ag2K15EJTpxnHAYTzL+aYQJIESYeXZmDBfOBE1HcpA==";
+      version = "1.13.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/es-abstract/-/es-abstract-1.13.0.tgz";
+        sha512 = "vDZfg/ykNxQVwup/8E1BZhVzFfBxs9NqMzGcvIJrqg5k2/5Za2bWo40dK2J1pgLngZ7c+Shh8lwYtLGyrwPutg==";
       };
     };
     "es-to-primitive-1.2.0" = {
@@ -2651,15 +2633,6 b' let'
         sha512 = "qZryBOJjV//LaxLTV6UC//WewneB3LcXOL9NP++ozKVXsIIIpm/2c13UDiD9Jp2eThsecw9m3jPqDwTyobcdbg==";
       };
     };
-    "es6-promise-4.2.5" = {
-      name = "es6-promise";
-      packageName = "es6-promise";
-      version = "4.2.5";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/es6-promise/-/es6-promise-4.2.5.tgz";
-        sha512 = "n6wvpdE43VFtJq+lUDYDBFUwV8TZbuGXLV4D6wKafg13ldznKsyEvatubnmUe31zcvelSzOHF+XbaT+Bl9ObDg==";
-      };
-    };
     "es6-templates-0.2.3" = {
       name = "es6-templates";
       packageName = "es6-templates";
@@ -2777,13 +2750,13 b' let'
         sha512 = "/f2Go4TognH/KvCISP7OUsHn85hT9nUkxxA9BEWxFn+Oj9o8ZNLm/40hdlgSLyuOimsrTKLUMEorQexp/aPQeA==";
       };
     };
-    "execa-0.10.0" = {
+    "execa-1.0.0" = {
       name = "execa";
       packageName = "execa";
-      version = "0.10.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/execa/-/execa-0.10.0.tgz";
-        sha512 = "7XOMnz8Ynx1gGo/3hyV9loYNPWM94jG3+3T3Y8tsfSstFmETmENCMU/A/zj8Lyaj1lkgEepKepvd6240tBRvlw==";
+      version = "1.0.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/execa/-/execa-1.0.0.tgz";
+        sha512 = "adbxcyWV46qiHyvSp50TKt05tB4tK3HcmF7/nxfAdhnox83seTDbwnaqKO4sXRy7roHAIFqJP/Rw/AuEbX61LA==";
       };
     };
     "exit-0.1.2" = {
@@ -2858,15 +2831,6 b' let'
         sha512 = "Nmb6QXkELsuBr24CJSkilo6UHHgbekK5UiZgfE6UHD3Eb27YC6oD+bhcT+tJ6cl8dmsgdQxnWlcry8ksBIBLpw==";
       };
     };
-    "extract-zip-1.6.7" = {
-      name = "extract-zip";
-      packageName = "extract-zip";
-      version = "1.6.7";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/extract-zip/-/extract-zip-1.6.7.tgz";
-        sha1 = "a840b4b8af6403264c8db57f4f1a74333ef81fe9";
-      };
-    };
     "extsprintf-1.3.0" = {
       name = "extsprintf";
       packageName = "extsprintf";
@@ -2876,15 +2840,6 b' let'
         sha1 = "96918440e3041a7a414f8c52e3c574eb3c3e1e05";
       };
     };
-    "eyes-0.1.8" = {
-      name = "eyes";
-      packageName = "eyes";
-      version = "0.1.8";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/eyes/-/eyes-0.1.8.tgz";
-        sha1 = "62cf120234c683785d902348a800ef3e0cc20bc0";
-      };
-    };
     "fast-deep-equal-2.0.1" = {
       name = "fast-deep-equal";
       packageName = "fast-deep-equal";
@@ -2930,15 +2885,6 b' let'
         sha1 = "c14c5b3bf14d7417ffbfd990c0a7495cd9f337bc";
       };
     };
-    "fd-slicer-1.0.1" = {
-      name = "fd-slicer";
-      packageName = "fd-slicer";
-      version = "1.0.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/fd-slicer/-/fd-slicer-1.0.1.tgz";
-        sha1 = "8b5bcbd9ec327c5041bf9ab023fd6750f1177e65";
-      };
-    };
     "file-sync-cmp-0.1.1" = {
       name = "file-sync-cmp";
       packageName = "file-sync-cmp";
@@ -3002,22 +2948,22 b' let'
         sha1 = "9326b1488c22d1a6088650a86901b2d9a90a2cbc";
       };
     };
-    "fined-1.1.0" = {
+    "fined-1.1.1" = {
       name = "fined";
       packageName = "fined";
-      version = "1.1.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/fined/-/fined-1.1.0.tgz";
-        sha1 = "b37dc844b76a2f5e7081e884f7c0ae344f153476";
-      };
-    };
-    "flagged-respawn-1.0.0" = {
+      version = "1.1.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/fined/-/fined-1.1.1.tgz";
+        sha512 = "jQp949ZmEbiYHk3gkbdtpJ0G1+kgtLQBNdP5edFP7Fh+WAYceLQz6yO1SBj72Xkg8GVyTB3bBzAYrHJVh5Xd5g==";
+      };
+    };
+    "flagged-respawn-1.0.1" = {
       name = "flagged-respawn";
       packageName = "flagged-respawn";
-      version = "1.0.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/flagged-respawn/-/flagged-respawn-1.0.0.tgz";
-        sha1 = "4e79ae9b2eb38bf86b3bb56bf3e0a56aa5fcabd7";
+      version = "1.0.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/flagged-respawn/-/flagged-respawn-1.0.1.tgz";
+        sha512 = "lNaHNVymajmk0OJMBn8fVUAU1BtDeKIqKoVhk4xAALB57aALg6b4W0MfJ/cUE0g9YBXy5XhSlPIpYIJ7HaY/3Q==";
       };
     };
     "flatten-1.0.2" = {
@@ -3092,15 +3038,6 b' let'
         sha1 = "8bfb5502bde4a4d36cfdeea007fcca21d7e382af";
       };
     };
-    "fs-extra-1.0.0" = {
-      name = "fs-extra";
-      packageName = "fs-extra";
-      version = "1.0.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/fs-extra/-/fs-extra-1.0.0.tgz";
-        sha1 = "cd3ce5f7e7cb6145883fcae3191e9877f8587950";
-      };
-    };
     "fs-write-stream-atomic-1.0.10" = {
       name = "fs-write-stream-atomic";
       packageName = "fs-write-stream-atomic";
@@ -3155,13 +3092,13 b' let'
         sha512 = "3t6rVToeoZfYSGd8YoLFR2DJkiQrIiUrGcjvFX2mDw3bn6k2OtwHN0TNCLbBO+w8qTvimhDkv+LSscbJY1vE6w==";
       };
     };
-    "get-stream-3.0.0" = {
+    "get-stream-4.1.0" = {
       name = "get-stream";
       packageName = "get-stream";
-      version = "3.0.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/get-stream/-/get-stream-3.0.0.tgz";
-        sha1 = "8e943d1358dc37555054ecbe2edb05aa174ede14";
+      version = "4.1.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/get-stream/-/get-stream-4.1.0.tgz";
+        sha512 = "GMat4EJ5161kIy2HevLlr4luNjBgvmj413KaQA7jt4V8B4RDsfpHk7WQ9GVqfYyyx8OS/L66Kox+rJRNklLK7w==";
       };
     };
     "get-value-2.0.6" = {
@@ -3236,13 +3173,13 b' let'
         sha512 = "sKzpEkf11GpOFuw0Zzjzmt4B4UZwjOcG757PPvrfhxcLFbq0wpsgpOqxpxtxFiCG4DtG93M6XRVbF2oGdev7bg==";
       };
     };
-    "global-modules-path-2.3.0" = {
+    "global-modules-path-2.3.1" = {
       name = "global-modules-path";
       packageName = "global-modules-path";
-      version = "2.3.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/global-modules-path/-/global-modules-path-2.3.0.tgz";
-        sha512 = "HchvMJNYh9dGSCy8pOQ2O8u/hoXaL+0XhnrwH0RyLiSXMMTl9W3N6KUU73+JFOg5PGjtzl6VZzUQsnrpm7Szag==";
+      version = "2.3.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/global-modules-path/-/global-modules-path-2.3.1.tgz";
+        sha512 = "y+shkf4InI7mPRHSo2b/k6ix6+NLDtyccYv86whhxrSGX9wjPX1VMITmrDbE1eh7zkzhiWtW2sHklJYoQ62Cxg==";
       };
     };
     "global-prefix-1.0.2" = {
@@ -3533,22 +3470,13 b' let'
         sha1 = "5fc8686847ecd73499403319a6b0a3f3f6ae4918";
       };
     };
-    "hash.js-1.1.5" = {
+    "hash.js-1.1.7" = {
       name = "hash.js";
       packageName = "hash.js";
-      version = "1.1.5";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/hash.js/-/hash.js-1.1.5.tgz";
-        sha512 = "eWI5HG9Np+eHV1KQhisXWwM+4EPPYe5dFX1UZZH7k/E3JzDEazVH+VGlZi6R94ZqImq+A3D1mCEtrFIfg/E7sA==";
-      };
-    };
-    "hasha-2.2.0" = {
-      name = "hasha";
-      packageName = "hasha";
-      version = "2.2.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/hasha/-/hasha-2.2.0.tgz";
-        sha1 = "78d7cbfc1e6d66303fe79837365984517b2f6ee1";
+      version = "1.1.7";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/hash.js/-/hash.js-1.1.7.tgz";
+        sha512 = "taOaskGt4z4SOANNseOviYDvjEJinIkRgmp7LbKP2YTTmVxWBl87s/uzK9r+44BclBSp2X7K1hqeNfz9JbBeXA==";
       };
     };
     "hawk-3.1.3" = {
@@ -4217,13 +4145,13 b' let'
         sha1 = "dd8b74278b27102d29df63eae28308a8cfa1b583";
       };
     };
-    "js-base64-2.4.9" = {
+    "js-base64-2.5.0" = {
       name = "js-base64";
       packageName = "js-base64";
-      version = "2.4.9";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/js-base64/-/js-base64-2.4.9.tgz";
-        sha512 = "xcinL3AuDJk7VSzsHgb9DvvIXayBbadtMZ4HFPx8rUszbW1MuNMlwYVC4zzCZ6e1sqZpnNS5ZFYOhXqA39T7LQ==";
+      version = "2.5.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/js-base64/-/js-base64-2.5.0.tgz";
+        sha512 = "wlEBIZ5LP8usDylWbDNhKPEFVFdI5hCHpnVoT/Ysvoi/PRhJENm/Rlh9TvjYB38HFfKZN7OzEbRjmjvLkFw11g==";
       };
     };
     "js-tokens-3.0.2" = {
@@ -4280,13 +4208,13 b' let'
         sha1 = "46c3fec8c1892b12b0833db9bc7622176dbab34b";
       };
     };
-    "jshint-2.9.6" = {
+    "jshint-2.9.7" = {
       name = "jshint";
       packageName = "jshint";
-      version = "2.9.6";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/jshint/-/jshint-2.9.6.tgz";
-        sha512 = "KO9SIAKTlJQOM4lE64GQUtGBRpTOuvbrRrSZw3AhUxMNG266nX9hK2cKA4SBhXOj0irJGyNyGSLT62HGOVDEOA==";
+      version = "2.9.7";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/jshint/-/jshint-2.9.7.tgz";
+        sha512 = "Q8XN38hGsVQhdlM+4gd1Xl7OB1VieSuCJf+fEJjpo59JH99bVJhXRXAh26qQ15wfdd1VPMuDWNeSWoNl53T4YA==";
       };
     };
     "json-parse-better-errors-1.0.2" = {
@@ -4343,13 +4271,13 b' let'
         sha1 = "1eade7acc012034ad84e2396767ead9fa5495821";
       };
     };
-    "jsonfile-2.4.0" = {
-      name = "jsonfile";
-      packageName = "jsonfile";
-      version = "2.4.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/jsonfile/-/jsonfile-2.4.0.tgz";
-        sha1 = "3736a2b428b87bbda0cc83b53fa3d633a35c2ae8";
+    "json5-1.0.1" = {
+      name = "json5";
+      packageName = "json5";
+      version = "1.0.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/json5/-/json5-1.0.1.tgz";
+        sha512 = "aKS4WQjPenRxiQsC93MNfjx+nbF4PAdYzmd/1JIj8HYzqfbu86beTuNgXDzPknWk0n0uARlyewZo4s++ES36Ow==";
       };
     };
     "jsonify-0.0.0" = {
@@ -4370,15 +4298,6 b' let'
         sha1 = "313e66bc1e5cc06e438bc1b7499c2e5c56acb6a2";
       };
     };
-    "kew-0.7.0" = {
-      name = "kew";
-      packageName = "kew";
-      version = "0.7.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/kew/-/kew-0.7.0.tgz";
-        sha1 = "79d93d2d33363d6fdd2970b335d9141ad591d79b";
-      };
-    };
     "kind-of-3.2.2" = {
       name = "kind-of";
       packageName = "kind-of";
@@ -4415,15 +4334,6 b' let'
         sha512 = "s5kLOcnH0XqDO+FvuaLX8DDjZ18CGFk7VygH40QoKPUQhW4e2rvM0rwUq0t8IQDOwYSeLK01U90OjzBTme2QqA==";
       };
     };
-    "klaw-1.3.1" = {
-      name = "klaw";
-      packageName = "klaw";
-      version = "1.3.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/klaw/-/klaw-1.3.1.tgz";
-        sha1 = "4088433b46b3b1ba259d78785d8e96f73ba02439";
-      };
-    };
     "lazy-cache-1.0.4" = {
       name = "lazy-cache";
       packageName = "lazy-cache";
@@ -4478,13 +4388,13 b' let'
         sha1 = "f86e6374d43205a6e6c60e9196f17c0299bfb348";
       };
     };
-    "loader-utils-1.1.0" = {
+    "loader-utils-1.2.3" = {
       name = "loader-utils";
       packageName = "loader-utils";
-      version = "1.1.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/loader-utils/-/loader-utils-1.1.0.tgz";
-        sha1 = "c98aef488bcceda2ffb5e2de646d6a754429f5cd";
+      version = "1.2.3";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/loader-utils/-/loader-utils-1.2.3.tgz";
+        sha512 = "fkpz8ejdnEMG3s37wGL07iSBDg99O9D5yflE9RGNH3hRdx9SOwYfnGYdZOUIZitN8E+E2vkq3MUMYMvPYl5ZZA==";
       };
     };
     "locate-path-2.0.0" = {
@@ -4676,6 +4586,15 b' let'
         sha1 = "ecdca8f13144e660f1b5bd41f12f3479d98dfb8f";
       };
     };
+    "mark.js-8.11.1" = {
+      name = "mark.js";
+      packageName = "mark.js";
+      version = "8.11.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/mark.js/-/mark.js-8.11.1.tgz";
+        sha1 = "180f1f9ebef8b0e638e4166ad52db879beb2ffc5";
+      };
+    };
     "math-expression-evaluator-1.2.17" = {
       name = "math-expression-evaluator";
       packageName = "math-expression-evaluator";
@@ -4820,6 +4739,15 b' let'
         sha1 = "857fcabfc3397d2625b8228262e86aa7a011b05d";
       };
     };
+    "minimist-1.2.0" = {
+      name = "minimist";
+      packageName = "minimist";
+      version = "1.2.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz";
+        sha1 = "a35008b20f41383eec1fb914f4cd5df79a264284";
+      };
+    };
     "mississippi-2.0.0" = {
       name = "mississippi";
       packageName = "mississippi";
@@ -4847,13 +4775,13 b' let'
         sha1 = "30057438eac6cf7f8c4767f38648d6697d75c903";
       };
     };
-    "moment-2.22.2" = {
+    "moment-2.23.0" = {
       name = "moment";
       packageName = "moment";
-      version = "2.22.2";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/moment/-/moment-2.22.2.tgz";
-        sha1 = "3c257f9839fc0e93ff53149632239eb90783ff66";
+      version = "2.23.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/moment/-/moment-2.23.0.tgz";
+        sha512 = "3IE39bHVqFbWWaPOMHZF98Q9c3LDKGTmypMiTM2QygGXXElkFWIH7GxfmlwmY2vwa+wmNsoYZmG2iusf1ZjJoA==";
       };
     };
     "mousetrap-1.6.2" = {
@@ -4883,13 +4811,13 b' let'
         sha1 = "5608aeadfc00be6c2901df5f9861788de0d597c8";
       };
     };
-    "nan-2.11.1" = {
+    "nan-2.12.1" = {
       name = "nan";
       packageName = "nan";
-      version = "2.11.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/nan/-/nan-2.11.1.tgz";
-        sha512 = "iji6k87OSXa0CcrLl9z+ZiYSuR2o+c0bGuNmXdrhTQTakxytAFsC56SArGYoiHlJlFoHSnvmhpceZJaXkVuOtA==";
+      version = "2.12.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/nan/-/nan-2.12.1.tgz";
+        sha512 = "JY7V6lRkStKcKTvHO5NVSQRv+RV+FIL5pvDoLiAtSL9pKlC5x9PKQcZDsq7m4FO4d57mkhC6Z+QhAh3Jdk5JFw==";
       };
     };
     "nanomatch-1.2.13" = {
@@ -5144,13 +5072,13 b' let'
         sha1 = "ffbc4988336e0e833de0c168c7ef152121aa7fb3";
       };
     };
-    "os-locale-3.0.1" = {
+    "os-locale-3.1.0" = {
       name = "os-locale";
       packageName = "os-locale";
-      version = "3.0.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/os-locale/-/os-locale-3.0.1.tgz";
-        sha512 = "7g5e7dmXPtzcP4bgsZ8ixDVqA7oWYuEz4lOSujeWyliPai4gfVDiFIcwBg3aGCPnmSGfzOKTK3ccPn0CKv3DBw==";
+      version = "3.1.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/os-locale/-/os-locale-3.1.0.tgz";
+        sha512 = "Z8l3R4wYWM40/52Z+S265okfFj8Kt2cC2MKY+xNi3kFs+XGI7WXu/I309QQQYbRW4ijiZ+yxs9pqEhJh0DqW3Q==";
       };
     };
     "os-tmpdir-1.0.2" = {
@@ -5207,13 +5135,13 b' let'
         sha512 = "vvcXsLAJ9Dr5rQOPk7toZQZJApBl2K4J6dANSsEuh6QI41JYcsS/qhTGa9ErIUUgK3WNQoJYvylxvjqmiqEA9Q==";
       };
     };
-    "p-limit-2.0.0" = {
+    "p-limit-2.1.0" = {
       name = "p-limit";
       packageName = "p-limit";
-      version = "2.0.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/p-limit/-/p-limit-2.0.0.tgz";
-        sha512 = "fl5s52lI5ahKCernzzIyAP0QAZbGIovtVHGwpcu1Jr/EpzLVDI2myISHwGqK7m8uQFugVWSrbxH7XnhGtvEc+A==";
+      version = "2.1.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/p-limit/-/p-limit-2.1.0.tgz";
+        sha512 = "NhURkNcrVB+8hNfLuysU8enY5xn2KXphsHBaC2YmRNTZRc7RWusw6apSpdEj3jo4CMb6W9nrF6tTnsJsJeyu6g==";
       };
     };
     "p-locate-2.0.0" = {
@@ -5432,15 +5360,6 b' let'
         sha512 = "U/il5MsrZp7mGg3mSQfn742na2T+1/vHDCG5/iTI3X9MKUuYUZVLQhyRsg06mCgDBTd57TxzgZt7P+fYfjRLtA==";
       };
     };
-    "pend-1.2.0" = {
-      name = "pend";
-      packageName = "pend";
-      version = "1.2.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz";
-        sha1 = "7a57eb550a6783f9115331fcf4663d5c8e007a50";
-      };
-    };
     "performance-now-0.2.0" = {
       name = "performance-now";
       packageName = "performance-now";
@@ -5450,24 +5369,6 b' let'
         sha1 = "33ef30c5c77d4ea21c5a53869d91b56d8f2555e5";
       };
     };
-    "phantom-4.0.12" = {
-      name = "phantom";
-      packageName = "phantom";
-      version = "4.0.12";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/phantom/-/phantom-4.0.12.tgz";
-        sha512 = "Tz82XhtPmwCk1FFPmecy7yRGZG2btpzY2KI9fcoPT7zT9det0CcMyfBFPp1S8DqzsnQnm8ZYEfdy528mwVtksA==";
-      };
-    };
-    "phantomjs-prebuilt-2.1.16" = {
-      name = "phantomjs-prebuilt";
-      packageName = "phantomjs-prebuilt";
-      version = "2.1.16";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/phantomjs-prebuilt/-/phantomjs-prebuilt-2.1.16.tgz";
-        sha1 = "efd212a4a3966d3647684ea8ba788549be2aefef";
-      };
-    };
     "pify-3.0.0" = {
       name = "pify";
       packageName = "pify";
@@ -5477,24 +5378,6 b' let'
         sha1 = "e5a4acd2c101fdf3d9a4d07f0dbc4db49dd28176";
       };
     };
-    "pinkie-2.0.4" = {
-      name = "pinkie";
-      packageName = "pinkie";
-      version = "2.0.4";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/pinkie/-/pinkie-2.0.4.tgz";
-        sha1 = "72556b80cfa0d48a974e80e77248e80ed4f7f870";
-      };
-    };
-    "pinkie-promise-2.0.1" = {
-      name = "pinkie-promise";
-      packageName = "pinkie-promise";
-      version = "2.0.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/pinkie-promise/-/pinkie-promise-2.0.1.tgz";
-        sha1 = "2135d6dfa7a358c069ac9b178776288228450ffa";
-      };
-    };
     "pkg-dir-2.0.0" = {
       name = "pkg-dir";
       packageName = "pkg-dir";
@@ -5882,15 +5765,6 b' let'
         sha512 = "MtEC1TqN0EU5nephaJ4rAtThHtC86dNN9qCuEhtshvpVBkAW5ZO7BASN9REnF9eoXGcRub+pFuKEpOHE+HbEMw==";
       };
     };
-    "progress-1.1.8" = {
-      name = "progress";
-      packageName = "progress";
-      version = "1.1.8";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/progress/-/progress-1.1.8.tgz";
-        sha1 = "e260c78f6161cdd9b0e56cc3e0a85de17c7a57be";
-      };
-    };
     "promise-7.3.1" = {
       name = "promise";
       packageName = "promise";
@@ -5945,6 +5819,15 b' let'
         sha512 = "ruPMNRkN3MHP1cWJc9OWr+T/xDP0jhXYCLfJcBuX54hhfIBnaQmAUMfDcG4DM5UMWByBbJY69QSphm3jtDKIkA==";
       };
     };
+    "pump-3.0.0" = {
+      name = "pump";
+      packageName = "pump";
+      version = "3.0.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/pump/-/pump-3.0.0.tgz";
+        sha512 = "LwZy+p3SFs1Pytd/jYct4wpv49HiYCqd9Rlc5ZVdk0V+8Yzv6jR5Blk3TRmPL1ft69TxP0IMZGJ+WPFU2BFhww==";
+      };
+    };
     "pumpify-1.5.1" = {
       name = "pumpify";
       packageName = "pumpify";
@@ -6278,15 +6161,6 b' let'
         sha1 = "c6928946a0e06c5f8d6f8a9333469ffda46298a0";
       };
     };
-    "request-progress-2.0.1" = {
-      name = "request-progress";
-      packageName = "request-progress";
-      version = "2.0.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/request-progress/-/request-progress-2.0.1.tgz";
-        sha1 = "5d36bb57961c673aa5b788dbc8141fdf23b44e08";
-      };
-    };
     "require-directory-2.1.1" = {
       name = "require-directory";
       packageName = "require-directory";
@@ -6305,13 +6179,13 b' let'
         sha1 = "97f717b69d48784f5f526a6c5aa8ffdda055a4d1";
       };
     };
-    "resolve-1.8.1" = {
+    "resolve-1.9.0" = {
       name = "resolve";
       packageName = "resolve";
-      version = "1.8.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/resolve/-/resolve-1.8.1.tgz";
-        sha512 = "AicPrAC7Qu1JxPCZ9ZgCZlY35QgFnNqc+0LtbRNxnVw4TXvjQ72wnuL9JQcEBgXkI9JM8MsT9kaQoHcpCRJOYA==";
+      version = "1.9.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/resolve/-/resolve-1.9.0.tgz";
+        sha512 = "TZNye00tI67lwYvzxCxHGjwTNlUV70io54/Ed4j6PscB8xVfuBJpRenI/o6dVk0cY0PYTY27AgCoGGxRnYuItQ==";
       };
     };
     "resolve-cwd-2.0.0" = {
@@ -6377,13 +6251,13 b' let'
         sha1 = "e439be2aaee327321952730f99a8929e4fc50582";
       };
     };
-    "rimraf-2.6.2" = {
+    "rimraf-2.6.3" = {
       name = "rimraf";
       packageName = "rimraf";
-      version = "2.6.2";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/rimraf/-/rimraf-2.6.2.tgz";
-        sha512 = "lreewLK/BlghmxtfH36YYVg1i8IAce4TI7oao75I1g245+6BctqTVQiBP3YUJ9C6DQOXJmkYR9X9fCLtCOJc5w==";
+      version = "2.6.3";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/rimraf/-/rimraf-2.6.3.tgz";
+        sha512 = "mwqeW5XsA2qAejG46gYdENaxXjx9onRNCfn7L0duuP4hCuTIi/QO7PDK07KJfp1d+izWPrzEJDcSqBa0OZQriA==";
       };
     };
     "ripemd160-2.0.2" = {
@@ -6467,13 +6341,13 b' let'
         sha512 = "RS9R6R35NYgQn++fkDWaOmqGoj4Ek9gGs+DPxNUZKuwE183xjJroKvyo1IzVFeXvUrvmALy6FWD5xrdJT25gMg==";
       };
     };
-    "serialize-javascript-1.5.0" = {
+    "serialize-javascript-1.6.1" = {
       name = "serialize-javascript";
       packageName = "serialize-javascript";
-      version = "1.5.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/serialize-javascript/-/serialize-javascript-1.5.0.tgz";
-        sha512 = "Ga8c8NjAAp46Br4+0oZ2WxJCwIzwP60Gq1YPgU+39PiTVxyed/iKE/zyZI6+UlVYH5Q4PaQdHhcegIFPZTUfoQ==";
+      version = "1.6.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/serialize-javascript/-/serialize-javascript-1.6.1.tgz";
+        sha512 = "A5MOagrPFga4YaKQSWHryl7AXvbQkEqpw4NNYMTNYUNV51bA8ABHgYFpqKx+YFFrw59xMV1qGH1R4AgoNIVgCw==";
       };
     };
     "set-blocking-2.0.0" = {
@@ -6701,15 +6575,6 b' let'
         sha1 = "3e935d7ddd73631b97659956d55128e87b5084a3";
       };
     };
-    "split-1.0.1" = {
-      name = "split";
-      packageName = "split";
-      version = "1.0.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/split/-/split-1.0.1.tgz";
-        sha512 = "mTyOoPbrivtXnwnIxZRFYRrPNtEFKlpB2fvjSnCQUiAA6qAZzqwna5envK4uk6OIeP17CsdF3rSBGYVBsU0Tkg==";
-      };
-    };
     "split-string-3.1.0" = {
       name = "split-string";
       packageName = "split-string";
@@ -6728,13 +6593,13 b' let'
         sha1 = "04e6926f662895354f3dd015203633b857297e2c";
       };
     };
-    "sshpk-1.15.2" = {
+    "sshpk-1.16.0" = {
       name = "sshpk";
       packageName = "sshpk";
-      version = "1.15.2";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/sshpk/-/sshpk-1.15.2.tgz";
-        sha512 = "Ra/OXQtuh0/enyl4ETZAfTaeksa6BXks5ZcjpSUNrjBr0DvrJKX+1fsKDPpT9TBXgHAFsa4510aNVgI8g/+SzA==";
+      version = "1.16.0";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/sshpk/-/sshpk-1.16.0.tgz";
+        sha512 = "Zhev35/y7hRMcID/upReIvRse+I9SVhyVre/KTJSJQWMz3C3+G+HpO7m1wK/yckEtujKZ7dS4hkVxAnmHaIGVQ==";
       };
     };
     "ssri-5.3.0" = {
@@ -6746,15 +6611,6 b' let'
         sha512 = "XRSIPqLij52MtgoQavH/x/dU1qVKtWUAAZeOHsR9c2Ddi4XerFy3mc1alf+dLJKl9EUIm/Ht+EowFkTUOA6GAQ==";
       };
     };
-    "stack-trace-0.0.10" = {
-      name = "stack-trace";
-      packageName = "stack-trace";
-      version = "0.0.10";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/stack-trace/-/stack-trace-0.0.10.tgz";
-        sha1 = "547c70b347e8d32b4e108ea1a2a159e5fdde19c0";
-      };
-    };
     "static-extend-0.1.2" = {
       name = "static-extend";
       packageName = "static-extend";
@@ -6989,15 +6845,6 b' let'
         sha512 = "9I2ydhj8Z9veORCw5PRm4u9uebCn0mcCa6scWoNcbZ6dAtoo2618u9UUzxgmsCOreJpqDDuv61LvwofW7hLcBA==";
       };
     };
-    "throttleit-1.0.0" = {
-      name = "throttleit";
-      packageName = "throttleit";
-      version = "1.0.0";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/throttleit/-/throttleit-1.0.0.tgz";
-        sha1 = "9e785836daf46743145a5984b6268d828528ac6c";
-      };
-    };
     "through-2.3.8" = {
       name = "through";
       packageName = "through";
@@ -7259,15 +7106,6 b' let'
         sha1 = "8cdd8fbac4e2d2ea1e7e2e8097c42f442280f85b";
       };
     };
-    "unicode-5.2.0-0.7.5" = {
-      name = "unicode-5.2.0";
-      packageName = "unicode-5.2.0";
-      version = "0.7.5";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/unicode-5.2.0/-/unicode-5.2.0-0.7.5.tgz";
-        sha512 = "KVGLW1Bri30x00yv4HNM8kBxoqFXr0Sbo55735nvrlsx4PYBZol3UtoWgO492fSwmsetzPEZzy73rbU8OGXJcA==";
-      };
-    };
     "union-value-1.0.0" = {
       name = "union-value";
       packageName = "union-value";
@@ -7439,13 +7277,13 b' let'
         sha512 = "1wFuMUIM16MDJRCrpbpuEPTUGmM5QMUg0cr3KFwra2XgOgFcPGDQHDh3CszSCD2Zewc/dh/pamNEW8CbfDebUw==";
       };
     };
-    "v8flags-3.1.1" = {
+    "v8flags-3.1.2" = {
       name = "v8flags";
       packageName = "v8flags";
-      version = "3.1.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/v8flags/-/v8flags-3.1.1.tgz";
-        sha512 = "iw/1ViSEaff8NJ3HLyEjawk/8hjJib3E7pvG4pddVXfUg1983s3VGsiClDjhK64MQVDGqc1Q8r18S4VKQZS9EQ==";
+      version = "3.1.2";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/v8flags/-/v8flags-3.1.2.tgz";
+        sha512 = "MtivA7GF24yMPte9Rp/BWGCYQNaUj86zeYxV/x2RRJMKagImbbv3u8iJC57lNhWLPcGLJmHcHmFWkNsplbbLWw==";
       };
     };
     "vendors-1.0.2" = {
@@ -7583,15 +7421,6 b' let'
         sha1 = "5438cd2ea93b202efa3a19fe8887aee7c94f9c9d";
       };
     };
-    "winston-2.4.4" = {
-      name = "winston";
-      packageName = "winston";
-      version = "2.4.4";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/winston/-/winston-2.4.4.tgz";
-        sha512 = "NBo2Pepn4hK4V01UfcWcDlmiVTs7VTB1h7bgnB0rgP146bYhMxX0ypCz3lBOfNxCO4Zuek7yeT+y/zM1OfMw4Q==";
-      };
-    };
     "wordwrap-0.0.2" = {
       name = "wordwrap";
       packageName = "wordwrap";
@@ -7682,15 +7511,6 b' let'
         sha512 = "C6kB/WJDiaxONLJQnF8ccx9SEeoTTLek8RVbaOIsrAUS8VrBEXfmeSnCZxygc+XC2sNMBIwOOnfcxiynjHsVSQ==";
       };
     };
-    "yauzl-2.4.1" = {
-      name = "yauzl";
-      packageName = "yauzl";
-      version = "2.4.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/yauzl/-/yauzl-2.4.1.tgz";
-        sha1 = "9528f442dab1b2284e58b4379bb194e22e0c4005";
-      };
-    };
   };
   args = {
     name = "rhodecode-enterprise";
@@ -7743,7 +7563,7 b' let'
       sources."@webassemblyjs/wasm-parser-1.7.10"
       sources."@webassemblyjs/wast-parser-1.7.10"
       sources."@webassemblyjs/wast-printer-1.7.10"
-      sources."@webcomponents/shadycss-1.6.0"
+      sources."@webcomponents/shadycss-1.7.1"
       sources."@webcomponents/webcomponentsjs-2.2.1"
       sources."@xtuc/ieee754-1.2.0"
       sources."@xtuc/long-4.2.1"
@@ -7818,6 +7638,7 b' let'
       })
       (sources."babel-core-6.26.3" // {
         dependencies = [
+          sources."json5-0.5.1"
           sources."lodash-4.17.11"
           sources."minimatch-3.0.4"
         ];
@@ -7916,7 +7737,7 b' let'
       })
       sources."base64-js-1.3.0"
       sources."bcrypt-pbkdf-1.0.2"
-      sources."big.js-3.2.0"
+      sources."big.js-5.2.2"
       sources."binary-extensions-1.12.0"
       sources."bluebird-3.5.3"
       sources."bn.js-4.11.8"
@@ -7946,7 +7767,7 b' let'
           sources."graceful-fs-4.1.15"
           sources."lru-cache-4.1.5"
           sources."minimatch-3.0.4"
-          sources."rimraf-2.6.2"
+          sources."rimraf-2.6.3"
         ];
       })
       sources."cache-base-1.0.1"
@@ -7957,8 +7778,8 b' let'
           sources."browserslist-1.7.7"
         ];
       })
-      sources."caniuse-db-1.0.30000912"
-      sources."caniuse-lite-1.0.30000912"
+      sources."caniuse-db-1.0.30000927"
+      sources."caniuse-lite-1.0.30000927"
       sources."caseless-0.12.0"
       sources."center-align-0.1.3"
       sources."chalk-0.5.1"
@@ -8045,7 +7866,7 b' let'
         dependencies = [
           sources."glob-7.1.3"
           sources."minimatch-3.0.4"
-          sources."rimraf-2.6.2"
+          sources."rimraf-2.6.3"
         ];
       })
       sources."copy-descriptor-0.1.1"
@@ -8055,7 +7876,7 b' let'
           sources."minimatch-3.0.4"
         ];
       })
-      sources."core-js-2.5.7"
+      sources."core-js-2.6.1"
       sources."core-util-is-1.0.2"
       sources."create-ecdh-4.0.3"
       sources."create-hash-1.2.0"
@@ -8079,7 +7900,6 b' let'
       sources."cssesc-0.1.0"
       sources."cssnano-3.10.0"
       sources."csso-2.3.2"
-      sources."cycle-1.0.3"
       sources."cyclist-0.2.2"
       (sources."dashdash-1.14.1" // {
         dependencies = [
@@ -8116,7 +7936,7 b' let'
         ];
       })
       sources."domain-browser-1.2.0"
-      sources."domelementtype-1.3.0"
+      sources."domelementtype-1.3.1"
       sources."domhandler-2.3.0"
       sources."domutils-1.5.1"
       (sources."duplexify-3.6.1" // {
@@ -8126,7 +7946,7 b' let'
         ];
       })
       sources."ecc-jsbn-0.1.2"
-      sources."electron-to-chromium-1.3.85"
+      sources."electron-to-chromium-1.3.98"
       sources."elliptic-6.4.1"
       sources."emojis-list-2.1.0"
       sources."end-of-stream-1.4.1"
@@ -8137,9 +7957,8 b' let'
       })
       sources."entities-1.0.0"
       sources."errno-0.1.7"
-      sources."es-abstract-1.12.0"
+      sources."es-abstract-1.13.0"
       sources."es-to-primitive-1.2.0"
-      sources."es6-promise-4.2.5"
       sources."es6-templates-0.2.3"
       sources."escape-string-regexp-1.0.5"
       sources."eslint-scope-4.0.0"
@@ -8151,7 +7970,7 b' let'
       sources."eventemitter2-0.4.14"
       sources."events-1.1.1"
       sources."evp_bytestokey-1.0.3"
-      sources."execa-0.10.0"
+      sources."execa-1.0.0"
       sources."exit-0.1.2"
       (sources."expand-brackets-2.1.4" // {
         dependencies = [
@@ -8185,15 +8004,12 b' let'
           sources."extend-shallow-2.0.1"
         ];
       })
-      sources."extract-zip-1.6.7"
       sources."extsprintf-1.3.0"
-      sources."eyes-0.1.8"
       sources."fast-deep-equal-2.0.1"
       sources."fast-json-stable-stringify-2.0.0"
       sources."fastparse-1.1.2"
       sources."favico.js-0.3.10"
       sources."faye-websocket-0.4.4"
-      sources."fd-slicer-1.0.1"
       sources."file-sync-cmp-0.1.1"
       (sources."fill-range-4.0.0" // {
         dependencies = [
@@ -8209,8 +8025,8 b' let'
           sources."minimatch-0.3.0"
         ];
       })
-      sources."fined-1.1.0"
-      sources."flagged-respawn-1.0.0"
+      sources."fined-1.1.1"
+      sources."flagged-respawn-1.0.1"
       sources."flatten-1.0.2"
       (sources."flush-write-stream-1.0.3" // {
         dependencies = [
@@ -8229,11 +8045,6 b' let'
           sources."string_decoder-1.1.1"
         ];
       })
-      (sources."fs-extra-1.0.0" // {
-        dependencies = [
-          sources."graceful-fs-4.1.15"
-        ];
-      })
       (sources."fs-write-stream-atomic-1.0.10" // {
         dependencies = [
           sources."graceful-fs-4.1.15"
@@ -8244,7 +8055,11 b' let'
       sources."function-bind-1.1.1"
       sources."gaze-0.5.2"
       sources."get-caller-file-1.0.3"
-      sources."get-stream-3.0.0"
+      (sources."get-stream-4.1.0" // {
+        dependencies = [
+          sources."pump-3.0.0"
+        ];
+      })
       sources."get-value-2.0.6"
       sources."getobject-0.1.0"
       (sources."getpass-0.1.7" // {
@@ -8259,7 +8074,7 b' let'
       })
       sources."glob-parent-3.1.0"
       sources."global-modules-1.0.0"
-      sources."global-modules-path-2.3.0"
+      sources."global-modules-path-2.3.1"
       (sources."global-prefix-1.0.2" // {
         dependencies = [
           sources."which-1.3.1"
@@ -8351,8 +8166,7 b' let'
         ];
       })
       sources."hash-base-3.0.4"
-      sources."hash.js-1.1.5"
-      sources."hasha-2.2.0"
+      sources."hash.js-1.1.7"
       sources."hawk-3.1.3"
       sources."he-1.2.0"
       sources."hmac-drbg-1.0.1"
@@ -8369,6 +8183,8 b' let'
       })
       (sources."html-webpack-plugin-3.2.0" // {
         dependencies = [
+          sources."big.js-3.2.0"
+          sources."json5-0.5.1"
           sources."loader-utils-0.2.17"
           sources."lodash-4.17.11"
         ];
@@ -8381,7 +8197,7 b' let'
       (sources."icss-utils-2.1.0" // {
         dependencies = [
           sources."ansi-styles-3.2.1"
-          sources."chalk-2.4.1"
+          sources."chalk-2.4.2"
           sources."postcss-6.0.23"
           sources."source-map-0.6.1"
           sources."supports-color-5.5.0"
@@ -8395,7 +8211,7 b' let'
         dependencies = [
           sources."find-up-3.0.0"
           sources."locate-path-3.0.0"
-          sources."p-limit-2.0.0"
+          sources."p-limit-2.1.0"
           sources."p-locate-3.0.0"
           sources."p-try-2.0.0"
           sources."pkg-dir-3.0.0"
@@ -8445,12 +8261,12 b' let'
       sources."isobject-3.0.1"
       sources."isstream-0.1.2"
       sources."jquery-1.11.3"
-      sources."js-base64-2.4.9"
+      sources."js-base64-2.5.0"
       sources."js-tokens-3.0.2"
       sources."js-yaml-2.0.5"
       sources."jsbn-0.1.1"
       sources."jsesc-1.3.0"
-      (sources."jshint-2.9.6" // {
+      (sources."jshint-2.9.7" // {
         dependencies = [
           sources."lodash-4.17.11"
           sources."minimatch-3.0.4"
@@ -8461,25 +8277,14 b' let'
       sources."json-schema-traverse-0.4.1"
       sources."json-stable-stringify-1.0.1"
       sources."json-stringify-safe-5.0.1"
-      sources."json5-0.5.1"
-      (sources."jsonfile-2.4.0" // {
-        dependencies = [
-          sources."graceful-fs-4.1.15"
-        ];
-      })
+      sources."json5-1.0.1"
       sources."jsonify-0.0.0"
       (sources."jsprim-1.4.1" // {
         dependencies = [
           sources."assert-plus-1.0.0"
         ];
       })
-      sources."kew-0.7.0"
       sources."kind-of-6.0.2"
-      (sources."klaw-1.3.1" // {
-        dependencies = [
-          sources."graceful-fs-4.1.15"
-        ];
-      })
       sources."lazy-cache-1.0.4"
       sources."lcid-2.0.0"
       (sources."less-2.7.3" // {
@@ -8493,7 +8298,7 b' let'
         ];
       })
       sources."loader-runner-2.3.1"
-      sources."loader-utils-1.1.0"
+      sources."loader-utils-1.2.3"
       sources."locate-path-2.0.0"
       sources."lodash-0.9.2"
       sources."lodash.camelcase-4.3.0"
@@ -8510,6 +8315,7 b' let'
       sources."map-age-cleaner-0.1.3"
       sources."map-cache-0.2.2"
       sources."map-visit-1.0.0"
+      sources."mark.js-8.11.1"
       sources."math-expression-evaluator-1.2.17"
       sources."md5.js-1.3.5"
       sources."mem-4.0.0"
@@ -8528,25 +8334,29 b' let'
       sources."minimalistic-assert-1.0.1"
       sources."minimalistic-crypto-utils-1.0.1"
       sources."minimatch-0.2.14"
-      sources."minimist-0.0.8"
+      sources."minimist-1.2.0"
       sources."mississippi-2.0.0"
       (sources."mixin-deep-1.3.1" // {
         dependencies = [
           sources."is-extendable-1.0.1"
         ];
       })
-      sources."mkdirp-0.5.1"
-      sources."moment-2.22.2"
+      (sources."mkdirp-0.5.1" // {
+        dependencies = [
+          sources."minimist-0.0.8"
+        ];
+      })
+      sources."moment-2.23.0"
       sources."mousetrap-1.6.2"
       (sources."move-concurrently-1.0.1" // {
         dependencies = [
           sources."glob-7.1.3"
           sources."minimatch-3.0.4"
-          sources."rimraf-2.6.2"
+          sources."rimraf-2.6.3"
         ];
       })
       sources."ms-2.0.0"
-      sources."nan-2.11.1"
+      sources."nan-2.12.1"
       sources."nanomatch-1.2.13"
       sources."neo-async-2.6.0"
       sources."nice-try-1.0.5"
@@ -8598,7 +8408,7 b' let'
       sources."once-1.4.0"
       sources."os-browserify-0.3.0"
       sources."os-homedir-1.0.2"
-      sources."os-locale-3.0.1"
+      sources."os-locale-3.1.0"
       sources."os-tmpdir-1.0.2"
       sources."osenv-0.1.5"
       sources."p-defer-1.0.0"
@@ -8635,22 +8445,13 b' let'
       sources."path-root-regex-0.1.2"
       sources."path-type-3.0.0"
       sources."pbkdf2-3.0.17"
-      sources."pend-1.2.0"
       sources."performance-now-0.2.0"
-      sources."phantom-4.0.12"
-      (sources."phantomjs-prebuilt-2.1.16" // {
-        dependencies = [
-          sources."which-1.3.1"
-        ];
-      })
       sources."pify-3.0.0"
-      sources."pinkie-2.0.4"
-      sources."pinkie-promise-2.0.1"
       sources."pkg-dir-2.0.0"
       (sources."polymer-webpack-loader-2.0.3" // {
         dependencies = [
           sources."ansi-styles-3.2.1"
-          sources."chalk-2.4.1"
+          sources."chalk-2.4.2"
           sources."html-loader-0.5.5"
           (sources."postcss-6.0.23" // {
             dependencies = [
@@ -8700,7 +8501,7 b' let'
       (sources."postcss-modules-extract-imports-1.2.1" // {
         dependencies = [
           sources."ansi-styles-3.2.1"
-          sources."chalk-2.4.1"
+          sources."chalk-2.4.2"
           sources."postcss-6.0.23"
           sources."source-map-0.6.1"
           sources."supports-color-5.5.0"
@@ -8709,7 +8510,7 b' let'
       (sources."postcss-modules-local-by-default-1.2.0" // {
         dependencies = [
           sources."ansi-styles-3.2.1"
-          sources."chalk-2.4.1"
+          sources."chalk-2.4.2"
           sources."postcss-6.0.23"
           sources."source-map-0.6.1"
           sources."supports-color-5.5.0"
@@ -8718,7 +8519,7 b' let'
       (sources."postcss-modules-scope-1.1.0" // {
         dependencies = [
           sources."ansi-styles-3.2.1"
-          sources."chalk-2.4.1"
+          sources."chalk-2.4.2"
           sources."postcss-6.0.23"
           sources."source-map-0.6.1"
           sources."supports-color-5.5.0"
@@ -8727,7 +8528,7 b' let'
       (sources."postcss-modules-values-1.3.0" // {
         dependencies = [
           sources."ansi-styles-3.2.1"
-          sources."chalk-2.4.1"
+          sources."chalk-2.4.2"
           sources."postcss-6.0.23"
           sources."source-map-0.6.1"
           sources."supports-color-5.5.0"
@@ -8749,7 +8550,6 b' let'
       sources."private-0.1.8"
       sources."process-0.11.10"
       sources."process-nextick-args-2.0.0"
-      sources."progress-1.1.8"
       sources."promise-7.3.1"
       sources."promise-inflight-1.0.1"
       sources."prr-1.0.1"
@@ -8823,10 +8623,9 b' let'
       sources."repeat-string-1.6.1"
       sources."repeating-2.0.1"
       sources."request-2.81.0"
-      sources."request-progress-2.0.1"
       sources."require-directory-2.1.1"
       sources."require-main-filename-1.0.1"
-      sources."resolve-1.8.1"
+      sources."resolve-1.9.0"
       sources."resolve-cwd-2.0.0"
       sources."resolve-dir-1.0.1"
       sources."resolve-from-3.0.0"
@@ -8842,12 +8641,12 b' let'
       sources."sax-1.2.4"
       (sources."schema-utils-0.4.7" // {
         dependencies = [
-          sources."ajv-6.6.1"
+          sources."ajv-6.6.2"
         ];
       })
       sources."select-1.1.2"
       sources."semver-5.6.0"
-      sources."serialize-javascript-1.5.0"
+      sources."serialize-javascript-1.6.1"
       sources."set-blocking-2.0.0"
       (sources."set-value-2.0.0" // {
         dependencies = [
@@ -8897,16 +8696,14 b' let'
       sources."source-map-resolve-0.5.2"
       sources."source-map-support-0.4.18"
       sources."source-map-url-0.4.0"
-      sources."split-1.0.1"
       sources."split-string-3.1.0"
       sources."sprintf-js-1.0.3"
-      (sources."sshpk-1.15.2" // {
+      (sources."sshpk-1.16.0" // {
         dependencies = [
           sources."assert-plus-1.0.0"
         ];
       })
       sources."ssri-5.3.0"
-      sources."stack-trace-0.0.10"
       (sources."static-extend-0.1.2" // {
         dependencies = [
           sources."define-property-0.2.5"
@@ -8963,7 +8760,6 b' let'
         ];
       })
       sources."tapable-1.1.1"
-      sources."throttleit-1.0.0"
       sources."through-2.3.8"
       (sources."through2-2.0.5" // {
         dependencies = [
@@ -8993,9 +8789,11 b' let'
       sources."trim-right-1.0.1"
       (sources."ts-loader-1.3.3" // {
         dependencies = [
-          sources."colors-1.3.2"
+          sources."big.js-3.2.0"
+          sources."colors-1.3.3"
           sources."enhanced-resolve-3.4.1"
           sources."graceful-fs-4.1.15"
+          sources."json5-0.5.1"
           sources."loader-utils-0.2.17"
           sources."tapable-0.2.9"
         ];
@@ -9025,7 +8823,6 b' let'
       sources."unc-path-regex-0.1.2"
       sources."underscore-1.7.0"
       sources."underscore.string-2.2.1"
-      sources."unicode-5.2.0-0.7.5"
       (sources."union-value-1.0.0" // {
         dependencies = [
           sources."extend-shallow-2.0.1"
@@ -9066,7 +8863,7 b' let'
       sources."utila-0.4.0"
       sources."uuid-3.3.2"
       sources."v8-compile-cache-2.0.2"
-      sources."v8flags-3.1.1"
+      sources."v8flags-3.1.2"
       sources."vendors-1.0.2"
       (sources."verror-1.10.0" // {
         dependencies = [
@@ -9082,13 +8879,13 b' let'
       sources."waypoints-4.0.1"
       (sources."webpack-4.23.1" // {
         dependencies = [
-          sources."ajv-6.6.1"
+          sources."ajv-6.6.2"
         ];
       })
       (sources."webpack-cli-3.1.2" // {
         dependencies = [
           sources."ansi-styles-3.2.1"
-          sources."chalk-2.4.1"
+          sources."chalk-2.4.2"
           sources."supports-color-5.5.0"
         ];
       })
@@ -9121,12 +8918,6 b' let'
       sources."which-1.0.9"
       sources."which-module-2.0.0"
       sources."window-size-0.1.0"
-      (sources."winston-2.4.4" // {
-        dependencies = [
-          sources."async-1.0.0"
-          sources."colors-1.0.3"
-        ];
-      })
       sources."wordwrap-0.0.2"
       sources."worker-farm-1.6.0"
       (sources."wrap-ansi-2.1.0" // {
@@ -9144,13 +8935,12 b' let'
         dependencies = [
           sources."find-up-3.0.0"
           sources."locate-path-3.0.0"
-          sources."p-limit-2.0.0"
+          sources."p-limit-2.1.0"
           sources."p-locate-3.0.0"
           sources."p-try-2.0.0"
         ];
       })
       sources."yargs-parser-11.1.1"
-      sources."yauzl-2.4.1"
     ];
     buildInputs = globalBuildInputs;
     meta = {

pkgs/python-packages.nix

@@ -219,11 +219,11 b' self: super: {'
     };
   };
   "click" = super.buildPythonPackage {
-    name = "click-6.6";
+    name = "click-7.0";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/7a/00/c14926d8232b36b08218067bcd5853caefb4737cda3f0a47437151344792/click-6.6.tar.gz";
-      sha256 = "1sggipyz52crrybwbr9xvwxd4aqigvplf53k9w3ygxmzivd1jsnc";
+      url = "https://files.pythonhosted.org/packages/f8/5c/f60e9d8a1e77005f664b76ff8aeaee5bc05d0a91798afd7f53fc998dbc47/Click-7.0.tar.gz";
+      sha256 = "1mzjixd4vjbjvzb6vylki9w1556a9qmdh35kzmq6cign46av952v";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -260,11 +260,11 b' self: super: {'
     };
   };
   "configparser" = super.buildPythonPackage {
-    name = "configparser-3.5.0";
+    name = "configparser-3.7.1";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/7c/69/c2ce7e91c89dc073eb1aa74c0621c3eefbffe8216b3f9af9d3885265c01c/configparser-3.5.0.tar.gz";
-      sha256 = "0fi7vf09vi1588jd8f16a021m5y6ih2hy7rpbjb408xw45qb822k";
+      url = "https://files.pythonhosted.org/packages/b6/a6/eceea7c5a5dbcf56815bed411c38cabd8a879386be10717b160e7362b5a2/configparser-3.7.1.tar.gz";
+      sha256 = "0cnz213il9lhgda6x70fw7mfqr8da43s3wm343lwzhqx94mgmmav";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -374,11 +374,14 b' self: super: {'
     };
   };
   "dogpile.cache" = super.buildPythonPackage {
-    name = "dogpile.cache-0.6.7";
+    name = "dogpile.cache-0.7.1";
     doCheck = false;
+    propagatedBuildInputs = [
+      self."decorator"
+    ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/ee/bd/440da735a11c6087eed7cc8747fc4b995cbac2464168682f8ee1c8e43844/dogpile.cache-0.6.7.tar.gz";
-      sha256 = "1aw8rx8vhb75y7zc6gi67g21sw057jdx7i8m3jq7kf3nqavxx9zw";
+      url = "https://files.pythonhosted.org/packages/84/3e/dbf1cfc5228f1d3dca80ef714db2c5aaec5cd9efaf54d7e3daef6bc48b19/dogpile.cache-0.7.1.tar.gz";
+      sha256 = "0caazmrzhnfqb5yrp8myhw61ny637jj69wcngrpbvi31jlcpy6v9";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -407,30 +410,75 b' self: super: {'
     };
   };
   "elasticsearch" = super.buildPythonPackage {
-    name = "elasticsearch-2.3.0";
+    name = "elasticsearch-6.3.1";
     doCheck = false;
     propagatedBuildInputs = [
       self."urllib3"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/10/35/5fd52c5f0b0ee405ed4b5195e8bce44c5e041787680dc7b94b8071cac600/elasticsearch-2.3.0.tar.gz";
-      sha256 = "10ad2dk73xsys9vajwsncibs69asa63w1hgwz6lz1prjpyi80c5y";
+      url = "https://files.pythonhosted.org/packages/9d/ce/c4664e8380e379a9402ecfbaf158e56396da90d520daba21cfa840e0eb71/elasticsearch-6.3.1.tar.gz";
+      sha256 = "12y93v0yn7a4xmf969239g8gb3l4cdkclfpbk1qc8hx5qkymrnma";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   "elasticsearch-dsl" = super.buildPythonPackage {
-    name = "elasticsearch-dsl-2.2.0";
+    name = "elasticsearch-dsl-6.3.1";
     doCheck = false;
     propagatedBuildInputs = [
       self."six"
       self."python-dateutil"
       self."elasticsearch"
+      self."ipaddress"
+    ];
+    src = fetchurl {
+      url = "https://files.pythonhosted.org/packages/4c/0d/1549f50c591db6bb4e66cbcc8d34a6e537c3d89aa426b167c244fd46420a/elasticsearch-dsl-6.3.1.tar.gz";
+      sha256 = "1gh8a0shqi105k325hgwb9avrpdjh0mc6mxwfg9ba7g6lssb702z";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
+  };
+  "elasticsearch1" = super.buildPythonPackage {
+    name = "elasticsearch1-1.10.0";
+    doCheck = false;
+    propagatedBuildInputs = [
+      self."urllib3"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/66/2f/52a086968788e58461641570f45c3207a52d46ebbe9b77dc22b6a8ffda66/elasticsearch-dsl-2.2.0.tar.gz";
-      sha256 = "1g4kxzxsdwlsl2a9kscmx11pafgimhj7y8wrfksv8pgvpkfb9fwr";
+      url = "https://files.pythonhosted.org/packages/a6/eb/73e75f9681fa71e3157b8ee878534235d57f24ee64f0e77f8d995fb57076/elasticsearch1-1.10.0.tar.gz";
+      sha256 = "0g89444kd5zwql4vbvyrmi2m6l6dcj6ga98j4hqxyyyz6z20aki2";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
+  };
+  "elasticsearch1-dsl" = super.buildPythonPackage {
+    name = "elasticsearch1-dsl-0.0.12";
+    doCheck = false;
+    propagatedBuildInputs = [
+      self."six"
+      self."python-dateutil"
+      self."elasticsearch1"
+    ];
+    src = fetchurl {
+      url = "https://files.pythonhosted.org/packages/eb/9d/785342775cb10eddc9b8d7457d618a423b4f0b89d8b2b2d1bc27190d71db/elasticsearch1-dsl-0.0.12.tar.gz";
+      sha256 = "0ig1ly39v93hba0z975wnhbmzwj28w6w1sqlr2g7cn5spp732bhk";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
+  };
+  "elasticsearch2" = super.buildPythonPackage {
+    name = "elasticsearch2-2.5.0";
+    doCheck = false;
+    propagatedBuildInputs = [
+      self."urllib3"
+    ];
+    src = fetchurl {
+      url = "https://files.pythonhosted.org/packages/84/77/63cf63d4ba11d913b5278406f2a37b0712bec6fc85edfb6151a33eaeba25/elasticsearch2-2.5.0.tar.gz";
+      sha256 = "0ky0q16lbvz022yv6q3pix7aamf026p1y994537ccjf0p0dxnbxr";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
@@ -517,14 +565,14 b' self: super: {'
     };
   };
   "gevent" = super.buildPythonPackage {
-    name = "gevent-1.3.7";
+    name = "gevent-1.4.0";
     doCheck = false;
     propagatedBuildInputs = [
       self."greenlet"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/10/c1/9499b146bfa43aa4f1e0ed1bab1bd3209a4861d25650c11725036c731cf5/gevent-1.3.7.tar.gz";
-      sha256 = "0b0fr04qdk1p4sniv87fh8z5psac60x01pv054kpgi94520g81iz";
+      url = "https://files.pythonhosted.org/packages/ed/27/6c49b70808f569b66ec7fac2e78f076e9b204db9cf5768740cff3d5a07ae/gevent-1.4.0.tar.gz";
+      sha256 = "1lchr4akw2jkm5v4kz7bdm4wv3knkfhbfn9vkkz4s5yrkcxzmdqy";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -673,11 +721,11 b' self: super: {'
     };
   };
   "iso8601" = super.buildPythonPackage {
-    name = "iso8601-0.1.11";
+    name = "iso8601-0.1.12";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/c0/75/c9209ee4d1b5975eb8c2cba4428bde6b61bd55664a98290dd015cdb18e98/iso8601-0.1.11.tar.gz";
-      sha256 = "0c7gh3lsdjds262h0v1sqc66l7hqgfwbakn96qrhdbl0i3vm5yz8";
+      url = "https://files.pythonhosted.org/packages/45/13/3db24895497345fb44c4248c08b16da34a9eb02643cea2754b21b5ed08b0/iso8601-0.1.12.tar.gz";
+      sha256 = "10nyvvnrhw2w3p09v1ica4lgj6f4g9j3kkfx17qmraiq3w7b5i29";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -768,14 +816,14 b' self: super: {'
     };
   };
   "kombu" = super.buildPythonPackage {
-    name = "kombu-4.2.0";
+    name = "kombu-4.2.1";
     doCheck = false;
     propagatedBuildInputs = [
       self."amqp"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/ab/b1/46a7a8babf5e60f3b2ca081a100af8edfcf132078a726375f52a054e70cf/kombu-4.2.0.tar.gz";
-      sha256 = "1yz19qlqf0inl1mnwlpq9j6kj9r67clpy0xg99phyg4329rw80fn";
+      url = "https://files.pythonhosted.org/packages/39/9f/556b988833abede4a80dbd18b2bdf4e8ff4486dd482ed45da961347e8ed2/kombu-4.2.1.tar.gz";
+      sha256 = "10lh3hncvw67fz0k5vgbx3yh9gjfpqdlia1f13i28cgnc1nfrbc6";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -818,11 +866,11 b' self: super: {'
     };
   };
   "markupsafe" = super.buildPythonPackage {
-    name = "markupsafe-1.0";
+    name = "markupsafe-1.1.0";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz";
-      sha256 = "0rdn1s8x9ni7ss8rfiacj7x1085lx8mh2zdwqslnw8xc3l4nkgm6";
+      url = "https://files.pythonhosted.org/packages/ac/7e/1b4c2e05809a4414ebce0892fe1e32c14ace86ca7d50c70f00979ca9b3a3/MarkupSafe-1.1.0.tar.gz";
+      sha256 = "1lxirjypbdd3l9jl4vliilhfnhy7c7f2vlldqg1b0i74khn375sf";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -862,14 +910,14 b' self: super: {'
     };
   };
   "more-itertools" = super.buildPythonPackage {
-    name = "more-itertools-4.3.0";
+    name = "more-itertools-5.0.0";
     doCheck = false;
     propagatedBuildInputs = [
       self."six"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/88/ff/6d485d7362f39880810278bdc906c13300db05485d9c65971dec1142da6a/more-itertools-4.3.0.tar.gz";
-      sha256 = "17h3na0rdh8xq30w4b9pizgkdxmm51896bxw600x84jflg9vaxn4";
+      url = "https://files.pythonhosted.org/packages/dd/26/30fc0d541d9fdf55faf5ba4b0fd68f81d5bd2447579224820ad525934178/more-itertools-5.0.0.tar.gz";
+      sha256 = "1r12cm6mcdwdzz7d47a6g4l437xsvapdlgyhqay3i2nrlv03da9q";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -960,32 +1008,32 b' self: super: {'
     };
   };
   "paste" = super.buildPythonPackage {
-    name = "paste-2.0.3";
+    name = "paste-3.0.5";
     doCheck = false;
     propagatedBuildInputs = [
       self."six"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/30/c3/5c2f7c7a02e4f58d4454353fa1c32c94f79fa4e36d07a67c0ac295ea369e/Paste-2.0.3.tar.gz";
-      sha256 = "062jk0nlxf6lb2wwj6zc20rlvrwsnikpkh90y0dn8cjch93s6ii3";
+      url = "https://files.pythonhosted.org/packages/d4/41/91bde422400786b1b06357c1e6e3a5379f54dc3002aeb337cb767233304e/Paste-3.0.5.tar.gz";
+      sha256 = "1a6i8fh1fg8r4x800fvy9r82m15clwjim6yf2g9r4dff0y40dchv";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   "pastedeploy" = super.buildPythonPackage {
-    name = "pastedeploy-1.5.2";
+    name = "pastedeploy-2.0.1";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/0f/90/8e20cdae206c543ea10793cbf4136eb9a8b3f417e04e40a29d72d9922cbd/PasteDeploy-1.5.2.tar.gz";
-      sha256 = "1jz3m4hq8v6hyhfjz9425nd3nvn52cvbfipdcd72krjmla4qz1fm";
+      url = "https://files.pythonhosted.org/packages/19/a0/5623701df7e2478a68a1b685d1a84518024eef994cde7e4da8449a31616f/PasteDeploy-2.0.1.tar.gz";
+      sha256 = "02imfbbx1mi2h546f3sr37m47dk9qizaqhzzlhx8bkzxa6fzn8yl";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   "pastescript" = super.buildPythonPackage {
-    name = "pastescript-2.0.2";
+    name = "pastescript-3.0.0";
     doCheck = false;
     propagatedBuildInputs = [
       self."paste"
@@ -993,23 +1041,23 b' self: super: {'
       self."six"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/e5/f0/78e766c3dcc61a4f3a6f71dd8c95168ae9c7a31722b5663d19c1fdf62cb6/PasteScript-2.0.2.tar.gz";
-      sha256 = "1h3nnhn45kf4pbcv669ik4faw04j58k8vbj1hwrc532k0nc28gy0";
+      url = "https://files.pythonhosted.org/packages/08/2a/3797377a884ab9a064ad4d564ed612e54d26d7997caa8229c9c9df4eac31/PasteScript-3.0.0.tar.gz";
+      sha256 = "1hvmyz1sbn7ws1syw567ph7km9fi0wi75r3vlyzx6sk0z26xkm6r";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   "pathlib2" = super.buildPythonPackage {
-    name = "pathlib2-2.3.2";
+    name = "pathlib2-2.3.3";
     doCheck = false;
     propagatedBuildInputs = [
       self."six"
       self."scandir"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/db/a8/7d6439c1aec525ed70810abee5b7d7f3aa35347f59bc28343e8f62019aa2/pathlib2-2.3.2.tar.gz";
-      sha256 = "10yb0iv5x2hs631rcppkhbddx799d3h8pcwmkbh2a66ns3w71ccf";
+      url = "https://files.pythonhosted.org/packages/bf/d7/a2568f4596b75d2c6e2b4094a7e64f620decc7887f69a1f2811931ea15b9/pathlib2-2.3.3.tar.gz";
+      sha256 = "0hpp92vqqgcd8h92msm9slv161b1q160igjwnkf2ag6cx0c96695";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -1084,11 +1132,11 b' self: super: {'
     };
   };
   "pluggy" = super.buildPythonPackage {
-    name = "pluggy-0.8.0";
+    name = "pluggy-0.8.1";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/65/25/81d0de17cd00f8ca994a4e74e3c4baf7cd25072c0b831dad5c7d9d6138f8/pluggy-0.8.0.tar.gz";
-      sha256 = "1580p47l2zqzsza8jcnw1h2wh3vvmygk6ly8bvi4w0g8j14sjys4";
+      url = "https://files.pythonhosted.org/packages/38/e1/83b10c17688af7b2998fa5342fec58ecbd2a5a7499f31e606ae6640b71ac/pluggy-0.8.1.tar.gz";
+      sha256 = "05l6g42p9ilmabw0hlbiyxy6gyzjri41m5l11a8dzgvi77q35p4d";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -1110,11 +1158,11 b' self: super: {'
     };
   };
   "psutil" = super.buildPythonPackage {
-    name = "psutil-5.4.7";
+    name = "psutil-5.4.8";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/7d/9a/1e93d41708f8ed2b564395edfa3389f0fd6d567597401c2e5e2775118d8b/psutil-5.4.7.tar.gz";
-      sha256 = "0fsgmvzwbdbszkwfnqhib8jcxm4w6zyhvlxlcda0rfm5cyqj4qsv";
+      url = "https://files.pythonhosted.org/packages/e3/58/0eae6e4466e5abf779d7e2b71fac7fba5f59e00ea36ddb3ed690419ccb0f/psutil-5.4.8.tar.gz";
+      sha256 = "1hyna338sml2cl1mfb2gs89np18z27mvyhmq4ifh22x07n7mq9kf";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -1180,25 +1228,25 b' self: super: {'
     };
   };
   "pyasn1" = super.buildPythonPackage {
-    name = "pyasn1-0.4.4";
+    name = "pyasn1-0.4.5";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/10/46/059775dc8e50f722d205452bced4b3cc965d27e8c3389156acd3b1123ae3/pyasn1-0.4.4.tar.gz";
-      sha256 = "0drilmx5j25aplfr5wrml0030cs5fgxp9yp94fhllxgx28yjm3zm";
+      url = "https://files.pythonhosted.org/packages/46/60/b7e32f6ff481b8a1f6c8f02b0fd9b693d1c92ddd2efb038ec050d99a7245/pyasn1-0.4.5.tar.gz";
+      sha256 = "1xqh3jh2nfi2bflk5a0vn59y3pp1vn54f3ksx652sid92gz2096s";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   "pyasn1-modules" = super.buildPythonPackage {
-    name = "pyasn1-modules-0.2.2";
+    name = "pyasn1-modules-0.2.4";
     doCheck = false;
     propagatedBuildInputs = [
       self."pyasn1"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/37/33/74ebdc52be534e683dc91faf263931bc00ae05c6073909fde53999088541/pyasn1-modules-0.2.2.tar.gz";
-      sha256 = "0ivm850yi7ajjbi8j115qpsj95bgxdsx48nbjzg0zip788c3xkx0";
+      url = "https://files.pythonhosted.org/packages/bd/a5/ef7bf693e8a8f015386c9167483199f54f8a8ec01d1c737e05524f16e792/pyasn1-modules-0.2.4.tar.gz";
+      sha256 = "0z3w5dqrrvdplg9ma45j8n23xvyrj9ki8mg4ibqbn7l4qpl90855";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -1238,30 +1286,16 b' self: super: {'
     };
   };
   "pygments" = super.buildPythonPackage {
-    name = "pygments-2.3.0";
+    name = "pygments-2.3.1";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/63/a2/91c31c4831853dedca2a08a0f94d788fc26a48f7281c99a303769ad2721b/Pygments-2.3.0.tar.gz";
-      sha256 = "1z34ms51dh4jq4h3cizp7vd1dmsxcbvffkjsd2xxfav22nn6lrl2";
+      url = "https://files.pythonhosted.org/packages/64/69/413708eaf3a64a6abb8972644e0f20891a55e621c6759e2c3f3891e05d63/Pygments-2.3.1.tar.gz";
+      sha256 = "0ji87g09jph8jqcvclgb02qvxasdnr9pzvk90rl66d90yqcxmyjz";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
-  "pygments-markdown-lexer" = super.buildPythonPackage {
-    name = "pygments-markdown-lexer-0.1.0.dev39";
-    doCheck = false;
-    propagatedBuildInputs = [
-      self."pygments"
-    ];
-    src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/c3/12/674cdee66635d638cedb2c5d9c85ce507b7b2f91bdba29e482f1b1160ff6/pygments-markdown-lexer-0.1.0.dev39.zip";
-      sha256 = "1pzb5wy23q3fhs0rqzasjnw6hdzwjngpakb73i98cn0b8lk8q4jc";
-    };
-    meta = {
-      license = [ pkgs.lib.licenses.asl20 ];
-    };
-  };
   "pymysql" = super.buildPythonPackage {
     name = "pymysql-0.8.1";
     doCheck = false;
@@ -1285,35 +1319,34 b' self: super: {'
     };
   };
   "pyparsing" = super.buildPythonPackage {
-    name = "pyparsing-1.5.7";
+    name = "pyparsing-2.3.0";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/6f/2c/47457771c02a8ff0f302b695e094ec309e30452232bd79198ee94fda689f/pyparsing-1.5.7.tar.gz";
-      sha256 = "17z7ws076z977sclj628fvwrp8y9j2rvdjcsq42v129n1gwi8vk4";
+      url = "https://files.pythonhosted.org/packages/d0/09/3e6a5eeb6e04467b737d55f8bba15247ac0876f98fae659e58cd744430c6/pyparsing-2.3.0.tar.gz";
+      sha256 = "14k5v7n3xqw8kzf42x06bzp184spnlkya2dpjyflax6l3yrallzk";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   "pyramid" = super.buildPythonPackage {
-    name = "pyramid-1.9.2";
+    name = "pyramid-1.10.1";
     doCheck = false;
     propagatedBuildInputs = [
-      self."setuptools"
-      self."webob"
-      self."repoze.lru"
-      self."zope.interface"
-      self."zope.deprecation"
-      self."venusian"
-      self."translationstring"
-      self."pastedeploy"
+      self."hupper"
       self."plaster"
       self."plaster-pastedeploy"
-      self."hupper"
+      self."setuptools"
+      self."translationstring"
+      self."venusian"
+      self."webob"
+      self."zope.deprecation"
+      self."zope.interface"
+      self."repoze.lru"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/a0/c1/b321d07cfc4870541989ad131c86a1d593bfe802af0eca9718a0dadfb97a/pyramid-1.9.2.tar.gz";
-      sha256 = "09drsl0346nchgxp2j7sa5hlk7mkhfld9wvbd0wicacrp26a92fg";
+      url = "https://files.pythonhosted.org/packages/0a/3e/22e3ac9be1b70a01139adba8906ee4b8f628bb469fea3c52f6c97b73063c/pyramid-1.10.1.tar.gz";
+      sha256 = "1h5105nfh6rsrfjiyw20aavyibj36la3hajy6vh1fa77xb4y3hrp";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
@@ -1335,7 +1368,7 b' self: super: {'
     };
   };
   "pyramid-debugtoolbar" = super.buildPythonPackage {
-    name = "pyramid-debugtoolbar-4.4";
+    name = "pyramid-debugtoolbar-4.5";
     doCheck = false;
     propagatedBuildInputs = [
       self."pyramid"
@@ -1345,8 +1378,8 b' self: super: {'
       self."ipaddress"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/00/6f/c04eb4e715a7a5a4b24079ab7ffd1dceb1f70b2e24fc17686a2922dbac0a/pyramid_debugtoolbar-4.4.tar.gz";
-      sha256 = "17p7nxvapvy2hab1rah3ndq2kbs4v83pixj8x2n4m7008ai9lxsz";
+      url = "https://files.pythonhosted.org/packages/14/28/1f240239af340d19ee271ac62958158c79edb01a44ad8c9885508dd003d2/pyramid_debugtoolbar-4.5.tar.gz";
+      sha256 = "0x2p3409pnx66n6dx5vc0mk2r1cp1ydr8mp120w44r9pwcngbibl";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } pkgs.lib.licenses.bsdOriginal ];
@@ -1519,11 +1552,11 b' self: super: {'
     };
   };
   "python-editor" = super.buildPythonPackage {
-    name = "python-editor-1.0.3";
+    name = "python-editor-1.0.4";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/65/1e/adf6e000ea5dc909aa420352d6ba37f16434c8a3c2fa030445411a1ed545/python-editor-1.0.3.tar.gz";
-      sha256 = "0rf5xz8vw93v7mhdcvind7fkykipzga430wkcd7wk892xsn6dh53";
+      url = "https://files.pythonhosted.org/packages/0a/85/78f4a216d28343a67b7397c99825cff336330893f00601443f7c7b2f2234/python-editor-1.0.4.tar.gz";
+      sha256 = "0yrjh8w72ivqxi4i7xsg5b1vz15x8fg51xra7c3bgfyxqnyadzai";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 { fullName = "Apache"; } ];
@@ -1657,7 +1690,7 b' self: super: {'
     };
   };
   "rhodecode-enterprise-ce" = super.buildPythonPackage {
-    name = "rhodecode-enterprise-ce-4.15.2";
+    name = "rhodecode-enterprise-ce-4.16.0";
     buildInputs = [
       self."pytest"
       self."py"
@@ -1668,9 +1701,10 b' self: super: {'
       self."pytest-timeout"
       self."gprof2dot"
       self."mock"
-      self."webtest"
       self."cov-core"
       self."coverage"
+      self."webtest"
+      self."beautifulsoup4"
       self."configobj"
     ];
     doCheck = true;
@@ -1723,7 +1757,6 b' self: super: {'
       self."pycrypto"
       self."pycurl"
       self."pyflakes"
-      self."pygments-markdown-lexer"
       self."pygments"
       self."pyparsing"
       self."pyramid-beaker"
@@ -1794,9 +1827,10 b' self: super: {'
       self."pytest-timeout"
       self."gprof2dot"
       self."mock"
-      self."webtest"
       self."cov-core"
       self."coverage"
+      self."webtest"
+      self."beautifulsoup4"
     ];
     src = ./.;
     meta = {
@@ -1804,7 +1838,7 b' self: super: {'
     };
   };
   "rhodecode-tools" = super.buildPythonPackage {
-    name = "rhodecode-tools-1.0.1";
+    name = "rhodecode-tools-1.2.1";
     doCheck = false;
     propagatedBuildInputs = [
       self."click"
@@ -1813,14 +1847,16 b' self: super: {'
       self."mako"
       self."markupsafe"
       self."requests"
-      self."elasticsearch"
-      self."elasticsearch-dsl"
       self."urllib3"
       self."whoosh"
+      self."elasticsearch"
+      self."elasticsearch-dsl"
+      self."elasticsearch2"
+      self."elasticsearch1-dsl"
     ];
     src = fetchurl {
-      url = "https://code.rhodecode.com/rhodecode-tools-ce/archive/v1.0.1.tar.gz?md5=ffb5d6bcb855305b93cfe23ad42e500b";
-      sha256 = "0nr300s4sg685qs4wgbwlplwriawrwi6jq79z37frcnpyc89gpvm";
+      url = "https://code.rhodecode.com/rhodecode-tools-ce/archive/v1.2.1.tar.gz?md5=25bc2f7de1da318e547236d3fb463d28";
+      sha256 = "1k8l3s4mvshza1zay6dfxprq54fyb5dc85dqdva9wa3f466y0adk";
     };
     meta = {
       license = [ { fullName = "Apache 2.0 and Proprietary"; } ];
@@ -1864,11 +1900,11 b' self: super: {'
     };
   };
   "setuptools" = super.buildPythonPackage {
-    name = "setuptools-40.6.2";
+    name = "setuptools-40.8.0";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/b0/d1/8acb42f391cba52e35b131e442e80deffbb8d0676b93261d761b1f0ef8fb/setuptools-40.6.2.zip";
-      sha256 = "0r2c5hapirlzm34h7pl1lgkm6gk7bcrlrdj28qgsvaqg3f74vfw6";
+      url = "https://files.pythonhosted.org/packages/c2/f7/c7b501b783e5a74cf1768bc174ee4fb0a8a6ee5af6afa92274ff964703e0/setuptools-40.8.0.zip";
+      sha256 = "0k9hifpgahnw2a26w3cr346iy733k6d3nwh3f7g9m13y6f8fqkkf";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -1897,11 +1933,11 b' self: super: {'
     };
   };
   "simplejson" = super.buildPythonPackage {
-    name = "simplejson-3.11.1";
+    name = "simplejson-3.16.0";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/08/48/c97b668d6da7d7bebe7ea1817a6f76394b0ec959cb04214ca833c34359df/simplejson-3.11.1.tar.gz";
-      sha256 = "1rr58dppsq73p0qcd9bsw066cdd3v63sqv7j6sqni8frvm4jv8h1";
+      url = "https://files.pythonhosted.org/packages/e3/24/c35fb1c1c315fc0fffe61ea00d3f88e85469004713dab488dee4f35b0aff/simplejson-3.16.0.tar.gz";
+      sha256 = "19cws1syk8jzq2pw43878dv6fjkb0ifvjpx0i9aajix6kc9jkwxi";
     };
     meta = {
       license = [ { fullName = "Academic Free License (AFL)"; } pkgs.lib.licenses.mit ];
@@ -1945,25 +1981,25 b' self: super: {'
     };
   };
   "subprocess32" = super.buildPythonPackage {
-    name = "subprocess32-3.5.2";
+    name = "subprocess32-3.5.3";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/c3/5f/7117737fc7114061837a4f51670d863dd7f7f9c762a6546fa8a0dcfe61c8/subprocess32-3.5.2.tar.gz";
-      sha256 = "11v62shwmdys48g7ncs3a8jwwnkcl8d4zcwy6dk73z1zy2f9hazb";
+      url = "https://files.pythonhosted.org/packages/be/2b/beeba583e9877e64db10b52a96915afc0feabf7144dcbf2a0d0ea68bf73d/subprocess32-3.5.3.tar.gz";
+      sha256 = "1hr5fan8i719hmlmz73hf8rhq74014w07d8ryg7krvvf6692kj3b";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   "supervisor" = super.buildPythonPackage {
-    name = "supervisor-3.3.4";
+    name = "supervisor-3.3.5";
     doCheck = false;
     propagatedBuildInputs = [
       self."meld3"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/44/60/698e54b4a4a9b956b2d709b4b7b676119c833d811d53ee2500f1b5e96dc3/supervisor-3.3.4.tar.gz";
-      sha256 = "0wp62z9xprvz2krg02xnbwcnq6pxfq3byd8cxx8c2d8xznih28i1";
+      url = "https://files.pythonhosted.org/packages/ba/65/92575a8757ed576beaee59251f64a3287bde82bdc03964b89df9e1d29e1b/supervisor-3.3.5.tar.gz";
+      sha256 = "1w3ahridzbc6rxfpbyx8lij6pjlcgf2ymzyg53llkjqxalp6sk8v";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
@@ -2059,11 +2095,11 b' self: super: {'
     };
   };
   "urllib3" = super.buildPythonPackage {
-    name = "urllib3-1.21";
+    name = "urllib3-1.24.1";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/34/95/7b28259d0006ed681c424cd71a668363265eac92b67dddd018eb9a22bff8/urllib3-1.21.tar.gz";
-      sha256 = "0irnj4wvh2y36s4q3l2vas9qr9m766w6w418nb490j3mf8a8zw6h";
+      url = "https://files.pythonhosted.org/packages/b1/53/37d82ab391393565f2f831b8eedbffd57db5a718216f82f1a8b4d381a1c1/urllib3-1.24.1.tar.gz";
+      sha256 = "08lwd9f3hqznyf32vnzwvp87pchx062nkbgyrf67rwlkgj0jk5fy";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
@@ -2081,22 +2117,22 b' self: super: {'
     };
   };
   "venusian" = super.buildPythonPackage {
-    name = "venusian-1.1.0";
+    name = "venusian-1.2.0";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/38/24/b4b470ab9e0a2e2e9b9030c7735828c8934b4c6b45befd1bb713ec2aeb2d/venusian-1.1.0.tar.gz";
-      sha256 = "0zapz131686qm0gazwy8bh11vr57pr89jbwbl50s528sqy9f80lr";
+      url = "https://files.pythonhosted.org/packages/7e/6f/40a9d43ac77cb51cb62be5b5662d170f43f8037bdc4eab56336c4ca92bb7/venusian-1.2.0.tar.gz";
+      sha256 = "0ghyx66g8ikx9nx1mnwqvdcqm11i1vlq0hnvwl50s48bp22q5v34";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   "vine" = super.buildPythonPackage {
-    name = "vine-1.1.4";
+    name = "vine-1.2.0";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/32/23/36284986e011f3c130d802c3c66abd8f1aef371eae110ddf80c5ae22e1ff/vine-1.1.4.tar.gz";
-      sha256 = "0wkskb2hb494v9gixqnf4bl972p4ibcmxdykzpwjlfa5picns4aj";
+      url = "https://files.pythonhosted.org/packages/46/1a/c94317efa98040c5d50fe3cf9080cafb0372ff5afb0283dc018c751c6746/vine-1.2.0.tar.gz";
+      sha256 = "0xjz2sjbr5jrpjk411b7alkghdskhphgsqqrbi7abqfh2pli6j7f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -2182,18 +2218,18 b' self: super: {'
     };
   };
   "webob" = super.buildPythonPackage {
-    name = "webob-1.7.4";
+    name = "webob-1.8.4";
     doCheck = false;
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/75/34/731e23f52371852dfe7490a61644826ba7fe70fd52a377aaca0f4956ba7f/WebOb-1.7.4.tar.gz";
-      sha256 = "1na01ljg04z40il7vcrn8g29vaw7nvg1xvhk64cr4jys5wcay44d";
+      url = "https://files.pythonhosted.org/packages/e4/6c/99e322c3d4cc11d9060a67a9bf2f7c9c581f40988c11fffe89bb8c36bc5e/WebOb-1.8.4.tar.gz";
+      sha256 = "16cfg5y4n6sihz59vsmns2yqbfm0gfsn3l5xgz2g0pdhilaib0x4";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   "webtest" = super.buildPythonPackage {
-    name = "webtest-2.0.29";
+    name = "webtest-2.0.32";
     doCheck = false;
     propagatedBuildInputs = [
       self."six"
@@ -2202,8 +2238,8 b' self: super: {'
       self."beautifulsoup4"
     ];
     src = fetchurl {
-      url = "https://files.pythonhosted.org/packages/94/de/8f94738be649997da99c47b104aa3c3984ecec51a1d8153ed09638253d56/WebTest-2.0.29.tar.gz";
-      sha256 = "0bcj1ica5lnmj5zbvk46x28kgphcsgh7sfnwjmn0cr94mhawrg6v";
+      url = "https://files.pythonhosted.org/packages/27/9f/9e74449d272ffbef4fb3012e6dbc53c0b24822d545e7a33a342f80131e59/WebTest-2.0.32.tar.gz";
+      sha256 = "0qp0nnbazzm4ibjiyqfcn6f230svk09i4g58zg2i9x1ga06h48a2";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];

release.nix

@@ -1,12 +1,15 b''
 # This file defines how to "build" for packaging.
 
-{ doCheck ? false
+{ pkgs ? import <nixpkgs> {}
+, system ? builtins.currentSystem
+, doCheck ? false
 }:
 
 let
   enterprise_ce = import ./default.nix {
     inherit
-      doCheck;
+      doCheck
+      system;
 
     # disable checkPhase for build
     checkPhase = ''

requirements.txt

@@ -12,7 +12,7 b' bleach==3.0.2'
 celery==4.1.1
 chameleon==2.24
 channelstream==0.5.2
-click==6.6
+click==7.0
 colander==1.5.1
 # our custom configobj
 https://code.rhodecode.com/upstream/configobj/archive/a11ff0a0bd4fbda9e3a91267e720f88329efb4a6.tar.gz?md5=9916c524ea11a6c418217af6b28d4b3c#egg=configobj==5.0.6
@@ -20,7 +20,7 b' cssselect==1.0.3'
 decorator==4.1.2
 deform==2.0.7
 docutils==0.14.0
-dogpile.cache==0.6.7
+dogpile.cache==0.7.1
 dogpile.core==0.4.1
 ecdsa==0.13
 formencode==1.2.4
@@ -28,42 +28,41 b' future==0.14.3'
 futures==3.0.2
 gnureadline==6.3.8
 infrae.cache==1.0.1
-iso8601==0.1.11
+iso8601==0.1.12
 itsdangerous==0.24
 jinja2==2.9.6
 billiard==3.5.0.3
-kombu==4.2.0
+kombu==4.2.1
 lxml==4.2.5
 mako==1.0.7
 markdown==2.6.11
-markupsafe==1.0.0
+markupsafe==1.1.0
 msgpack-python==0.5.6
 pyotp==2.2.7
 packaging==15.2
-paste==2.0.3
-pastedeploy==1.5.2
-pastescript==2.0.2
-pathlib2==2.3.2
+paste==3.0.5
+pastedeploy==2.0.1
+pastescript==3.0.0
+pathlib2==2.3.3
 peppercorn==0.6
-psutil==5.4.7
+psutil==5.4.8
 py-bcrypt==0.4
 pycrypto==2.6.1
 pycurl==7.43.0.2
 pyflakes==0.8.1
-pygments-markdown-lexer==0.1.0.dev39
-pygments==2.3.0
-pyparsing==1.5.7
+pygments==2.3.1
+pyparsing==2.3.0
 pyramid-beaker==0.8
-pyramid-debugtoolbar==4.4.0
+pyramid-debugtoolbar==4.5.0
 pyramid-jinja2==2.7
 pyramid-mako==1.0.2
-pyramid==1.9.2
+pyramid==1.10.1
 pyramid_mailer==0.15.1
 python-dateutil
 python-ldap==3.1.0
 python-memcached==1.59
 python-pam==1.8.4
-python-saml
+python-saml==2.4.2
 pytz==2018.4
 tzlocal==1.5.1
 pyzmq==14.6.0
@@ -72,21 +71,21 b' redis==2.10.6'
 repoze.lru==0.7
 requests==2.9.1
 routes==2.4.1
-simplejson==3.11.1
+simplejson==3.16.0
 six==1.11.0
 sqlalchemy==1.1.18
 sshpubkeys==2.2.0
-subprocess32==3.5.2
-supervisor==3.3.4
+subprocess32==3.5.3
+supervisor==3.3.5
 tempita==0.5.2
 translationstring==1.3
-urllib3==1.21
+urllib3==1.24.1
 urlobject==2.4.3
-venusian==1.1.0
+venusian==1.2.0
 weberror==0.10.3
 webhelpers2==2.0
 webhelpers==1.3
-webob==1.7.4
+webob==1.8.4
 whoosh==2.7.4
 wsgiref==0.1.2
 zope.cachedescriptors==4.3.1
@@ -113,7 +112,7 b' invoke==0.13.0'
 bumpversion==0.5.3
 
 ## http servers
-gevent==1.3.7
+gevent==1.4.0
 greenlet==0.4.15
 gunicorn==19.9.0
 waitress==1.1.0
@@ -124,7 +123,7 b' ipdb==0.11.0'
 ipython==5.1.0
 
 ## rhodecode-tools, special case
-https://code.rhodecode.com/rhodecode-tools-ce/archive/v1.0.1.tar.gz?md5=ffb5d6bcb855305b93cfe23ad42e500b#egg=rhodecode-tools==1.0.1
+https://code.rhodecode.com/rhodecode-tools-ce/archive/v1.2.1.tar.gz?md5=25bc2f7de1da318e547236d3fb463d28#egg=rhodecode-tools==1.2.1
 
 ## appenlight
 appenlight-client==0.6.26

requirements_test.txt

@@ -9,6 +9,8 b' pytest-timeout==1.3.2'
 gprof2dot==2017.9.19
 
 mock==1.0.1
-webtest==2.0.29
 cov-core==1.15.0
 coverage==4.5.1
+
+webtest==2.0.32
+beautifulsoup4==4.6.3

rhodecode/VERSION

@@ -1,1 +1,1 b''
-4.15.2
  No newline at end of file
+4.16.0
  No newline at end of file

rhodecode/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -18,12 +18,6 b''
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
-"""
-
-RhodeCode, a web based repository management software
-versioning implementation: http://www.python.org/dev/peps/pep-0386/
-"""
-
 import os
 import sys
 import platform
@@ -51,7 +45,7 b' PYRAMID_SETTINGS = {}'
 EXTENSIONS = {}
 
 __version__ = ('.'.join((str(each) for each in VERSION[:3])))
-__dbversion__ = 91  # defines current db version for migrations
+__dbversion__ = 95  # defines current db version for migrations
 __platform__ = platform.system()
 __license__ = 'AGPLv3, and Commercial License'
 __author__ = 'RhodeCode GmbH'

rhodecode/api/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -21,6 +21,7 b''
 import inspect
 import itertools
 import logging
+import sys
 import types
 import fnmatch
 
@@ -38,6 +39,7 b' from rhodecode.api.exc import ('
 from rhodecode.apps._base import TemplateArgs
 from rhodecode.lib.auth import AuthUser
 from rhodecode.lib.base import get_ip_addr, attach_context_attributes
+from rhodecode.lib.exc_tracking import store_exception
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.utils2 import safe_str
 from rhodecode.lib.plugins.utils import get_plugin_settings
@@ -140,15 +142,14 b' def jsonrpc_error(request, message, reti'
 def exception_view(exc, request):
     rpc_id = getattr(request, 'rpc_id', None)
 
-    fault_message = 'undefined error'
     if isinstance(exc, JSONRPCError):
-        fault_message = exc.message
+        fault_message = safe_str(exc.message)
         log.debug('json-rpc error rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, JSONRPCValidationError):
         colander_exc = exc.colander_exception
         # TODO(marcink): think maybe of nicer way to serialize errors ?
         fault_message = colander_exc.asdict()
-        log.debug('json-rpc error rpc_id:%s "%s"', rpc_id, fault_message)
+        log.debug('json-rpc colander error rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, JSONRPCForbidden):
         fault_message = 'Access was denied to this resource.'
         log.warning('json-rpc forbidden call rpc_id:%s "%s"', rpc_id, fault_message)
@@ -170,6 +171,10 b' def exception_view(exc, request):'
 
         fault_message = "No such method: {}. Similar methods: {}".format(
             method, similar)
+    else:
+        fault_message = 'undefined error'
+        exc_info = exc.exc_info()
+        store_exception(id(exc_info), exc_info, prefix='rhodecode-api')
 
     return jsonrpc_error(request, fault_message, rpc_id)
 
@@ -292,8 +297,10 b' def request_view(request):'
         raise
     except Exception:
         log.exception('Unhandled exception occurred on api call: %s', func)
-        return jsonrpc_error(request, retid=request.rpc_id,
-                             message='Internal server error')
+        exc_info = sys.exc_info()
+        store_exception(id(exc_info), exc_info, prefix='rhodecode-api')
+        return jsonrpc_error(
+            request, retid=request.rpc_id, message='Internal server error')
 
 
 def setup_request(request):
@@ -414,8 +421,7 b' def add_jsonrpc_method(config, view, **k'
 
     if method is None:
         raise ConfigurationError(
-            'Cannot register a JSON-RPC method without specifying the '
-            '"method"')
+            'Cannot register a JSON-RPC method without specifying the "method"')
 
     # we define custom predicate, to enable to detect conflicting methods,
     # those predicates are kind of "translation" from the decorator variables
@@ -524,6 +530,7 b' def includeme(config):'
 
     # match filter by given method only
     config.add_view_predicate('jsonrpc_method', MethodPredicate)
+    config.add_view_predicate('jsonrpc_method_not_found', NotFoundPredicate)
 
     config.add_renderer(DEFAULT_RENDERER, ExtJsonRenderer(
         serializer=json.dumps, indent=4))
@@ -538,5 +545,4 b' def includeme(config):'
     config.scan(plugin_module, ignore='rhodecode.api.tests')
     # register some exception handling view
     config.add_view(exception_view, context=JSONRPCBaseError)
-    config.add_view_predicate('jsonrpc_method_not_found', NotFoundPredicate)
     config.add_notfound_view(exception_view, jsonrpc_method_not_found=True)

rhodecode/api/exc.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/conftest.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_add_field_to_repo.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_add_user_to_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -86,7 +86,9 b' class TestApi(object):'
     def test_api_non_existing_method_have_similar(self, request):
         id_, params = build_data(self.apikey, 'comment', args='xx')
         response = api_call(self.app, params)
-        expected = 'No such method: comment. Similar methods: changeset_comment, comment_pull_request, get_pull_request_comments, comment_commit'
+        expected = 'No such method: comment. ' \
+                   'Similar methods: changeset_comment, comment_pull_request, ' \
+                   'get_pull_request_comments, comment_commit, get_repo_comments'
         assert_error(id_, expected, given=response.body)
 
     def test_api_disabled_user(self, request):

rhodecode/api/tests/test_cleanup_sessions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_close_pull_request.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_comment_commit.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_comment_pull_request.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_create_gist.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_create_pull_request.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_create_repo.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_create_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_create_user.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_create_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_delete_gist.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_delete_repo.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_delete_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_delete_user.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_delete_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_deprecated_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_fork_repo.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_gist.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_gists.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_ip.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_locks.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_method.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -38,7 +38,7 b' class TestGetMethod(object):'
         response = api_call(self.app, params)
 
         expected = ['changeset_comment', 'comment_pull_request',
-                    'get_pull_request_comments', 'comment_commit']
+                    'get_pull_request_comments', 'comment_commit', 'get_repo_comments']
         assert_ok(id_, expected, given=response.body)
 
     def test_get_methods_on_single_match(self):

rhodecode/api/tests/test_get_pull_request.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -65,6 +65,7 b' class TestGetPullRequest(object):'
             'title': pull_request.title,
             'description': pull_request.description,
             'status': pull_request.status,
+            'state': pull_request.pull_request_state,
             'created_on': pull_request.created_on,
             'updated_on': pull_request.updated_on,
             'commit_ids': pull_request.revisions,

rhodecode/api/tests/test_get_pull_request_comments.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -59,6 +59,7 b' class TestGetPullRequestComments(object)'
                                 'status_lbl': 'Under Review'},
              'comment_text': 'Auto status change to |new_status|\n\n.. |new_status| replace:: *"Under Review"*',
              'comment_type': 'note',
+             'comment_resolved_by': None,
              'pull_request_version': None}
         ]
         assert_ok(id_, expected, response.body)

rhodecode/api/tests/test_get_pull_requests.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_repo.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_repo_changeset.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_repo_groups.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_repo_nodes.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_repo_refs.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_repos.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_server_info.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_user.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_user_groups.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_get_users.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_grant_user_group_permission.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_grant_user_group_permission_to_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_grant_user_group_permission_to_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_grant_user_permission.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_grant_user_permission_to_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_grant_user_permission_to_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_invalidate_cache.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_merge_pull_request.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -29,11 +29,10 b' from rhodecode.api.tests.utils import ('
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestMergePullRequest(object):
+
     @pytest.mark.backends("git", "hg")
     def test_api_merge_pull_request_merge_failed(self, pr_util, no_notifications):
         pull_request = pr_util.create_pull_request(mergeable=True)
-        author = pull_request.user_id
-        repo = pull_request.target_repo.repo_id
         pull_request_id = pull_request.pull_request_id
         pull_request_repo = pull_request.target_repo.repo_name
 
@@ -46,8 +45,7 b' class TestMergePullRequest(object):'
 
         # The above api call detaches the pull request DB object from the
         # session because of an unconditional transaction rollback in our
-        # middleware. Therefore we need to add it back here if we want to use
-        # it.
+        # middleware. Therefore we need to add it back here if we want to use it.
         Session().add(pull_request)
 
         expected = 'merge not possible for following reasons: ' \
@@ -55,6 +53,29 b' class TestMergePullRequest(object):'
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.backends("git", "hg")
+    def test_api_merge_pull_request_merge_failed_disallowed_state(
+            self, pr_util, no_notifications):
+        pull_request = pr_util.create_pull_request(mergeable=True, approved=True)
+        pull_request_id = pull_request.pull_request_id
+        pull_request_repo = pull_request.target_repo.repo_name
+
+        pr = PullRequest.get(pull_request_id)
+        pr.pull_request_state = pull_request.STATE_UPDATING
+        Session().add(pr)
+        Session().commit()
+
+        id_, params = build_data(
+            self.apikey, 'merge_pull_request',
+            repoid=pull_request_repo,
+            pullrequestid=pull_request_id)
+
+        response = api_call(self.app, params)
+        expected = 'Operation forbidden because pull request is in state {}, '\
+                   'only state {} is allowed.'.format(PullRequest.STATE_UPDATING,
+                                                      PullRequest.STATE_CREATED)
+        assert_error(id_, expected, given=response.body)
+
+    @pytest.mark.backends("git", "hg")
     def test_api_merge_pull_request(self, pr_util, no_notifications):
         pull_request = pr_util.create_pull_request(mergeable=True, approved=True)
         author = pull_request.user_id
@@ -88,6 +109,7 b' class TestMergePullRequest(object):'
         expected = {
             'executed': True,
             'failure_reason': 0,
+            'merge_status_message': 'This pull request can be automatically merged.',
             'possible': True,
             'merge_commit_id': pull_request.shadow_merge_ref.commit_id,
             'merge_ref': pull_request.shadow_merge_ref._asdict()
@@ -112,6 +134,107 b' class TestMergePullRequest(object):'
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.backends("git", "hg")
+    def test_api_merge_pull_request_as_another_user_no_perms_to_merge(
+            self, pr_util, no_notifications, user_util):
+        merge_user = user_util.create_user()
+        merge_user_id = merge_user.user_id
+        merge_user_username = merge_user.username
+
+        pull_request = pr_util.create_pull_request(mergeable=True, approved=True)
+
+        pull_request_id = pull_request.pull_request_id
+        pull_request_repo = pull_request.target_repo.repo_name
+
+        id_, params = build_data(
+            self.apikey, 'comment_pull_request',
+            repoid=pull_request_repo,
+            pullrequestid=pull_request_id,
+            status='approved')
+
+        response = api_call(self.app, params)
+        expected = {
+            'comment_id': response.json.get('result', {}).get('comment_id'),
+            'pull_request_id': pull_request_id,
+            'status': {'given': 'approved', 'was_changed': True}
+        }
+        assert_ok(id_, expected, given=response.body)
+        id_, params = build_data(
+            self.apikey, 'merge_pull_request',
+            repoid=pull_request_repo,
+            pullrequestid=pull_request_id,
+            userid=merge_user_id
+        )
+
+        response = api_call(self.app, params)
+        expected = 'merge not possible for following reasons: User `{}` ' \
+                   'not allowed to perform merge.'.format(merge_user_username)
+        assert_error(id_, expected, response.body)
+
+    @pytest.mark.backends("git", "hg")
+    def test_api_merge_pull_request_as_another_user(self, pr_util, no_notifications, user_util):
+        merge_user = user_util.create_user()
+        merge_user_id = merge_user.user_id
+        pull_request = pr_util.create_pull_request(mergeable=True, approved=True)
+        user_util.grant_user_permission_to_repo(
+            pull_request.target_repo, merge_user, 'repository.write')
+        author = pull_request.user_id
+        repo = pull_request.target_repo.repo_id
+        pull_request_id = pull_request.pull_request_id
+        pull_request_repo = pull_request.target_repo.repo_name
+
+        id_, params = build_data(
+            self.apikey, 'comment_pull_request',
+            repoid=pull_request_repo,
+            pullrequestid=pull_request_id,
+            status='approved')
+
+        response = api_call(self.app, params)
+        expected = {
+            'comment_id': response.json.get('result', {}).get('comment_id'),
+            'pull_request_id': pull_request_id,
+            'status': {'given': 'approved', 'was_changed': True}
+        }
+        assert_ok(id_, expected, given=response.body)
+
+        id_, params = build_data(
+            self.apikey, 'merge_pull_request',
+            repoid=pull_request_repo,
+            pullrequestid=pull_request_id,
+            userid=merge_user_id
+        )
+
+        response = api_call(self.app, params)
+
+        pull_request = PullRequest.get(pull_request_id)
+
+        expected = {
+            'executed': True,
+            'failure_reason': 0,
+            'merge_status_message': 'This pull request can be automatically merged.',
+            'possible': True,
+            'merge_commit_id': pull_request.shadow_merge_ref.commit_id,
+            'merge_ref': pull_request.shadow_merge_ref._asdict()
+        }
+
+        assert_ok(id_, expected, response.body)
+
+        journal = UserLog.query() \
+            .filter(UserLog.user_id == merge_user_id) \
+            .filter(UserLog.repository_id == repo) \
+            .order_by('user_log_id') \
+            .all()
+        assert journal[-2].action == 'repo.pull_request.merge'
+        assert journal[-1].action == 'repo.pull_request.close'
+
+        id_, params = build_data(
+            self.apikey, 'merge_pull_request',
+            repoid=pull_request_repo, pullrequestid=pull_request_id, userid=merge_user_id)
+        response = api_call(self.app, params)
+
+        expected = 'merge not possible for following reasons: This pull request is closed.'
+        assert_error(id_, expected, given=response.body)
+
+    @pytest.mark.backends("git", "hg")
     def test_api_merge_pull_request_repo_error(self, pr_util):
         pull_request = pr_util.create_pull_request()
         id_, params = build_data(
@@ -123,8 +246,7 b' class TestMergePullRequest(object):'
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.backends("git", "hg")
-    def test_api_merge_pull_request_non_admin_with_userid_error(self,
-                                                                pr_util):
+    def test_api_merge_pull_request_non_admin_with_userid_error(self, pr_util):
         pull_request = pr_util.create_pull_request(mergeable=True)
         id_, params = build_data(
             self.apikey_regular, 'merge_pull_request',

rhodecode/api/tests/test_pull.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_remove_field_from_repo.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_remove_user_from_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_repo_locking.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_rescan_repos.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_revoke_user_group_permission.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_revoke_user_group_permission_from_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_revoke_user_group_permission_from_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_revoke_user_permission.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_revoke_user_permission_from_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_revoke_user_permission_from_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_update_pull_request.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_update_repo.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_update_repo_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_update_user.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_update_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/test_utils.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/tests/utils.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -28,6 +28,19 b' from rhodecode.lib.ext_json import json'
 API_URL = '/_admin/api'
 
 
+def assert_call_ok(id_, given):
+    expected = jsonify({
+        'id': id_,
+        'error': None,
+        'result': None
+    })
+    given = json.loads(given)
+
+    assert expected['id'] == given['id']
+    assert expected['error'] == given['error']
+    return given['result']
+
+
 def assert_ok(id_, expected, given):
     expected = jsonify({
         'id': id_,
@@ -55,8 +68,6 b' def jsonify(obj):'
 def build_data(apikey, method, **kw):
     """
     Builds API data with given random ID
-
-    :param random_id:
     """
     random_id = random.randrange(1, 9999)
     return random_id, json.dumps({

rhodecode/api/utils.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2014-2018 RhodeCode GmbH
+# Copyright (C) 2014-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -30,7 +30,7 b' from rhodecode.lib.auth import ('
     HasPermissionAnyApi, HasRepoPermissionAnyApi, HasRepoGroupPermissionAnyApi)
 from rhodecode.lib.utils import safe_unicode
 from rhodecode.lib.vcs.exceptions import RepositoryError
-from rhodecode.controllers.utils import get_commit_from_ref_name
+from rhodecode.lib.view_utils import get_commit_from_ref_name
 from rhodecode.lib.utils2 import str2bool
 
 log = logging.getLogger(__name__)

rhodecode/api/views/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2015-2018 RhodeCode GmbH
+# Copyright (C) 2015-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/views/deprecated_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/views/gist_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/views/pull_request_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -26,13 +26,13 b' from rhodecode.api import jsonrpc_method'
 from rhodecode.api.utils import (
     has_superadmin_permission, Optional, OAttr, get_repo_or_error,
     get_pull_request_or_error, get_commit_or_error, get_user_or_error,
-    validate_repo_permissions, resolve_ref_or_error)
+    validate_repo_permissions, resolve_ref_or_error, validate_set_owner_permissions)
 from rhodecode.lib.auth import (HasRepoPermissionAnyApi)
 from rhodecode.lib.base import vcs_operation_context
 from rhodecode.lib.utils2 import str2bool
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import CommentsModel
-from rhodecode.model.db import Session, ChangesetStatus, ChangesetComment
+from rhodecode.model.db import Session, ChangesetStatus, ChangesetComment, PullRequest
 from rhodecode.model.pull_request import PullRequestModel, MergeCheck
 from rhodecode.model.settings import SettingsModel
 from rhodecode.model.validation_schema import Invalid
@@ -128,16 +128,21 b' def get_pull_request(request, apiuser, p'
     else:
         repo = pull_request.target_repo
 
-    if not PullRequestModel().check_user_read(
-            pull_request, apiuser, api=True):
+    if not PullRequestModel().check_user_read(pull_request, apiuser, api=True):
         raise JSONRPCError('repository `%s` or pull request `%s` '
                            'does not exist' % (repoid, pullrequestid))
-    data = pull_request.get_api_data()
+
+    # NOTE(marcink): only calculate and return merge state if the pr state is 'created'
+    # otherwise we can lock the repo on calculation of merge state while update/merge
+    # is happening.
+    merge_state = pull_request.pull_request_state == pull_request.STATE_CREATED
+    data = pull_request.get_api_data(with_merge_state=merge_state)
     return data
 
 
 @jsonrpc_method()
-def get_pull_requests(request, apiuser, repoid, status=Optional('new')):
+def get_pull_requests(request, apiuser, repoid, status=Optional('new'),
+                      merge_state=Optional(True)):
     """
     Get all pull requests from the repository specified in `repoid`.
 
@@ -151,6 +156,9 b' def get_pull_requests(request, apiuser, '
         * ``open``
         * ``closed``
     :type status: str
+    :param merge_state: Optional calculate merge state for each repository.
+        This could result in longer time to fetch the data
+    :type merge_state: bool
 
     Example output:
 
@@ -228,8 +236,10 b' def get_pull_requests(request, apiuser, '
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     status = Optional.extract(status)
-    pull_requests = PullRequestModel().get_all(repo, statuses=[status])
-    data = [pr.get_api_data() for pr in pull_requests]
+    merge_state = Optional.extract(merge_state, binary=True)
+    pull_requests = PullRequestModel().get_all(repo, statuses=[status],
+                                               order_by='id', order_dir='desc')
+    data = [pr.get_api_data(with_merge_state=merge_state) for pr in pull_requests]
     return data
 
 
@@ -259,6 +269,7 b' def merge_pull_request('
         "result": {
             "executed":         "<bool>",
             "failure_reason":   "<int>",
+            "merge_status_message":   "<str>",
             "merge_commit_id":  "<merge_commit_id>",
             "possible":         "<bool>",
             "merge_ref":        {
@@ -274,17 +285,25 b' def merge_pull_request('
         repo = get_repo_or_error(repoid)
     else:
         repo = pull_request.target_repo
-
+    auth_user = apiuser
     if not isinstance(userid, Optional):
         if (has_superadmin_permission(apiuser) or
                 HasRepoPermissionAnyApi('repository.admin')(
                     user=apiuser, repo_name=repo.repo_name)):
             apiuser = get_user_or_error(userid)
+            auth_user = apiuser.AuthUser()
         else:
             raise JSONRPCError('userid is not the same as your user')
 
-    check = MergeCheck.validate(
-        pull_request, auth_user=apiuser, translator=request.translate)
+    if pull_request.pull_request_state != PullRequest.STATE_CREATED:
+        raise JSONRPCError(
+            'Operation forbidden because pull request is in state {}, '
+            'only state {} is allowed.'.format(
+                pull_request.pull_request_state, PullRequest.STATE_CREATED))
+
+    with pull_request.set_state(PullRequest.STATE_UPDATING):
+        check = MergeCheck.validate(pull_request, auth_user=auth_user,
+                                    translator=request.translate)
     merge_possible = not check.failed
 
     if not merge_possible:
@@ -300,20 +319,20 b' def merge_pull_request('
     target_repo = pull_request.target_repo
     extras = vcs_operation_context(
         request.environ, repo_name=target_repo.repo_name,
-        username=apiuser.username, action='push',
+        username=auth_user.username, action='push',
         scm=target_repo.repo_type)
+    with pull_request.set_state(PullRequest.STATE_UPDATING):
     merge_response = PullRequestModel().merge_repo(
         pull_request, apiuser, extras=extras)
     if merge_response.executed:
-        PullRequestModel().close_pull_request(
-            pull_request.pull_request_id, apiuser)
+        PullRequestModel().close_pull_request(pull_request.pull_request_id, auth_user)
 
         Session().commit()
 
     # In previous versions the merge response directly contained the merge
     # commit id. It is now contained in the merge reference object. To be
     # backwards compatible we have to extract it again.
-    merge_response = merge_response._asdict()
+    merge_response = merge_response.asdict()
     merge_response['merge_commit_id'] = merge_response['merge_ref'].commit_id
 
     return merge_response
@@ -474,14 +493,21 b' def comment_pull_request('
     else:
         repo = pull_request.target_repo
 
+    auth_user = apiuser
     if not isinstance(userid, Optional):
         if (has_superadmin_permission(apiuser) or
                 HasRepoPermissionAnyApi('repository.admin')(
                     user=apiuser, repo_name=repo.repo_name)):
             apiuser = get_user_or_error(userid)
+            auth_user = apiuser.AuthUser()
         else:
             raise JSONRPCError('userid is not the same as your user')
 
+    if pull_request.is_closed():
+        raise JSONRPCError(
+            'pull request `%s` comment failed, pull request is closed' % (
+                pullrequestid,))
+
     if not PullRequestModel().check_user_read(
             pull_request, apiuser, api=True):
         raise JSONRPCError('repository `%s` does not exist' % (repoid,))
@@ -549,7 +575,7 b' def comment_pull_request('
         renderer=renderer,
         comment_type=comment_type,
         resolves_comment_id=resolves_comment_id,
-        auth_user=apiuser
+        auth_user=auth_user
     )
 
     if allowed_to_change_status and status:
@@ -589,8 +615,8 b' def comment_pull_request('
 @jsonrpc_method()
 def create_pull_request(
         request, apiuser, source_repo, target_repo, source_ref, target_ref,
-        title=Optional(''), description=Optional(''), description_renderer=Optional(''),
-        reviewers=Optional(None)):
+        owner=Optional(OAttr('apiuser')), title=Optional(''), description=Optional(''),
+        description_renderer=Optional(''), reviewers=Optional(None)):
     """
     Creates a new pull request.
 
@@ -611,6 +637,8 b' def create_pull_request('
     :type source_ref: str
     :param target_ref: Set the target ref name.
     :type target_ref: str
+    :param owner: user_id or username
+    :type owner: Optional(str)
     :param title: Optionally Set the pull request title, it's generated otherwise
     :type title: str
     :param description: Set the pull request description.
@@ -634,6 +662,8 b' def create_pull_request('
         _perms = ('repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, source_repo, source_db_repo, _perms)
 
+    owner = validate_set_owner_permissions(apiuser, owner)
+
     full_source_ref = resolve_ref_or_error(source_ref, source_db_repo)
     full_target_ref = resolve_ref_or_error(target_ref, target_db_repo)
 
@@ -679,7 +709,7 b' def create_pull_request('
 
     # recalculate reviewers logic, to make sure we can validate this
     reviewer_rules = get_default_reviewers_data(
-        apiuser.get_instance(), source_db_repo,
+        owner, source_db_repo,
         source_commit, target_db_repo, target_commit)
 
     # now MERGE our given with the calculated
@@ -706,7 +736,7 b' def create_pull_request('
     description_renderer = Optional.extract(description_renderer) or default_system_renderer
 
     pull_request = PullRequestModel().create(
-        created_by=apiuser.user_id,
+        created_by=owner.user_id,
         source_repo=source_repo,
         source_ref=full_source_ref,
         target_repo=target_repo,
@@ -844,9 +874,16 b' def update_pull_request('
 
     commit_changes = {"added": [], "common": [], "removed": []}
     if str2bool(Optional.extract(update_commits)):
+
+        if pull_request.pull_request_state != PullRequest.STATE_CREATED:
+            raise JSONRPCError(
+                'Operation forbidden because pull request is in state {}, '
+                'only state {} is allowed.'.format(
+                    pull_request.pull_request_state, PullRequest.STATE_CREATED))
+
+        with pull_request.set_state(PullRequest.STATE_UPDATING):
         if PullRequestModel().has_valid_update_type(pull_request):
-            update_response = PullRequestModel().update_commits(
-                pull_request)
+                update_response = PullRequestModel().update_commits(pull_request)
             commit_changes = update_response.changes or commit_changes
         Session().commit()
 

rhodecode/api/views/repo_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -29,13 +29,15 b' from rhodecode.api.utils import ('
     get_user_group_or_error, get_user_or_error, validate_repo_permissions,
     get_perm_or_error, parse_args, get_origin, build_commit_data,
     validate_set_owner_permissions)
-from rhodecode.lib import audit_logger
+from rhodecode.lib import audit_logger, rc_cache
 from rhodecode.lib import repo_maintenance
 from rhodecode.lib.auth import HasPermissionAnyApi, HasUserGroupPermissionAnyApi
 from rhodecode.lib.celerylib.utils import get_task_id
-from rhodecode.lib.utils2 import str2bool, time_to_datetime, safe_str
+from rhodecode.lib.utils2 import str2bool, time_to_datetime, safe_str, safe_int
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.exceptions import StatusChangeOnClosedPullRequestError
+from rhodecode.lib.vcs import RepositoryError
+from rhodecode.lib.vcs.exceptions import NodeDoesNotExistError
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import CommentsModel
 from rhodecode.model.db import (
@@ -380,7 +382,7 b' def get_repo_changesets(request, apiuser'
 
     try:
         commits = vcs_repo.get_commits(
-            start_id=start_rev, pre_load=pre_load)
+            start_id=start_rev, pre_load=pre_load, translate_tags=False)
     except TypeError as e:
         raise JSONRPCError(safe_str(e))
     except Exception:
@@ -428,8 +430,8 b' def get_repo_nodes(request, apiuser, rep'
         ``all`` (default), ``files`` and ``dirs``.
     :type ret_type: Optional(str)
     :param details: Returns extended information about nodes, such as
-        md5, binary, and or content.  The valid options are ``basic`` and
-        ``full``.
+        md5, binary, and or content.
+        The valid options are ``basic`` and ``full``.
     :type details: Optional(str)
     :param max_file_bytes: Only return file content under this file size bytes
     :type details: Optional(int)
@@ -441,10 +443,15 b' def get_repo_nodes(request, apiuser, rep'
         id : <id_given_in_input>
         result: [
                   {
-                    "name" : "<name>"
-                    "type" : "<type>",
-                    "binary": "<true|false>" (only in extended mode)
-                    "md5"  : "<md5 of file content>" (only in extended mode)
+                      "binary": false,
+                      "content": "File line\nLine2\n",
+                      "extension": "md",
+                      "lines": 2,
+                      "md5": "059fa5d29b19c0657e384749480f6422",
+                      "mimetype": "text/x-minidsrc",
+                      "name": "file.md",
+                      "size": 580,
+                      "type": "file"
                   },
                   ...
                 ]
@@ -453,16 +460,14 b' def get_repo_nodes(request, apiuser, rep'
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
-        _perms = (
-            'repository.admin', 'repository.write', 'repository.read',)
+        _perms = ('repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     ret_type = Optional.extract(ret_type)
     details = Optional.extract(details)
     _extended_types = ['basic', 'full']
     if details not in _extended_types:
-        raise JSONRPCError(
-            'ret_type must be one of %s' % (','.join(_extended_types)))
+        raise JSONRPCError('ret_type must be one of %s' % (','.join(_extended_types)))
     extended_info = False
     content = False
     if details == 'basic':
@@ -499,6 +504,149 b' def get_repo_nodes(request, apiuser, rep'
 
 
 @jsonrpc_method()
+def get_repo_file(request, apiuser, repoid, commit_id, file_path,
+                  max_file_bytes=Optional(None), details=Optional('basic'),
+                  cache=Optional(True)):
+    """
+    Returns a single file from repository at given revision.
+
+    This command can only be run using an |authtoken| with admin rights,
+    or users with at least read rights to |repos|.
+
+    :param apiuser: This is filled automatically from the |authtoken|.
+    :type apiuser: AuthUser
+    :param repoid: The repository name or repository ID.
+    :type repoid: str or int
+    :param commit_id: The revision for which listing should be done.
+    :type commit_id: str
+    :param file_path: The path from which to start displaying.
+    :type file_path: str
+    :param details: Returns different set of information about nodes.
+        The valid options are ``minimal`` ``basic`` and ``full``.
+    :type details: Optional(str)
+    :param max_file_bytes: Only return file content under this file size bytes
+    :type max_file_bytes: Optional(int)
+    :param cache: Use internal caches for fetching files. If disabled fetching
+        files is slower but more memory efficient
+    :type cache: Optional(bool)
+    Example output:
+
+    .. code-block:: bash
+
+        id : <id_given_in_input>
+        result: {
+            "binary": false,
+            "extension": "py",
+            "lines": 35,
+            "content": "....",
+            "md5": "76318336366b0f17ee249e11b0c99c41",
+            "mimetype": "text/x-python",
+            "name": "python.py",
+            "size": 817,
+            "type": "file",
+        }
+        error:  null
+    """
+
+    repo = get_repo_or_error(repoid)
+    if not has_superadmin_permission(apiuser):
+        _perms = ('repository.admin', 'repository.write', 'repository.read',)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
+
+    cache = Optional.extract(cache, binary=True)
+    details = Optional.extract(details)
+    _extended_types = ['minimal', 'minimal+search', 'basic', 'full']
+    if details not in _extended_types:
+        raise JSONRPCError(
+            'ret_type must be one of %s, got %s' % (','.join(_extended_types)), details)
+    extended_info = False
+    content = False
+
+    if details == 'minimal':
+        extended_info = False
+
+    elif details == 'basic':
+        extended_info = True
+
+    elif details == 'full':
+        extended_info = content = True
+
+    try:
+        # check if repo is not empty by any chance, skip quicker if it is.
+        _scm = repo.scm_instance()
+        if _scm.is_empty():
+            return None
+
+        node = ScmModel().get_node(
+            repo, commit_id, file_path, extended_info=extended_info,
+            content=content, max_file_bytes=max_file_bytes, cache=cache)
+    except NodeDoesNotExistError:
+        raise JSONRPCError('There is no file in repo: `{}` at path `{}` for commit: `{}`'.format(
+            repo.repo_name, file_path, commit_id))
+    except Exception:
+        log.exception("Exception occurred while trying to get repo %s file",
+                      repo.repo_name)
+        raise JSONRPCError('failed to get repo: `{}` file at path {}'.format(
+            repo.repo_name, file_path))
+
+    return node
+
+
+@jsonrpc_method()
+def get_repo_fts_tree(request, apiuser, repoid, commit_id, root_path):
+    """
+    Returns a list of tree nodes for path at given revision. This api is built
+    strictly for usage in full text search building, and shouldn't be consumed
+
+    This command can only be run using an |authtoken| with admin rights,
+    or users with at least read rights to |repos|.
+
+    """
+
+    repo = get_repo_or_error(repoid)
+    if not has_superadmin_permission(apiuser):
+        _perms = ('repository.admin', 'repository.write', 'repository.read',)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
+
+    repo_id = repo.repo_id
+    cache_seconds = safe_int(rhodecode.CONFIG.get('rc_cache.cache_repo.expiration_time'))
+    cache_on = cache_seconds > 0
+
+    cache_namespace_uid = 'cache_repo.{}'.format(repo_id)
+    region = rc_cache.get_or_create_region('cache_repo', cache_namespace_uid)
+
+    @region.conditional_cache_on_arguments(namespace=cache_namespace_uid,
+                                           condition=cache_on)
+    def compute_fts_tree(repo_id, commit_id, root_path, cache_ver):
+        return ScmModel().get_fts_data(repo_id, commit_id, root_path)
+
+    try:
+        # check if repo is not empty by any chance, skip quicker if it is.
+        _scm = repo.scm_instance()
+        if _scm.is_empty():
+            return []
+    except RepositoryError:
+        log.exception("Exception occurred while trying to get repo nodes")
+        raise JSONRPCError('failed to get repo: `%s` nodes' % repo.repo_name)
+
+    try:
+        # we need to resolve commit_id to a FULL sha for cache to work correctly.
+        # sending 'master' is a pointer that needs to be translated to current commit.
+        commit_id = _scm.get_commit(commit_id=commit_id).raw_id
+        log.debug(
+            'Computing FTS REPO TREE for repo_id %s commit_id `%s` '
+            'with caching: %s[TTL: %ss]' % (
+                repo_id, commit_id, cache_on, cache_seconds or 0))
+
+        tree_files = compute_fts_tree(repo_id, commit_id, root_path, 'v1')
+        return tree_files
+
+    except Exception:
+        log.exception("Exception occurred while trying to get repo nodes")
+        raise JSONRPCError('failed to get repo: `%s` nodes' % repo.repo_name)
+
+
+@jsonrpc_method()
 def get_repo_refs(request, apiuser, repoid):
     """
     Returns a dictionary of current references. It returns
@@ -1506,6 +1654,73 b' def comment_commit('
 
 
 @jsonrpc_method()
+def get_repo_comments(request, apiuser, repoid,
+                      commit_id=Optional(None), comment_type=Optional(None),
+                      userid=Optional(None)):
+    """
+    Get all comments for a repository
+
+    :param apiuser: This is filled automatically from the |authtoken|.
+    :type apiuser: AuthUser
+    :param repoid: Set the repository name or repository ID.
+    :type repoid: str or int
+    :param commit_id: Optionally filter the comments by the commit_id
+    :type commit_id: Optional(str), default: None
+    :param comment_type: Optionally filter the comments by the comment_type
+        one of: 'note', 'todo'
+    :type comment_type: Optional(str), default: None
+    :param userid: Optionally filter the comments by the author of comment
+    :type userid: Optional(str or int), Default: None
+
+    Example error output:
+
+    .. code-block:: bash
+
+        {
+            "id" : <id_given_in_input>,
+            "result" : [
+                {
+                  "comment_author": <USER_DETAILS>,
+                  "comment_created_on": "2017-02-01T14:38:16.309",
+                  "comment_f_path": "file.txt",
+                  "comment_id": 282,
+                  "comment_lineno": "n1",
+                  "comment_resolved_by": null,
+                  "comment_status": [],
+                  "comment_text": "This file needs a header",
+                  "comment_type": "todo"
+                }
+            ],
+            "error" :  null
+        }
+
+    """
+    repo = get_repo_or_error(repoid)
+    if not has_superadmin_permission(apiuser):
+        _perms = ('repository.read', 'repository.write', 'repository.admin')
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
+
+    commit_id = Optional.extract(commit_id)
+
+    userid = Optional.extract(userid)
+    if userid:
+        user = get_user_or_error(userid)
+    else:
+        user = None
+
+    comment_type = Optional.extract(comment_type)
+    if comment_type and comment_type not in ChangesetComment.COMMENT_TYPES:
+        raise JSONRPCError(
+                'comment_type must be one of `{}` got {}'.format(
+                    ChangesetComment.COMMENT_TYPES, comment_type)
+            )
+
+    comments = CommentsModel().get_repository_comments(
+        repo=repo, comment_type=comment_type, user=user, commit_id=commit_id)
+    return comments
+
+
+@jsonrpc_method()
 def grant_user_permission(request, apiuser, repoid, userid, perm):
     """
     Grant permissions for the specified user on the given repository,
@@ -1543,9 +1758,18 b' def grant_user_permission(request, apius'
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
+    perm_additions = [[user.user_id, perm.permission_name, "user"]]
     try:
+        changes = RepoModel().update_permissions(
+            repo=repo, perm_additions=perm_additions, cur_user=apiuser)
 
-        RepoModel().grant_user_permission(repo=repo, user=user, perm=perm)
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo.edit.permissions', action_data=action_data, user=apiuser, repo=repo)
 
         Session().commit()
         return {
@@ -1555,8 +1779,7 b' def grant_user_permission(request, apius'
             'success': True
         }
     except Exception:
-        log.exception(
-            "Exception occurred while trying edit permissions for repo")
+        log.exception("Exception occurred while trying edit permissions for repo")
         raise JSONRPCError(
             'failed to edit permission for user: `%s` in repo: `%s`' % (
                 userid, repoid
@@ -1597,8 +1820,19 b' def revoke_user_permission(request, apiu'
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
+    perm_deletions = [[user.user_id, None, "user"]]
     try:
-        RepoModel().revoke_user_permission(repo=repo, user=user)
+        changes = RepoModel().update_permissions(
+            repo=repo, perm_deletions=perm_deletions, cur_user=user)
+
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo.edit.permissions', action_data=action_data, user=apiuser, repo=repo)
+
         Session().commit()
         return {
             'msg': 'Revoked perm for user: `%s` in repo: `%s`' % (
@@ -1607,8 +1841,7 b' def revoke_user_permission(request, apiu'
             'success': True
         }
     except Exception:
-        log.exception(
-            "Exception occurred while trying revoke permissions to repo")
+        log.exception("Exception occurred while trying revoke permissions to repo")
         raise JSONRPCError(
             'failed to edit permission for user: `%s` in repo: `%s`' % (
                 userid, repoid
@@ -1674,9 +1907,17 b' def grant_user_group_permission(request,'
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
+    perm_additions = [[user_group.users_group_id, perm.permission_name, "user_group"]]
     try:
-        RepoModel().grant_user_group_permission(
-            repo=repo, group_name=user_group, perm=perm)
+        changes = RepoModel().update_permissions(
+            repo=repo, perm_additions=perm_additions, cur_user=apiuser)
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo.edit.permissions', action_data=action_data, user=apiuser, repo=repo)
 
         Session().commit()
         return {
@@ -1739,9 +1980,17 b' def revoke_user_group_permission(request'
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
+    perm_deletions = [[user_group.users_group_id, None, "user_group"]]
     try:
-        RepoModel().revoke_user_group_permission(
-            repo=repo, group_name=user_group)
+        changes = RepoModel().update_permissions(
+            repo=repo, perm_deletions=perm_deletions, cur_user=apiuser)
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo.edit.permissions', action_data=action_data, user=apiuser, repo=repo)
 
         Session().commit()
         return {

rhodecode/api/views/repo_group_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -453,10 +453,19 b' def grant_user_permission_to_repo_group('
 
     perm_additions = [[user.user_id, perm.permission_name, "user"]]
     try:
-        RepoGroupModel().update_permissions(repo_group=repo_group,
-                                            perm_additions=perm_additions,
-                                            recursive=apply_to_children,
-                                            cur_user=apiuser)
+        changes = RepoGroupModel().update_permissions(
+            repo_group=repo_group, perm_additions=perm_additions,
+            recursive=apply_to_children, cur_user=apiuser)
+
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
         return {
             'msg': 'Granted perm: `%s` (recursive:%s) for user: '
@@ -527,10 +536,19 b' def revoke_user_permission_from_repo_gro'
 
     perm_deletions = [[user.user_id, None, "user"]]
     try:
-        RepoGroupModel().update_permissions(repo_group=repo_group,
-                                            perm_deletions=perm_deletions,
-                                            recursive=apply_to_children,
-                                            cur_user=apiuser)
+        changes = RepoGroupModel().update_permissions(
+            repo_group=repo_group, perm_deletions=perm_deletions,
+            recursive=apply_to_children, cur_user=apiuser)
+
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
         return {
             'msg': 'Revoked perm (recursive:%s) for user: '
@@ -611,10 +629,19 b' def grant_user_group_permission_to_repo_'
 
     perm_additions = [[user_group.users_group_id, perm.permission_name, "user_group"]]
     try:
-        RepoGroupModel().update_permissions(repo_group=repo_group,
-                                            perm_additions=perm_additions,
-                                            recursive=apply_to_children,
-                                            cur_user=apiuser)
+        changes = RepoGroupModel().update_permissions(
+            repo_group=repo_group, perm_additions=perm_additions,
+            recursive=apply_to_children, cur_user=apiuser)
+
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
         return {
             'msg': 'Granted perm: `%s` (recursive:%s) '
@@ -694,10 +721,19 b' def revoke_user_group_permission_from_re'
 
     perm_deletions = [[user_group.users_group_id, None, "user_group"]]
     try:
-        RepoGroupModel().update_permissions(repo_group=repo_group,
-                                            perm_deletions=perm_deletions,
-                                            recursive=apply_to_children,
-                                            cur_user=apiuser)
+        changes = RepoGroupModel().update_permissions(
+            repo_group=repo_group, perm_deletions=perm_deletions,
+            recursive=apply_to_children, cur_user=apiuser)
+
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'repo_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
         return {
             'msg': 'Revoked perm (recursive:%s) for user group: '
@@ -716,4 +752,3 b' def revoke_user_group_permission_from_re'
                 user_group.users_group_name, repo_group.name
             )
         )
-

rhodecode/api/views/server_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -21,6 +21,9 b''
 import inspect
 import logging
 import itertools
+import base64
+
+from pyramid import compat
 
 from rhodecode.api import (
     jsonrpc_method, JSONRPCError, JSONRPCForbidden, find_methods)
@@ -30,10 +33,15 b' from rhodecode.api.utils import ('
 from rhodecode.lib.utils import repo2db_mapper
 from rhodecode.lib import system_info
 from rhodecode.lib import user_sessions
+from rhodecode.lib import exc_tracking
+from rhodecode.lib.ext_json import json
 from rhodecode.lib.utils2 import safe_int
 from rhodecode.model.db import UserIpMap
 from rhodecode.model.scm import ScmModel
 from rhodecode.model.settings import VcsSettingsModel
+from rhodecode.apps.file_store import utils
+from rhodecode.apps.file_store.exceptions import FileNotAllowedException, \
+    FileOverSizeException
 
 log = logging.getLogger(__name__)
 
@@ -293,7 +301,7 b' def get_method(request, apiuser, pattern'
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param pattern: pattern to match method names against
-    :type older_then: Optional("*")
+    :type pattern: Optional("*")
 
     Example output:
 
@@ -349,3 +357,63 b' def get_method(request, apiuser, pattern'
         args_desc.append(func_kwargs)
 
     return matches.keys() + args_desc
+
+
+@jsonrpc_method()
+def store_exception(request, apiuser, exc_data_json, prefix=Optional('rhodecode')):
+    """
+    Stores sent exception inside the built-in exception tracker in |RCE| server.
+
+    This command can only be run using an |authtoken| with admin rights to
+    the specified repository.
+
+    This command takes the following options:
+
+    :param apiuser: This is filled automatically from the |authtoken|.
+    :type apiuser: AuthUser
+
+    :param exc_data_json: JSON data with exception e.g
+        {"exc_traceback": "Value `1` is not allowed", "exc_type_name": "ValueError"}
+    :type exc_data_json: JSON data
+
+    :param prefix: prefix for error type, e.g 'rhodecode', 'vcsserver', 'rhodecode-tools'
+    :type prefix: Optional("rhodecode")
+
+    Example output:
+
+    .. code-block:: bash
+
+      id : <id_given_in_input>
+      "result": {
+        "exc_id": 139718459226384,
+        "exc_url": "http://localhost:8080/_admin/settings/exceptions/139718459226384"
+      }
+      error :  null
+    """
+    if not has_superadmin_permission(apiuser):
+        raise JSONRPCForbidden()
+
+    prefix = Optional.extract(prefix)
+    exc_id = exc_tracking.generate_id()
+
+    try:
+        exc_data = json.loads(exc_data_json)
+    except Exception:
+        log.error('Failed to parse JSON: %r', exc_data_json)
+        raise JSONRPCError('Failed to parse JSON data from exc_data_json field. '
+                           'Please make sure it contains a valid JSON.')
+
+    try:
+        exc_traceback = exc_data['exc_traceback']
+        exc_type_name = exc_data['exc_type_name']
+    except KeyError as err:
+        raise JSONRPCError('Missing exc_traceback, or exc_type_name '
+                           'in exc_data_json field. Missing: {}'.format(err))
+
+    exc_tracking._store_exception(
+        exc_id=exc_id, exc_traceback=exc_traceback,
+        exc_type_name=exc_type_name, prefix=prefix)
+
+    exc_url = request.route_url(
+        'admin_settings_exception_tracker_show', exception_id=exc_id)
+    return {'exc_id': exc_id, 'exc_url': exc_url}

rhodecode/api/views/testing_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/api/views/user_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -19,6 +19,7 b''
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
+from pyramid import compat
 
 from rhodecode.api import (
     jsonrpc_method, JSONRPCError, JSONRPCForbidden, JSONRPCValidationError)
@@ -241,7 +242,7 b' def create_user(request, apiuser, userna'
         # generate temporary password if user is external
         password = PasswordGenerator().gen_password(length=16)
     create_repo_group = Optional.extract(create_personal_repo_group)
-    if isinstance(create_repo_group, basestring):
+    if isinstance(create_repo_group, compat.string_types):
         create_repo_group = str2bool(create_repo_group)
 
     username = Optional.extract(username)

rhodecode/api/views/user_group_api.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/_base/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -22,11 +22,12 b' import time'
 import logging
 import operator
 
+from pyramid import compat
 from pyramid.httpexceptions import HTTPFound, HTTPForbidden, HTTPBadRequest
 
 from rhodecode.lib import helpers as h, diffs
 from rhodecode.lib.utils2 import (
-    StrictAttributeDict, safe_int, datetime_to_time, safe_unicode)
+    StrictAttributeDict, str2bool, safe_int, datetime_to_time, safe_unicode)
 from rhodecode.lib.vcs.exceptions import RepositoryRequirementError
 from rhodecode.model import repo
 from rhodecode.model import repo_group
@@ -249,6 +250,12 b' class RepoAppView(BaseAppView):'
             else:  # redirect if we don't show missing requirements
                 raise HTTPFound(h.route_path('home'))
 
+        c.has_origin_repo_read_perm = False
+        if self.db_repo.fork:
+            c.has_origin_repo_read_perm = h.HasRepoPermissionAny(
+                'repository.write', 'repository.read', 'repository.admin')(
+                self.db_repo.fork.repo_name, 'summary fork link')
+
         return c
 
     def _get_f_path_unchecked(self, matchdict, default=None):
@@ -271,6 +278,13 b' class RepoAppView(BaseAppView):'
         settings = settings_model.get_general_settings()
         return settings.get(settings_key, default)
 
+    def get_recache_flag(self):
+        for flag_name in ['force_recache', 'force-recache', 'no-cache']:
+            flag_val = self.request.GET.get(flag_name)
+            if str2bool(flag_val):
+                return True
+        return False
+
 
 class PathFilter(object):
 
@@ -327,6 +341,13 b' class RepoGroupAppView(BaseAppView):'
         self.db_repo_group = request.db_repo_group
         self.db_repo_group_name = self.db_repo_group.group_name
 
+    def _get_local_tmpl_context(self, include_app_defaults=True):
+        _ = self.request.translate
+        c = super(RepoGroupAppView, self)._get_local_tmpl_context(
+            include_app_defaults=include_app_defaults)
+        c.repo_group = self.db_repo_group
+        return c
+
     def _revoke_perms_on_yourself(self, form_result):
         _updates = filter(lambda u: self._rhodecode_user.user_id == int(u[0]),
                           form_result['perm_updates'])
@@ -389,7 +410,7 b' class DataGridAppView(object):'
         return draw, start, length
 
     def _get_order_col(self, order_by, model):
-        if isinstance(order_by, basestring):
+        if isinstance(order_by, compat.string_types):
             try:
                 return operator.attrgetter(order_by)(model)
             except AttributeError:

rhodecode/apps/_base/interfaces.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/_base/navigation.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/_base/subscribers.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -280,8 +280,12 b' def admin_routes(config):'
         pattern='/users/{user_id:\d+}/delete',
         user_route=True)
     config.add_route(
-        name='user_force_password_reset',
-        pattern='/users/{user_id:\d+}/password_reset',
+        name='user_enable_force_password_reset',
+        pattern='/users/{user_id:\d+}/password_reset_enable',
+        user_route=True)
+    config.add_route(
+        name='user_disable_force_password_reset',
+        pattern='/users/{user_id:\d+}/password_reset_disable',
         user_route=True)
     config.add_route(
         name='user_create_personal_repo_group',

rhodecode/apps/admin/tests/test_admin_audit_logs.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_auth_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_defaults.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_main_views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_repos.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_repository_groups.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_user_groups.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/tests/test_admin_users.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -59,8 +59,6 b' def route_path(name, params=None, **kwar'
             ADMIN_PREFIX + '/users/{user_id}/update',
         'user_delete':
             ADMIN_PREFIX + '/users/{user_id}/delete',
-        'user_force_password_reset':
-            ADMIN_PREFIX + '/users/{user_id}/password_reset',
         'user_create_personal_repo_group':
             ADMIN_PREFIX + '/users/{user_id}/create_repo_group',
 

rhodecode/apps/admin/tests/test_admin_users_ssh_keys.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/audit_logs.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/defaults.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/exception_tracker.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2018-2018 RhodeCode GmbH
+# Copyright (C) 2018-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/main_views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/open_source_licenses.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/process_management.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -119,8 +119,7 b' class AdminProcessManagementView(BaseApp'
         result = []
 
         def on_terminate(proc):
-            msg = "process `PID:{}` terminated with exit code {}".format(
-                proc.pid, proc.returncode or 0)
+            msg = "terminated"
             result.append(msg)
 
         procs = []

rhodecode/apps/admin/views/repo_groups.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/repositories.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/sessions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -666,8 +666,8 b' class AdminSettingsView(BaseAppView):'
         c = self.load_default_context()
         c.active = 'search'
 
-        searcher = searcher_from_config(self.request.registry.settings)
-        c.statistics = searcher.statistics(self.request.translate)
+        c.searcher = searcher_from_config(self.request.registry.settings)
+        c.statistics = c.searcher.statistics(self.request.translate)
 
         return self._get_template_context(c)
 

rhodecode/apps/admin/views/svn_config.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/system_info.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/user_groups.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/admin/views/users.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -604,12 +604,9 b' class UsersView(UserAppView):'
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
-        route_name='user_force_password_reset', request_method='POST',
+        route_name='user_enable_force_password_reset', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
-    def user_force_password_reset(self):
-        """
-        toggle reset password flag for this user
-        """
+    def user_enable_force_password_reset(self):
         _ = self.request.translate
         c = self.load_default_context()
 
@@ -617,19 +614,41 b' class UsersView(UserAppView):'
         c.user = self.db_user
 
         try:
-            old_value = c.user.user_data.get('force_password_change')
-            c.user.update_userdata(force_password_change=not old_value)
+            c.user.update_userdata(force_password_change=True)
+
+            msg = _('Force password change enabled for user')
+            audit_logger.store_web('user.edit.password_reset.enabled',
+                                   user=c.rhodecode_user)
+
+            Session().commit()
+            h.flash(msg, category='success')
+        except Exception:
+            log.exception("Exception during password reset for user")
+            h.flash(_('An error occurred during password reset for user'),
+                    category='error')
+
+        raise HTTPFound(h.route_path('user_edit_advanced', user_id=user_id))
 
-            if old_value:
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @CSRFRequired()
+    @view_config(
+        route_name='user_disable_force_password_reset', request_method='POST',
+        renderer='rhodecode:templates/admin/users/user_edit.mako')
+    def user_disable_force_password_reset(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        user_id = self.db_user_id
+        c.user = self.db_user
+
+        try:
+            c.user.update_userdata(force_password_change=False)
+
                 msg = _('Force password change disabled for user')
                 audit_logger.store_web(
                     'user.edit.password_reset.disabled',
                     user=c.rhodecode_user)
-            else:
-                msg = _('Force password change enabled for user')
-                audit_logger.store_web(
-                    'user.edit.password_reset.enabled',
-                    user=c.rhodecode_user)
 
             Session().commit()
             h.flash(msg, category='success')

rhodecode/apps/channelstream/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/channelstream/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/debug_style/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/debug_style/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/gist/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/gist/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/gist/tests/test_admin_gists.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/gist/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2013-2018 RhodeCode GmbH
+# Copyright (C) 2013-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/home/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -56,6 +56,10 b' def includeme(config):'
         pattern='/_repos')
 
     config.add_route(
+        name='repo_group_list_data',
+        pattern='/_repo_groups')
+
+    config.add_route(
         name='goto_switcher_data',
         pattern='/_goto_data')
 

rhodecode/apps/home/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -31,7 +31,14 b' def assert_and_get_main_filter_content(r'
         assert data_item['url']
 
         if data_item['type'] == 'search':
-            assert data_item['value_display'].startswith('Full text search for:')
+            display_val = data_item['value_display']
+            if data_item['id'] == -1:
+                assert 'File search for:' in display_val, display_val
+            elif data_item['id'] == -2:
+                assert 'Commit search for:' in display_val, display_val
+            else:
+                assert False, 'No Proper ID returned {}'.format(data_item['id'])
+
         elif data_item['type'] == 'repo':
             repos.append(data_item)
         elif data_item['type'] == 'repo_group':

rhodecode/apps/home/tests/test_get_goto_switched_data.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/home/tests/test_get_repo_list_data.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/home/tests/test_get_user_data.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/home/tests/test_get_user_group_data.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -19,7 +19,7 b''
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/home/tests/test_home.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/home/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -172,7 +172,9 b' class HomeView(BaseAppView):'
                 'id': obj.group_name,
                 'value': org_query,
                 'value_display': obj.group_name,
+                'text': obj.group_name,
                 'type': 'repo_group',
+                'repo_group_id': obj.group_id,
                 'url': h.route_path(
                     'repo_group_home', repo_group_name=obj.group_name)
             }
@@ -246,9 +248,9 b' class HomeView(BaseAppView):'
             }
             for obj in acl_iter]
 
-    def _get_hash_commit_list(self, auth_user, query):
+    def _get_hash_commit_list(self, auth_user, searcher, query):
         org_query = query
-        if not query or len(query) < 3:
+        if not query or len(query) < 3 or not searcher:
             return []
 
         commit_hashes = re.compile('(?:commit:)([0-9a-f]{2,40})').findall(query)
@@ -257,24 +259,34 b' class HomeView(BaseAppView):'
             return []
         commit_hash = commit_hashes[0]
 
-        searcher = searcher_from_config(self.request.registry.settings)
         result = searcher.search(
-            'commit_id:%s*' % commit_hash, 'commit', auth_user,
+            'commit_id:{}*'.format(commit_hash), 'commit', auth_user,
             raise_on_exc=False)
 
-        return [
-            {
+        commits = []
+        for entry in result['results']:
+            repo_data = {
+                'repository_id': entry.get('repository_id'),
+                'repository_type': entry.get('repo_type'),
+                'repository_name': entry.get('repository'),
+            }
+
+            commit_entry = {
                 'id': entry['commit_id'],
                 'value': org_query,
-                'value_display': 'repo `{}` commit: {}'.format(
+                'value_display': '`{}` commit: {}'.format(
                     entry['repository'], entry['commit_id']),
                 'type': 'commit',
                 'repo': entry['repository'],
+                'repo_data': repo_data,
+
                 'url': h.route_path(
                     'repo_commit',
                     repo_name=entry['repository'], commit_id=entry['commit_id'])
             }
-            for entry in result['results']]
+
+            commits.append(commit_entry)
+        return commits
 
     @LoginRequired()
     @view_config(
@@ -305,6 +317,144 b' class HomeView(BaseAppView):'
 
     @LoginRequired()
     @view_config(
+        route_name='repo_group_list_data', request_method='GET',
+        renderer='json_ext', xhr=True)
+    def repo_group_list_data(self):
+        _ = self.request.translate
+        self.load_default_context()
+
+        query = self.request.GET.get('query')
+
+        log.debug('generating repo group list, query:%s',
+                  query)
+
+        res = []
+        repo_groups = self._get_repo_group_list(query)
+        if repo_groups:
+            res.append({
+                'text': _('Repository Groups'),
+                'children': repo_groups
+            })
+
+        data = {
+            'more': False,
+            'results': res
+        }
+        return data
+
+    def _get_default_search_queries(self, search_context, searcher, query):
+        if not searcher:
+            return []
+
+        is_es_6 = searcher.is_es_6
+
+        queries = []
+        repo_group_name, repo_name, repo_context = None, None, None
+
+        # repo group context
+        if search_context.get('search_context[repo_group_name]'):
+            repo_group_name = search_context.get('search_context[repo_group_name]')
+        if search_context.get('search_context[repo_name]'):
+            repo_name = search_context.get('search_context[repo_name]')
+            repo_context = search_context.get('search_context[repo_view_type]')
+
+        if is_es_6 and repo_name:
+            # files
+            def query_modifier():
+                qry = query
+                return {'q': qry, 'type': 'content'}
+            label = u'File search for `{}` in this repository.'.format(query)
+            queries.append(
+                {
+                    'id': -10,
+                    'value': query,
+                    'value_display': label,
+                    'type': 'search',
+                    'url': h.route_path('search_repo',
+                                        repo_name=repo_name,
+                                        _query=query_modifier())
+                }
+            )
+
+            # commits
+            def query_modifier():
+                qry = query
+                return {'q': qry, 'type': 'commit'}
+
+            label = u'Commit search for `{}` in this repository.'.format(query)
+            queries.append(
+                {
+                    'id': -20,
+                    'value': query,
+                    'value_display': label,
+                    'type': 'search',
+                    'url': h.route_path('search_repo',
+                                        repo_name=repo_name,
+                                        _query=query_modifier())
+                }
+            )
+
+        elif is_es_6 and repo_group_name:
+            # files
+            def query_modifier():
+                qry = query
+                return {'q': qry, 'type': 'content'}
+
+            label = u'File search for `{}` in this repository group'.format(query)
+            queries.append(
+                {
+                    'id': -30,
+                    'value': query,
+                    'value_display': label,
+                    'type': 'search',
+                    'url': h.route_path('search_repo_group',
+                                        repo_group_name=repo_group_name,
+                                        _query=query_modifier())
+                }
+            )
+
+            # commits
+            def query_modifier():
+                qry = query
+                return {'q': qry, 'type': 'commit'}
+
+            label = u'Commit search for `{}` in this repository group'.format(query)
+            queries.append(
+                {
+                    'id': -40,
+                    'value': query,
+                    'value_display': label,
+                    'type': 'search',
+                    'url': h.route_path('search_repo_group',
+                                        repo_group_name=repo_group_name,
+                                        _query=query_modifier())
+                }
+            )
+
+        if not queries:
+            queries.append(
+                {
+                    'id': -1,
+                    'value': query,
+                    'value_display': u'File search for: `{}`'.format(query),
+                    'type': 'search',
+                    'url': h.route_path('search',
+                                        _query={'q': query, 'type': 'content'})
+                })
+            queries.append(
+                {
+                    'id': -2,
+                    'value': query,
+                    'value_display': u'Commit search for: `{}`'.format(query),
+                    'type': 'search',
+                    'url': h.route_path('search',
+                                        _query={'q': query, 'type': 'commit'})
+                })
+
+        return queries
+
+    @LoginRequired()
+    @view_config(
         route_name='goto_switcher_data', request_method='GET',
         renderer='json_ext', xhr=True)
     def goto_switcher_data(self):
@@ -315,26 +465,21 b' class HomeView(BaseAppView):'
         query = self.request.GET.get('query')
         log.debug('generating main filter data, query %s', query)
 
-        default_search_val = u'Full text search for: `{}`'.format(query)
         res = []
         if not query:
             return {'suggestions': res}
 
-        res.append({
-            'id': -1,
-            'value': query,
-            'value_display': default_search_val,
-            'type': 'search',
-            'url': h.route_path(
-                'search', _query={'q': query})
-        })
-        repo_group_id = safe_int(self.request.GET.get('repo_group_id'))
+        searcher = searcher_from_config(self.request.registry.settings)
+        for _q in self._get_default_search_queries(self.request.GET, searcher, query):
+            res.append(_q)
+
+        repo_group_id = safe_int(self.request.GET.get('search_context[repo_group_id]'))
         if repo_group_id:
             repo_group = RepoGroup.get(repo_group_id)
             composed_hint = '{}/{}'.format(repo_group.group_name, query)
             show_hint = not query.startswith(repo_group.group_name)
             if repo_group and show_hint:
-                hint = u'Group search: `{}`'.format(composed_hint)
+                hint = u'Repository search inside: `{}`'.format(composed_hint)
                 res.append({
                     'id': -1,
                     'value': composed_hint,
@@ -351,7 +496,7 b' class HomeView(BaseAppView):'
         for serialized_repo in repos:
             res.append(serialized_repo)
 
-        # TODO(marcink): permissions for that ?
+        # TODO(marcink): should all logged in users be allowed to search others?
         allowed_user_search = self._rhodecode_user.username != User.DEFAULT_USER
         if allowed_user_search:
             users = self._get_user_list(query)
@@ -362,7 +507,7 b' class HomeView(BaseAppView):'
             for serialized_user_group in user_groups:
                 res.append(serialized_user_group)
 
-        commits = self._get_hash_commit_list(c.auth_user, query)
+        commits = self._get_hash_commit_list(c.auth_user, searcher, query)
         if commits:
             unique_repos = collections.OrderedDict()
             for commit in commits:

rhodecode/apps/journal/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/journal/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/journal/tests/test_journal.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/journal/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/login/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/login/tests/test_login.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -107,6 +107,26 b' class TestLoginController(object):'
 
         response.mustcontain('/%s' % HG_REPO)
 
+    def test_login_regular_forbidden_when_super_admin_restriction(self):
+        from rhodecode.authentication.plugins.auth_rhodecode import RhodeCodeAuthPlugin
+        with fixture.auth_restriction(RhodeCodeAuthPlugin.AUTH_RESTRICTION_SUPER_ADMIN):
+            response = self.app.post(route_path('login'),
+                                     {'username': 'test_regular',
+                                      'password': 'test12'})
+
+            response.mustcontain('invalid user name')
+            response.mustcontain('invalid password')
+
+    def test_login_regular_forbidden_when_scope_restriction(self):
+        from rhodecode.authentication.plugins.auth_rhodecode import RhodeCodeAuthPlugin
+        with fixture.scope_restriction(RhodeCodeAuthPlugin.AUTH_RESTRICTION_SCOPE_VCS):
+            response = self.app.post(route_path('login'),
+                                     {'username': 'test_regular',
+                                      'password': 'test12'})
+
+            response.mustcontain('invalid user name')
+            response.mustcontain('invalid password')
+
     def test_login_ok_came_from(self):
         test_came_from = '/_admin/users?branch=stable'
         _url = '{}?came_from={}'.format(route_path('login'), test_came_from)

rhodecode/apps/login/tests/test_password_reset.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -74,20 +74,17 b' class TestPasswordReset(TestController):'
             'default_password_reset': pwd_reset_setting,
             'default_extern_activate': 'hg.extern_activate.auto',
         }
-        resp = self.app.post(route_path('admin_permissions_application_update'), params=params)
+        resp = self.app.post(
+            route_path('admin_permissions_application_update'), params=params)
         self.logout_user()
 
         login_page = self.app.get(route_path('login'))
         asr_login = AssertResponse(login_page)
-        index_page = self.app.get(h.route_path('home'))
-        asr_index = AssertResponse(index_page)
 
         if show_link:
             asr_login.one_element_exists('a.pwd_reset')
-            asr_index.one_element_exists('a.pwd_reset')
         else:
             asr_login.no_element_exists('a.pwd_reset')
-            asr_index.no_element_exists('a.pwd_reset')
 
         response = self.app.get(route_path('reset_password'))
         

rhodecode/apps/login/tests/test_register_captcha.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/login/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -95,6 +95,18 b' def includeme(config):'
         pattern=ADMIN_PREFIX + '/my_account/watched')
 
     config.add_route(
+        name='my_account_bookmarks',
+        pattern=ADMIN_PREFIX + '/my_account/bookmarks')
+
+    config.add_route(
+        name='my_account_bookmarks_update',
+        pattern=ADMIN_PREFIX + '/my_account/bookmarks/update')
+
+    config.add_route(
+        name='my_account_goto_bookmark',
+        pattern=ADMIN_PREFIX + '/my_account/bookmark/{bookmark_id}')
+
+    config.add_route(
         name='my_account_perms',
         pattern=ADMIN_PREFIX + '/my_account/perms')
 

rhodecode/apps/my_account/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_auth_tokens.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_edit.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -19,7 +19,7 b''
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_emails.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_notifications.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_password.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_profile.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_simple_views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/tests/test_my_account_ssh_keys.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/views/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/views/my_account.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -20,29 +20,30 b''
 
 import logging
 import datetime
+import string
 
 import formencode
 import formencode.htmlfill
+import peppercorn
 from pyramid.httpexceptions import HTTPFound
 from pyramid.view import view_config
-from pyramid.renderers import render
-from pyramid.response import Response
 
 from rhodecode.apps._base import BaseAppView, DataGridAppView
 from rhodecode import forms
 from rhodecode.lib import helpers as h
 from rhodecode.lib import audit_logger
 from rhodecode.lib.ext_json import json
-from rhodecode.lib.auth import LoginRequired, NotAnonymous, CSRFRequired
+from rhodecode.lib.auth import LoginRequired, NotAnonymous, CSRFRequired, \
+    HasRepoPermissionAny, HasRepoGroupPermissionAny
 from rhodecode.lib.channelstream import (
     channelstream_request, ChannelstreamException)
 from rhodecode.lib.utils2 import safe_int, md5, str2bool
 from rhodecode.model.auth_token import AuthTokenModel
 from rhodecode.model.comment import CommentsModel
 from rhodecode.model.db import (
-    Repository, UserEmailMap, UserApiKeys, UserFollowing, joinedload,
-    PullRequest)
-from rhodecode.model.forms import UserForm, UserExtraEmailForm
+    IntegrityError, joinedload,
+    Repository, UserEmailMap, UserApiKeys, UserFollowing,
+    PullRequest, UserBookmark, RepoGroup)
 from rhodecode.model.meta import Session
 from rhodecode.model.pull_request import PullRequestModel
 from rhodecode.model.scm import RepoList
@@ -392,6 +393,140 b' class MyAccountView(BaseAppView, DataGri'
     @LoginRequired()
     @NotAnonymous()
     @view_config(
+        route_name='my_account_bookmarks', request_method='GET',
+        renderer='rhodecode:templates/admin/my_account/my_account.mako')
+    def my_account_bookmarks(self):
+        c = self.load_default_context()
+        c.active = 'bookmarks'
+        return self._get_template_context(c)
+
+    def _process_entry(self, entry, user_id):
+        position = safe_int(entry.get('position'))
+        if position is None:
+            return
+
+        # check if this is an existing entry
+        is_new = False
+        db_entry = UserBookmark().get_by_position_for_user(position, user_id)
+
+        if db_entry and str2bool(entry.get('remove')):
+            log.debug('Marked bookmark %s for deletion', db_entry)
+            Session().delete(db_entry)
+            return
+
+        if not db_entry:
+            # new
+            db_entry = UserBookmark()
+            is_new = True
+
+        should_save = False
+        default_redirect_url = ''
+
+        # save repo
+        if entry.get('bookmark_repo'):
+            repo = Repository.get(entry['bookmark_repo'])
+            perm_check = HasRepoPermissionAny(
+                'repository.read', 'repository.write', 'repository.admin')
+            if repo and perm_check(repo_name=repo.repo_name):
+                db_entry.repository = repo
+                should_save = True
+                default_redirect_url = '${repo_url}'
+        # save repo group
+        elif entry.get('bookmark_repo_group'):
+            repo_group = RepoGroup.get(entry['bookmark_repo_group'])
+            perm_check = HasRepoGroupPermissionAny(
+                'group.read', 'group.write', 'group.admin')
+
+            if repo_group and perm_check(group_name=repo_group.group_name):
+                db_entry.repository_group = repo_group
+                should_save = True
+                default_redirect_url = '${repo_group_url}'
+        # save generic info
+        elif entry.get('title') and entry.get('redirect_url'):
+            should_save = True
+
+        if should_save:
+            log.debug('Saving bookmark %s, new:%s', db_entry, is_new)
+            # mark user and position
+            db_entry.user_id = user_id
+            db_entry.position = position
+            db_entry.title = entry.get('title')
+            db_entry.redirect_url = entry.get('redirect_url') or default_redirect_url
+
+            Session().add(db_entry)
+
+    @LoginRequired()
+    @NotAnonymous()
+    @CSRFRequired()
+    @view_config(
+        route_name='my_account_bookmarks_update', request_method='POST')
+    def my_account_bookmarks_update(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+        c.active = 'bookmarks'
+
+        controls = peppercorn.parse(self.request.POST.items())
+        user_id = c.user.user_id
+
+        try:
+            for entry in controls.get('bookmarks', []):
+                self._process_entry(entry, user_id)
+
+            Session().commit()
+            h.flash(_("Update Bookmarks"), category='success')
+        except IntegrityError:
+            h.flash(_("Failed to update bookmarks. "
+                      "Make sure an unique position is used"), category='error')
+
+        return HTTPFound(h.route_path('my_account_bookmarks'))
+
+    @LoginRequired()
+    @NotAnonymous()
+    @view_config(
+        route_name='my_account_goto_bookmark', request_method='GET',
+        renderer='rhodecode:templates/admin/my_account/my_account.mako')
+    def my_account_goto_bookmark(self):
+
+        bookmark_id = self.request.matchdict['bookmark_id']
+        user_bookmark = UserBookmark().query()\
+            .filter(UserBookmark.user_id == self.request.user.user_id) \
+            .filter(UserBookmark.position == bookmark_id).scalar()
+
+        redirect_url = h.route_path('my_account_bookmarks')
+        if not user_bookmark:
+            raise HTTPFound(redirect_url)
+
+        if user_bookmark.repository:
+            repo_name = user_bookmark.repository.repo_name
+            base_redirect_url = h.route_path(
+                'repo_summary', repo_name=repo_name)
+            if user_bookmark.redirect_url and \
+                    '${repo_url}' in user_bookmark.redirect_url:
+                redirect_url = string.Template(user_bookmark.redirect_url)\
+                    .safe_substitute({'repo_url': base_redirect_url})
+            else:
+                redirect_url = base_redirect_url
+
+        elif user_bookmark.repository_group:
+            repo_group_name = user_bookmark.repository_group.group_name
+            base_redirect_url = h.route_path(
+                'repo_group_home', repo_group_name=repo_group_name)
+            if user_bookmark.redirect_url and \
+                    '${repo_group_url}' in user_bookmark.redirect_url:
+                redirect_url = string.Template(user_bookmark.redirect_url)\
+                    .safe_substitute({'repo_group_url': base_redirect_url})
+            else:
+                redirect_url = base_redirect_url
+
+        elif user_bookmark.redirect_url:
+            redirect_url = user_bookmark.redirect_url
+
+        log.debug('Redirecting bookmark %s to %s', user_bookmark, redirect_url)
+        raise HTTPFound(redirect_url)
+
+    @LoginRequired()
+    @NotAnonymous()
+    @view_config(
         route_name='my_account_perms', request_method='GET',
         renderer='rhodecode:templates/admin/my_account/my_account.mako')
     def my_account_perms(self):

rhodecode/apps/my_account/views/my_account_notifications.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/my_account/views/my_account_ssh_keys.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -71,6 +71,7 b' class MyAccountSshKeysView(BaseAppView, '
         c = self.load_default_context()
 
         c.active = 'ssh_keys_generate'
+        if c.ssh_key_generator_enabled:
         comment = 'RhodeCode-SSH {}'.format(c.user.email or '')
         c.private, c.public = SshKeyModel().generate_keypair(comment=comment)
         c.target_form_url = h.route_path(

rhodecode/apps/ops/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ops/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -49,9 +49,13 b' class OpsView(BaseAppView):'
             'instance': self.request.registry.settings.get('instance_id'),
         }
         if getattr(self.request, 'user'):
+            caller_name = 'anonymous'
+            if self.request.user.user_id:
+                caller_name = self.request.user.username
+
             data.update({
                 'caller_ip': self.request.user.ip_addr,
-                'caller_name': self.request.user.username,
+                'caller_name': caller_name,
             })
         return {'ok': data}
 
@@ -65,11 +69,13 b' class OpsView(BaseAppView):'
         """
         Test exception handling and emails on errors
         """
+
         class TestException(Exception):
             pass
-
+        # add timeout so we add some sort of rate limiter
+        time.sleep(2)
         msg = ('RhodeCode Enterprise test exception. '
-               'Generation time: {}'.format(time.time()))
+               'Client:{}. Generation time: {}.'.format(self.request.user, time.time()))
         raise TestException(msg)
 
     @view_config(

rhodecode/apps/repo_group/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/tests/test_repo_groups_advanced.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/tests/test_repo_groups_permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/tests/test_repo_groups_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/views/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/views/repo_group_advanced.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/views/repo_group_permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repo_group/views/repo_group_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -271,8 +271,8 b' def includeme(config):'
         repo_route=True)
 
     config.add_route(
-        name='pullrequest_repo_destinations',
-        pattern='/{repo_name:.*?[^/]}/pull-request/repo-destinations',
+        name='pullrequest_repo_targets',
+        pattern='/{repo_name:.*?[^/]}/pull-request/repo-targets',
         repo_route=True)
 
     config.add_route(

rhodecode/apps/repository/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_pull_requests_list.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_bookmarks.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_branches.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_changelog.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_commit_comments.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_commits.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_compare.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_compare_local.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_compare_on_single_file.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_feed.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_files.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_forks.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_issue_tracker.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_pullrequests.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -32,7 +32,6 b' from rhodecode.model.pull_request import'
 from rhodecode.model.user import UserModel
 from rhodecode.tests import (
     assert_session_flash, TEST_USER_ADMIN_LOGIN, TEST_USER_REGULAR_LOGIN)
-from rhodecode.tests.utils import AssertResponse
 
 
 def route_path(name, params=None, **kwargs):
@@ -45,7 +44,7 b' def route_path(name, params=None, **kwar'
         'pullrequest_show_all': '/{repo_name}/pull-request',
         'pullrequest_show_all_data': '/{repo_name}/pull-request-data',
         'pullrequest_repo_refs': '/{repo_name}/pull-request/refs/{target_repo_name:.*?[^/]}',
-        'pullrequest_repo_destinations': '/{repo_name}/pull-request/repo-destinations',
+        'pullrequest_repo_targets': '/{repo_name}/pull-request/repo-destinations',
         'pullrequest_new': '/{repo_name}/pull-request/new',
         'pullrequest_create': '/{repo_name}/pull-request/create',
         'pullrequest_update': '/{repo_name}/pull-request/{pull_request_id}/update',
@@ -233,8 +232,7 b' class TestPullrequestsView(object):'
             route_path('pullrequest_update',
                 repo_name=pull_request.target_repo.repo_name,
                 pull_request_id=pull_request_id),
-            params={'update_commits': 'true',
-                    'csrf_token': csrf_token})
+            params={'update_commits': 'true', 'csrf_token': csrf_token})
 
         expected_msg = str(PullRequestModel.UPDATE_STATUS_MESSAGES[
             UpdateFailureReason.MISSING_SOURCE_REF])
@@ -244,7 +242,8 b' class TestPullrequestsView(object):'
         from rhodecode.lib.vcs.backends.base import MergeFailureReason
         pull_request = pr_util.create_pull_request(
             approved=True, mergeable=True)
-        pull_request.target_ref = 'branch:invalid-branch:invalid-commit-id'
+        unicode_reference = u'branch:invalid-branch:invalid-commit-id'
+        pull_request.target_ref = unicode_reference
         Session().add(pull_request)
         Session().commit()
 
@@ -255,12 +254,12 b' class TestPullrequestsView(object):'
             pull_request_id=pull_request_id)
 
         response = self.app.get(pull_request_url)
-
-        assertr = AssertResponse(response)
-        expected_msg = PullRequestModel.MERGE_STATUS_MESSAGES[
-            MergeFailureReason.MISSING_TARGET_REF]
-        assertr.element_contains(
-            'span[data-role="merge-message"]', str(expected_msg))
+        target_ref_id = 'invalid-branch'
+        merge_resp = MergeResponse(
+            True, True, '', MergeFailureReason.MISSING_TARGET_REF,
+            metadata={'target_ref': PullRequest.unicode_to_reference(unicode_reference)})
+        response.assert_response().element_contains(
+            'span[data-role="merge-message"]', merge_resp.merge_status_message)
 
     def test_comment_and_close_pull_request_custom_message_approved(
             self, pr_util, csrf_token, xhr_header):
@@ -608,8 +607,7 b' class TestPullrequestsView(object):'
 
         response = self.app.post(
             route_path('pullrequest_merge',
-                repo_name=repo_name,
-                pull_request_id=pull_request_id),
+                       repo_name=repo_name, pull_request_id=pull_request_id),
             params={'csrf_token': csrf_token}).follow()
 
         assert response.status_int == 200
@@ -624,10 +622,13 b' class TestPullrequestsView(object):'
         pull_request_id = pull_request.pull_request_id
         repo_name = pull_request.target_repo.scm_instance().name
 
+        merge_resp = MergeResponse(True, False, 'STUB_COMMIT_ID',
+                                   MergeFailureReason.PUSH_FAILED,
+                                   metadata={'target': 'shadow repo',
+                                             'merge_commit': 'xxx'})
         model_patcher = mock.patch.multiple(
             PullRequestModel,
-            merge_repo=mock.Mock(return_value=MergeResponse(
-                True, False, 'STUB_COMMIT_ID', MergeFailureReason.PUSH_FAILED)),
+            merge_repo=mock.Mock(return_value=merge_resp),
             merge_status=mock.Mock(return_value=(True, 'WRONG_MESSAGE')))
 
         with model_patcher:
@@ -637,8 +638,10 b' class TestPullrequestsView(object):'
                            pull_request_id=pull_request_id),
                 params={'csrf_token': csrf_token}, status=302)
 
-        assert_session_flash(response, PullRequestModel.MERGE_STATUS_MESSAGES[
-            MergeFailureReason.PUSH_FAILED])
+            merge_resp = MergeResponse(True, True, '', MergeFailureReason.PUSH_FAILED,
+                                       metadata={'target': 'shadow repo',
+                                                 'merge_commit': 'xxx'})
+        assert_session_flash(response, merge_resp.merge_status_message)
 
     def test_update_source_revision(self, backend, csrf_token):
         commits = [
@@ -652,20 +655,20 b' class TestPullrequestsView(object):'
 
         # create pr from a in source to A in target
         pull_request = PullRequest()
+
         pull_request.source_repo = source
-        # TODO: johbo: Make sure that we write the source ref this way!
         pull_request.source_ref = 'branch:{branch}:{commit_id}'.format(
             branch=backend.default_branch_name, commit_id=commit_ids['change'])
+
         pull_request.target_repo = target
-
         pull_request.target_ref = 'branch:{branch}:{commit_id}'.format(
-            branch=backend.default_branch_name,
-            commit_id=commit_ids['ancestor'])
+            branch=backend.default_branch_name, commit_id=commit_ids['ancestor'])
+
         pull_request.revisions = [commit_ids['change']]
         pull_request.title = u"Test"
         pull_request.description = u"Description"
-        pull_request.author = UserModel().get_by_username(
-            TEST_USER_ADMIN_LOGIN)
+        pull_request.author = UserModel().get_by_username(TEST_USER_ADMIN_LOGIN)
+        pull_request.pull_request_state = PullRequest.STATE_CREATED
         Session().add(pull_request)
         Session().commit()
         pull_request_id = pull_request.pull_request_id
@@ -676,23 +679,21 b' class TestPullrequestsView(object):'
         # update PR
         self.app.post(
             route_path('pullrequest_update',
+                       repo_name=target.repo_name, pull_request_id=pull_request_id),
+            params={'update_commits': 'true', 'csrf_token': csrf_token})
+
+        response = self.app.get(
+            route_path('pullrequest_show',
                 repo_name=target.repo_name,
-                pull_request_id=pull_request_id),
-            params={'update_commits': 'true',
-                    'csrf_token': csrf_token})
+                       pull_request_id=pull_request.pull_request_id))
+
+        assert response.status_int == 200
+        assert 'Pull request updated to' in response.body
+        assert 'with 1 added, 0 removed commits.' in response.body
 
         # check that we have now both revisions
         pull_request = PullRequest.get(pull_request_id)
-        assert pull_request.revisions == [
-            commit_ids['change-2'], commit_ids['change']]
-
-        # TODO: johbo: this should be a test on its own
-        response = self.app.get(route_path(
-            'pullrequest_new',
-            repo_name=target.repo_name))
-        assert response.status_int == 200
-        assert 'Pull request updated to' in response.body
-        assert 'with 1 added, 0 removed commits.' in response.body
+        assert pull_request.revisions == [commit_ids['change-2'], commit_ids['change']]
 
     def test_update_target_revision(self, backend, csrf_token):
         commits = [
@@ -707,21 +708,21 b' class TestPullrequestsView(object):'
 
         # create pr from a in source to A in target
         pull_request = PullRequest()
+
         pull_request.source_repo = source
-        # TODO: johbo: Make sure that we write the source ref this way!
         pull_request.source_ref = 'branch:{branch}:{commit_id}'.format(
             branch=backend.default_branch_name, commit_id=commit_ids['change'])
+
         pull_request.target_repo = target
-        # TODO: johbo: Target ref should be branch based, since tip can jump
-        # from branch to branch
         pull_request.target_ref = 'branch:{branch}:{commit_id}'.format(
-            branch=backend.default_branch_name,
-            commit_id=commit_ids['ancestor'])
+            branch=backend.default_branch_name, commit_id=commit_ids['ancestor'])
+
         pull_request.revisions = [commit_ids['change']]
         pull_request.title = u"Test"
         pull_request.description = u"Description"
-        pull_request.author = UserModel().get_by_username(
-            TEST_USER_ADMIN_LOGIN)
+        pull_request.author = UserModel().get_by_username(TEST_USER_ADMIN_LOGIN)
+        pull_request.pull_request_state = PullRequest.STATE_CREATED
+
         Session().add(pull_request)
         Session().commit()
         pull_request_id = pull_request.pull_request_id
@@ -736,21 +737,19 b' class TestPullrequestsView(object):'
             route_path('pullrequest_update',
                 repo_name=target.repo_name,
                 pull_request_id=pull_request_id),
-            params={'update_commits': 'true',
-                    'csrf_token': csrf_token},
+            params={'update_commits': 'true', 'csrf_token': csrf_token},
             status=200)
 
         # check that we have now both revisions
         pull_request = PullRequest.get(pull_request_id)
         assert pull_request.revisions == [commit_ids['change-rebased']]
         assert pull_request.target_ref == 'branch:{branch}:{commit_id}'.format(
-            branch=backend.default_branch_name,
-            commit_id=commit_ids['ancestor-new'])
+            branch=backend.default_branch_name, commit_id=commit_ids['ancestor-new'])
 
-        # TODO: johbo: This should be a test on its own
-        response = self.app.get(route_path(
-            'pullrequest_new',
-            repo_name=target.repo_name))
+        response = self.app.get(
+            route_path('pullrequest_show',
+                       repo_name=target.repo_name,
+                       pull_request_id=pull_request.pull_request_id))
         assert response.status_int == 200
         assert 'Pull request updated to' in response.body
         assert 'with 1 added, 1 removed commits.' in response.body
@@ -772,17 +771,14 b' class TestPullrequestsView(object):'
         # create pr from a in source to A in target
         pull_request = PullRequest()
         pull_request.source_repo = source
-        # TODO: johbo: Make sure that we write the source ref this way!
+
         pull_request.source_ref = 'branch:{branch}:{commit_id}'.format(
             branch=backend.default_branch_name,
             commit_id=commit_ids['master-commit-3-change-2'])
 
         pull_request.target_repo = target
-        # TODO: johbo: Target ref should be branch based, since tip can jump
-        # from branch to branch
         pull_request.target_ref = 'branch:{branch}:{commit_id}'.format(
-            branch=backend.default_branch_name,
-            commit_id=commit_ids['feat-commit-2'])
+            branch=backend.default_branch_name, commit_id=commit_ids['feat-commit-2'])
 
         pull_request.revisions = [
             commit_ids['feat-commit-1'],
@@ -790,8 +786,8 b' class TestPullrequestsView(object):'
         ]
         pull_request.title = u"Test"
         pull_request.description = u"Description"
-        pull_request.author = UserModel().get_by_username(
-            TEST_USER_ADMIN_LOGIN)
+        pull_request.author = UserModel().get_by_username(TEST_USER_ADMIN_LOGIN)
+        pull_request.pull_request_state = PullRequest.STATE_CREATED
         Session().add(pull_request)
         Session().commit()
         pull_request_id = pull_request.pull_request_id
@@ -807,13 +803,10 b' class TestPullrequestsView(object):'
             route_path('pullrequest_update',
                 repo_name=target.repo_name,
                 pull_request_id=pull_request_id),
-            params={'update_commits': 'true',
-                    'csrf_token': csrf_token},
+            params={'update_commits': 'true', 'csrf_token': csrf_token},
             status=200)
 
-        response = self.app.get(route_path(
-            'pullrequest_new',
-            repo_name=target.repo_name))
+        response = self.app.get(route_path('pullrequest_new', repo_name=target.repo_name))
         assert response.status_int == 200
         response.mustcontain('Pull request updated to')
         response.mustcontain('with 0 added, 0 removed commits.')
@@ -833,21 +826,17 b' class TestPullrequestsView(object):'
         # create pr from a in source to A in target
         pull_request = PullRequest()
         pull_request.source_repo = source
-        # TODO: johbo: Make sure that we write the source ref this way!
+
         pull_request.source_ref = 'branch:{branch}:{commit_id}'.format(
-            branch=backend.default_branch_name,
-            commit_id=commit_ids['change'])
+            branch=backend.default_branch_name, commit_id=commit_ids['change'])
         pull_request.target_repo = target
-        # TODO: johbo: Target ref should be branch based, since tip can jump
-        # from branch to branch
         pull_request.target_ref = 'branch:{branch}:{commit_id}'.format(
-            branch=backend.default_branch_name,
-            commit_id=commit_ids['ancestor'])
+            branch=backend.default_branch_name, commit_id=commit_ids['ancestor'])
         pull_request.revisions = [commit_ids['change']]
         pull_request.title = u"Test"
         pull_request.description = u"Description"
-        pull_request.author = UserModel().get_by_username(
-            TEST_USER_ADMIN_LOGIN)
+        pull_request.author = UserModel().get_by_username(TEST_USER_ADMIN_LOGIN)
+        pull_request.pull_request_state = PullRequest.STATE_CREATED
         Session().add(pull_request)
         Session().commit()
         pull_request_id = pull_request.pull_request_id
@@ -860,10 +849,8 b' class TestPullrequestsView(object):'
         # update PR
         self.app.post(
             route_path('pullrequest_update',
-                repo_name=target.repo_name,
-                pull_request_id=pull_request_id),
-            params={'update_commits': 'true',
-                    'csrf_token': csrf_token},
+                       repo_name=target.repo_name, pull_request_id=pull_request_id),
+            params={'update_commits': 'true', 'csrf_token': csrf_token},
             status=200)
 
         # Expect the target reference to be updated correctly
@@ -890,13 +877,12 b' class TestPullrequestsView(object):'
         pull_request.source_ref = 'branch:{branch}:{commit_id}'.format(
             branch=branch_name, commit_id=commit_ids['new-feature'])
         pull_request.target_ref = 'branch:{branch}:{commit_id}'.format(
-            branch=backend_git.default_branch_name,
-            commit_id=commit_ids['old-feature'])
+            branch=backend_git.default_branch_name, commit_id=commit_ids['old-feature'])
         pull_request.revisions = [commit_ids['new-feature']]
         pull_request.title = u"Test"
         pull_request.description = u"Description"
-        pull_request.author = UserModel().get_by_username(
-            TEST_USER_ADMIN_LOGIN)
+        pull_request.author = UserModel().get_by_username(TEST_USER_ADMIN_LOGIN)
+        pull_request.pull_request_state = PullRequest.STATE_CREATED
         Session().add(pull_request)
         Session().commit()
 
@@ -909,11 +895,11 b' class TestPullrequestsView(object):'
             pull_request_id=pull_request.pull_request_id))
 
         assert response.status_int == 200
-        assert_response = AssertResponse(response)
-        assert_response.element_contains(
+
+        response.assert_response().element_contains(
             '#changeset_compare_view_content .alert strong',
             'Missing commits')
-        assert_response.element_contains(
+        response.assert_response().element_contains(
             '#changeset_compare_view_content .alert',
             'This pull request cannot be displayed, because one or more'
             ' commits no longer exist in the source repository.')
@@ -941,15 +927,15 b' class TestPullrequestsView(object):'
             pull_request_id=pull_request.pull_request_id))
 
         assert response.status_int == 200
-        assert_response = AssertResponse(response)
-        assert_response.element_contains(
+
+        response.assert_response().element_contains(
             '#changeset_compare_view_content .alert strong',
             'Missing commits')
-        assert_response.element_contains(
+        response.assert_response().element_contains(
             '#changeset_compare_view_content .alert',
             'This pull request cannot be displayed, because one or more'
             ' commits no longer exist in the source repository.')
-        assert_response.element_contains(
+        response.assert_response().element_contains(
             '#update_commits',
             'Update commits')
 
@@ -987,8 +973,7 b' class TestPullrequestsView(object):'
             pull_request_id=pull_request.pull_request_id))
 
         assert response.status_int == 200
-        assert_response = AssertResponse(response)
-        assert_response.element_contains(
+        response.assert_response().element_contains(
             '#changeset_compare_view_content .alert strong',
             'Missing commits')
 
@@ -1004,12 +989,11 b' class TestPullrequestsView(object):'
             repo_name=pull_request.target_repo.scm_instance().name,
             pull_request_id=pull_request.pull_request_id))
         assert response.status_int == 200
-        assert_response = AssertResponse(response)
 
-        origin = assert_response.get_element('.pr-origininfo .tag')
+        origin = response.assert_response().get_element('.pr-origininfo .tag')
         origin_children = origin.getchildren()
         assert len(origin_children) == 1
-        target = assert_response.get_element('.pr-targetinfo .tag')
+        target = response.assert_response().get_element('.pr-targetinfo .tag')
         target_children = target.getchildren()
         assert len(target_children) == 1
 
@@ -1038,13 +1022,12 b' class TestPullrequestsView(object):'
             repo_name=pull_request.target_repo.scm_instance().name,
             pull_request_id=pull_request.pull_request_id))
         assert response.status_int == 200
-        assert_response = AssertResponse(response)
 
-        origin = assert_response.get_element('.pr-origininfo .tag')
+        origin = response.assert_response().get_element('.pr-origininfo .tag')
         assert origin.text.strip() == 'bookmark: origin'
         assert origin.getchildren() == []
 
-        target = assert_response.get_element('.pr-targetinfo .tag')
+        target = response.assert_response().get_element('.pr-targetinfo .tag')
         assert target.text.strip() == 'bookmark: target'
         assert target.getchildren() == []
 
@@ -1060,13 +1043,12 b' class TestPullrequestsView(object):'
             repo_name=pull_request.target_repo.scm_instance().name,
             pull_request_id=pull_request.pull_request_id))
         assert response.status_int == 200
-        assert_response = AssertResponse(response)
 
-        origin = assert_response.get_element('.pr-origininfo .tag')
+        origin = response.assert_response().get_element('.pr-origininfo .tag')
         assert origin.text.strip() == 'tag: origin'
         assert origin.getchildren() == []
 
-        target = assert_response.get_element('.pr-targetinfo .tag')
+        target = response.assert_response().get_element('.pr-targetinfo .tag')
         assert target.text.strip() == 'tag: target'
         assert target.getchildren() == []
 
@@ -1090,12 +1072,13 b' class TestPullrequestsView(object):'
             repo_name=target_repo.name,
             pull_request_id=pr_id))
 
-        assertr = AssertResponse(response)
         if mergeable:
-            assertr.element_value_contains('input.pr-mergeinfo', shadow_url)
-            assertr.element_value_contains('input.pr-mergeinfo ', 'pr-merge')
+            response.assert_response().element_value_contains(
+                'input.pr-mergeinfo', shadow_url)
+            response.assert_response().element_value_contains(
+                'input.pr-mergeinfo ', 'pr-merge')
         else:
-            assertr.no_element_exists('.pr-mergeinfo')
+            response.assert_response().no_element_exists('.pr-mergeinfo')
 
 
 @pytest.mark.usefixtures('app')

rhodecode/apps/repository/tests/test_repo_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_settings_advanced.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_summary.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_tags.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_repo_vcs_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/tests/test_vcs_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/utils.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_audit_logs.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_automation.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_bookmarks.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_branch_permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_branches.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_caches.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_changelog.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -203,7 +203,6 b' class RepoChangelogView(RepoAppView):'
         pre_load = self._get_preload_attrs()
 
         partial_xhr = self.request.environ.get('HTTP_X_PARTIAL_XHR')
-
         try:
             if f_path:
                 log.debug('generating changelog for path %s', f_path)
@@ -231,7 +230,7 b' class RepoChangelogView(RepoAppView):'
             else:
                 collection = self.rhodecode_vcs_repo.get_commits(
                     branch_name=branch_name, show_hidden=show_hidden,
-                    pre_load=pre_load)
+                    pre_load=pre_load, translate_tags=False)
 
             self._load_changelog_data(
                 c, collection, p, chunk_size, c.branch_name,
@@ -320,7 +319,8 b' class RepoChangelogView(RepoAppView):'
             collection = list(reversed(collection))
         else:
             collection = self.rhodecode_vcs_repo.get_commits(
-                branch_name=branch_name, show_hidden=show_hidden, pre_load=pre_load)
+                branch_name=branch_name, show_hidden=show_hidden, pre_load=pre_load,
+                translate_tags=False)
 
         p = safe_int(self.request.GET.get('page', 1), 1)
         try:

rhodecode/apps/repository/views/repo_checks.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_commits.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -97,7 +97,7 b' class RepoCommitsView(RepoAppView):'
             if len(commit_range) == 2:
                 commits = self.rhodecode_vcs_repo.get_commits(
                     start_id=commit_range[0], end_id=commit_range[1],
-                    pre_load=pre_load)
+                    pre_load=pre_load, translate_tags=False)
                 commits = list(commits)
             else:
                 commits = [self.rhodecode_vcs_repo.get_commit(

rhodecode/apps/repository/views/repo_compare.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -27,12 +27,13 b' from pyramid.renderers import render'
 from pyramid.response import Response
 
 from rhodecode.apps._base import RepoAppView
-from rhodecode.controllers.utils import parse_path_ref, get_commit_from_ref_name
+
 from rhodecode.lib import helpers as h
 from rhodecode.lib import diffs, codeblocks
 from rhodecode.lib.auth import LoginRequired, HasRepoPermissionAnyDecorator
 from rhodecode.lib.utils import safe_str
 from rhodecode.lib.utils2 import safe_unicode, str2bool
+from rhodecode.lib.view_utils import parse_path_ref, get_commit_from_ref_name
 from rhodecode.lib.vcs.exceptions import (
     EmptyRepositoryError, RepositoryError, RepositoryRequirementError,
     NodeDoesNotExistError)

rhodecode/apps/repository/views/repo_feed.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_files.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -34,9 +34,10 b' from pyramid.response import Response'
 import rhodecode
 from rhodecode.apps._base import RepoAppView
 
-from rhodecode.controllers.utils import parse_path_ref
+
 from rhodecode.lib import diffs, helpers as h, rc_cache
 from rhodecode.lib import audit_logger
+from rhodecode.lib.view_utils import parse_path_ref
 from rhodecode.lib.exceptions import NonRelativePathError
 from rhodecode.lib.codeblocks import (
     filenode_as_lines_tokens, filenode_as_annotated_lines_tokens)
@@ -83,6 +84,7 b' class RepoFilesView(RepoAppView):'
     def load_default_context(self):
         c = self._get_local_tmpl_context(include_app_defaults=True)
         c.rhodecode_repo = self.rhodecode_vcs_repo
+        c.enable_downloads = self.db_repo.enable_downloads
         return c
 
     def _ensure_not_locked(self):
@@ -227,10 +229,11 b' class RepoFilesView(RepoAppView):'
             self, c, commit_id, f_path, full_load=False):
 
         repo_id = self.db_repo.repo_id
+        force_recache = self.get_recache_flag()
 
         cache_seconds = safe_int(
             rhodecode.CONFIG.get('rc_cache.cache_repo.expiration_time'))
-        cache_on = cache_seconds > 0
+        cache_on = not force_recache and cache_seconds > 0
         log.debug(
             'Computing FILE TREE for repo_id %s commit_id `%s` and path `%s`'
             'with caching: %s[TTL: %ss]' % (

rhodecode/apps/repository/views/repo_forks.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_maintainance.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -50,8 +50,13 b' class RepoSettingsPermissionsView(RepoAp'
         route_name='edit_repo_perms', request_method='GET',
         renderer='rhodecode:templates/admin/repos/repo_edit.mako')
     def edit_permissions(self):
+        _ = self.request.translate
         c = self.load_default_context()
         c.active = 'permissions'
+        if self.request.GET.get('branch_permissions'):
+            h.flash(_('Explicitly add user or user group with write+ '
+                      'permission to modify their branch permissions.'),
+                    category='notice')
         return self._get_template_context(c)
 
     @LoginRequired()

rhodecode/apps/repository/views/repo_pull_requests.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -137,13 +137,6 b' class RepoPullRequestsView(RepoAppView, '
         })
         return data
 
-    def get_recache_flag(self):
-        for flag_name in ['force_recache', 'force-recache', 'no-cache']:
-            flag_val = self.request.GET.get(flag_name)
-            if str2bool(flag_val):
-                return True
-        return False
-
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
@@ -272,9 +265,22 b' class RepoPullRequestsView(RepoAppView, '
         route_name='pullrequest_show', request_method='GET',
         renderer='rhodecode:templates/pullrequests/pullrequest_show.mako')
     def pull_request_show(self):
-        pull_request_id = self.request.matchdict['pull_request_id']
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        pull_request = PullRequest.get_or_404(
+            self.request.matchdict['pull_request_id'])
+        pull_request_id = pull_request.pull_request_id
 
-        c = self.load_default_context()
+        if pull_request.pull_request_state != PullRequest.STATE_CREATED:
+            log.debug('show: forbidden because pull request is in state %s',
+                      pull_request.pull_request_state)
+            msg = _(u'Cannot show pull requests in state other than `{}`. '
+                    u'Current state is: `{}`').format(PullRequest.STATE_CREATED,
+                                                      pull_request.pull_request_state)
+            h.flash(msg, category='error')
+            raise HTTPFound(h.route_path('pullrequest_show_all',
+                                         repo_name=self.db_repo_name))
 
         version = self.request.GET.get('version')
         from_version = self.request.GET.get('from_version') or version
@@ -754,7 +760,7 b' class RepoPullRequestsView(RepoAppView, '
 
         default_target_repo = source_repo
 
-        if source_repo.parent:
+        if source_repo.parent and c.has_origin_repo_read_perm:
             parent_vcs_obj = source_repo.parent.scm_instance()
             if parent_vcs_obj and not parent_vcs_obj.is_empty():
                 # change default if we have a parent repo
@@ -811,37 +817,51 b' class RepoPullRequestsView(RepoAppView, '
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     @view_config(
-        route_name='pullrequest_repo_destinations', request_method='GET',
+        route_name='pullrequest_repo_targets', request_method='GET',
         renderer='json_ext', xhr=True)
-    def pull_request_repo_destinations(self):
+    def pullrequest_repo_targets(self):
         _ = self.request.translate
         filter_query = self.request.GET.get('query')
 
-        query = Repository.query() \
+        # get the parents
+        parent_target_repos = []
+        if self.db_repo.parent:
+            parents_query = Repository.query() \
             .order_by(func.length(Repository.repo_name)) \
-            .filter(
-                or_(Repository.repo_name == self.db_repo.repo_name,
-                    Repository.fork_id == self.db_repo.repo_id))
+                .filter(Repository.fork_id == self.db_repo.parent.repo_id)
 
         if filter_query:
             ilike_expression = u'%{}%'.format(safe_unicode(filter_query))
-            query = query.filter(
+                parents_query = parents_query.filter(
                 Repository.repo_name.ilike(ilike_expression))
+            parents = parents_query.limit(20).all()
+
+            for parent in parents:
+                parent_vcs_obj = parent.scm_instance()
+                if parent_vcs_obj and not parent_vcs_obj.is_empty():
+                    parent_target_repos.append(parent)
 
-        add_parent = False
-        if self.db_repo.parent:
-            if filter_query in self.db_repo.parent.repo_name:
-                parent_vcs_obj = self.db_repo.parent.scm_instance()
-                if parent_vcs_obj and not parent_vcs_obj.is_empty():
-                    add_parent = True
+        # get other forks, and repo itself
+        query = Repository.query() \
+            .order_by(func.length(Repository.repo_name)) \
+            .filter(
+                or_(Repository.repo_id == self.db_repo.repo_id,  # repo itself
+                    Repository.fork_id == self.db_repo.repo_id)  # forks of this repo
+            )  \
+            .filter(~Repository.repo_id.in_([x.repo_id for x in parent_target_repos]))
 
-        limit = 20 - 1 if add_parent else 20
-        all_repos = query.limit(limit).all()
-        if add_parent:
-            all_repos += [self.db_repo.parent]
+        if filter_query:
+            ilike_expression = u'%{}%'.format(safe_unicode(filter_query))
+            query = query.filter(Repository.repo_name.ilike(ilike_expression))
+
+        limit = max(20 - len(parent_target_repos), 5)  # not less then 5
+        target_repos = query.limit(limit).all()
+
+        all_target_repos = target_repos + parent_target_repos
 
         repos = []
-        for obj in ScmModel().get_repos(all_repos):
+        # This checks permissions to the repositories
+        for obj in ScmModel().get_repos(all_target_repos):
             repos.append({
                 'id': obj['name'],
                 'text': obj['name'],
@@ -904,12 +924,17 b' class RepoPullRequestsView(RepoAppView, '
         source_db_repo = Repository.get_by_repo_name(_form['source_repo'])
         target_db_repo = Repository.get_by_repo_name(_form['target_repo'])
 
+        if not (source_db_repo or target_db_repo):
+            h.flash(_('source_repo or target repo not found'), category='error')
+            raise HTTPFound(
+                h.route_path('pullrequest_new', repo_name=self.db_repo_name))
+
         # re-check permissions again here
         # source_repo we must have read permissions
 
         source_perm = HasRepoPermissionAny(
-            'repository.read',
-            'repository.write', 'repository.admin')(source_db_repo.repo_name)
+            'repository.read', 'repository.write', 'repository.admin')(
+            source_db_repo.repo_name)
         if not source_perm:
             msg = _('Not Enough permissions to source repo `{}`.'.format(
                 source_db_repo.repo_name))
@@ -923,8 +948,8 b' class RepoPullRequestsView(RepoAppView, '
         # target repo we must have read permissions, and also later on
         # we want to check branch permissions here
         target_perm = HasRepoPermissionAny(
-            'repository.read',
-            'repository.write', 'repository.admin')(target_db_repo.repo_name)
+            'repository.read', 'repository.write', 'repository.admin')(
+            target_db_repo.repo_name)
         if not target_perm:
             msg = _('Not Enough permissions to target repo `{}`.'.format(
                 target_db_repo.repo_name))
@@ -1027,6 +1052,15 b' class RepoPullRequestsView(RepoAppView, '
             h.flash(msg, category='error')
             return True
 
+        if pull_request.pull_request_state != PullRequest.STATE_CREATED:
+            log.debug('update: forbidden because pull request is in state %s',
+                      pull_request.pull_request_state)
+            msg = _(u'Cannot update pull requests in state other than `{}`. '
+                    u'Current state is: `{}`').format(PullRequest.STATE_CREATED,
+                                                      pull_request.pull_request_state)
+            h.flash(msg, category='error')
+            return True
+
         # only owner or admin can update it
         allowed_to_update = PullRequestModel().check_user_update(
             pull_request, self._rhodecode_user)
@@ -1069,6 +1103,8 b' class RepoPullRequestsView(RepoAppView, '
 
     def _update_commits(self, pull_request):
         _ = self.request.translate
+
+        with pull_request.set_state(PullRequest.STATE_UPDATING):
         resp = PullRequestModel().update_commits(pull_request)
 
         if resp.executed:
@@ -1082,8 +1118,7 b' class RepoPullRequestsView(RepoAppView, '
             else:
                 changed = 'nothing'
 
-            msg = _(
-                u'Pull request updated to "{source_commit_id}" with '
+            msg = _(u'Pull request updated to "{source_commit_id}" with '
                 u'{count_added} added, {count_removed} removed commits. '
                 u'Source of changes: {change_source}')
             msg = msg.format(
@@ -1094,8 +1129,7 b' class RepoPullRequestsView(RepoAppView, '
             h.flash(msg, category='success')
 
             channel = '/repo${}$/pr/{}'.format(
-                pull_request.target_repo.repo_name,
-                pull_request.pull_request_id)
+                pull_request.target_repo.repo_name, pull_request.pull_request_id)
             message = msg + (
                 ' - <a onclick="window.location.reload()">'
                 '<strong>{}</strong></a>'.format(_('Reload page')))
@@ -1128,8 +1162,23 b' class RepoPullRequestsView(RepoAppView, '
         """
         pull_request = PullRequest.get_or_404(
             self.request.matchdict['pull_request_id'])
+        _ = self.request.translate
+
+        if pull_request.pull_request_state != PullRequest.STATE_CREATED:
+            log.debug('show: forbidden because pull request is in state %s',
+                      pull_request.pull_request_state)
+            msg = _(u'Cannot merge pull requests in state other than `{}`. '
+                    u'Current state is: `{}`').format(PullRequest.STATE_CREATED,
+                                                      pull_request.pull_request_state)
+            h.flash(msg, category='error')
+            raise HTTPFound(
+                h.route_path('pullrequest_show',
+                             repo_name=pull_request.target_repo.repo_name,
+                             pull_request_id=pull_request.pull_request_id))
 
         self.load_default_context()
+
+        with pull_request.set_state(PullRequest.STATE_UPDATING):
         check = MergeCheck.validate(
             pull_request, auth_user=self._rhodecode_user,
             translator=self.request.translate)
@@ -1144,6 +1193,7 b' class RepoPullRequestsView(RepoAppView, '
                 self.request.environ, repo_name=pull_request.target_repo.repo_name,
                 username=self._rhodecode_db_user.username, action='push',
                 scm=pull_request.target_repo.repo_type)
+            with pull_request.set_state(PullRequest.STATE_UPDATING):
             self._merge_pull_request(
                 pull_request, self._rhodecode_db_user, extras)
         else:
@@ -1167,10 +1217,8 b' class RepoPullRequestsView(RepoAppView, '
             h.flash(msg, category='success')
         else:
             log.debug(
-                "The merge was not successful. Merge response: %s",
-                merge_resp)
-            msg = PullRequestModel().merge_status_message(
-                merge_resp.failure_reason)
+                "The merge was not successful. Merge response: %s", merge_resp)
+            msg = merge_resp.merge_status_message
             h.flash(msg, category='error')
 
     def _update_reviewers(self, pull_request, review_members, reviewer_rules):

rhodecode/apps/repository/views/repo_review_rules.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_settings_advanced.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -73,6 +73,8 b' class RepoSettingsView(RepoAppView):'
                 'repository.write', 'repository.read', 'repository.admin')(
                 self.db_repo.fork.repo_name, 'repo set as fork page')
 
+        c.ver_info_dict = self.rhodecode_vcs_repo.get_hooks_info()
+
         return self._get_template_context(c)
 
     @LoginRequired()

rhodecode/apps/repository/views/repo_settings_fields.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_settings_issue_trackers.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_settings_remote.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_settings_vcs.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_strip.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/repository/views/repo_summary.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -24,7 +24,7 b' import rhodecode'
 
 from pyramid.view import view_config
 
-from rhodecode.controllers import utils
+from rhodecode.lib.view_utils import get_format_ref_id
 from rhodecode.apps._base import RepoAppView
 from rhodecode.config.conf import (LANGUAGES_EXTENSIONS_MAP)
 from rhodecode.lib import helpers as h, rc_cache
@@ -141,7 +141,8 b' class RepoSummaryView(RepoAppView):'
 
         pre_load = ['author', 'branch', 'date', 'message']
         try:
-            collection = self.rhodecode_vcs_repo.get_commits(pre_load=pre_load)
+            collection = self.rhodecode_vcs_repo.get_commits(
+                pre_load=pre_load, translate_tags=False)
         except EmptyRepositoryError:
             collection = self.rhodecode_vcs_repo
 
@@ -351,7 +352,7 b' class RepoSummaryView(RepoAppView):'
         return data
 
     def _create_reference_data(self, repo, full_repo_name, refs_to_create):
-        format_ref_id = utils.get_format_ref_id(repo)
+        format_ref_id = get_format_ref_id(repo)
 
         result = []
         for title, refs, ref_type in refs_to_create:

rhodecode/apps/repository/views/repo_tags.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/search/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -28,7 +28,16 b' def includeme(config):'
 
     config.add_route(
         name='search_repo',
+        pattern='/{repo_name:.*?[^/]}/_search', repo_route=True)
+
+    config.add_route(
+        name='search_repo_alt',
         pattern='/{repo_name:.*?[^/]}/search', repo_route=True)
 
+    config.add_route(
+        name='search_repo_group',
+        pattern='/{repo_group_name:.*?[^/]}/_search',
+        repo_group_route=True)
+
     # Scan module for configuration decorators.
     config.scan('.views', ignore='.tests')

rhodecode/apps/search/tests/test_search.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/search/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2011-2018 RhodeCode GmbH
+# Copyright (C) 2011-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -23,8 +23,9 b' import urllib'
 from pyramid.view import view_config
 from webhelpers.util import update_params
 
-from rhodecode.apps._base import BaseAppView, RepoAppView
-from rhodecode.lib.auth import (LoginRequired, HasRepoPermissionAnyDecorator)
+from rhodecode.apps._base import BaseAppView, RepoAppView, RepoGroupAppView
+from rhodecode.lib.auth import (
+    LoginRequired, HasRepoPermissionAnyDecorator, HasRepoGroupPermissionAnyDecorator)
 from rhodecode.lib.helpers import Page
 from rhodecode.lib.utils2 import safe_str
 from rhodecode.lib.index import searcher_from_config
@@ -34,22 +35,25 b' from rhodecode.model.validation_schema.s'
 log = logging.getLogger(__name__)
 
 
-def search(request, tmpl_context, repo_name):
+def perform_search(request, tmpl_context, repo_name=None, repo_group_name=None):
     searcher = searcher_from_config(request.registry.settings)
     formatted_results = []
     execution_time = ''
 
     schema = search_schema.SearchParamsSchema()
-
+    search_tags = []
     search_params = {}
     errors = []
     try:
         search_params = schema.deserialize(
-            dict(search_query=request.GET.get('q'),
+            dict(
+                search_query=request.GET.get('q'),
                  search_type=request.GET.get('type'),
                  search_sort=request.GET.get('sort'),
+                search_max_lines=request.GET.get('max_lines'),
                  page_limit=request.GET.get('page_limit'),
-                 requested_page=request.GET.get('page'))
+                requested_page=request.GET.get('page'),
+             )
         )
     except validation_schema.Invalid as e:
         errors = e.children
@@ -57,20 +61,22 b' def search(request, tmpl_context, repo_n'
     def url_generator(**kw):
         q = urllib.quote(safe_str(search_query))
         return update_params(
-            "?q=%s&type=%s" % (q, safe_str(search_type)), **kw)
+            "?q=%s&type=%s&max_lines=%s" % (
+                q, safe_str(search_type), search_max_lines), **kw)
 
     c = tmpl_context
     search_query = search_params.get('search_query')
     search_type = search_params.get('search_type')
     search_sort = search_params.get('search_sort')
+    search_max_lines = search_params.get('search_max_lines')
     if search_params.get('search_query'):
         page_limit = search_params['page_limit']
         requested_page = search_params['requested_page']
 
         try:
             search_result = searcher.search(
-                search_query, search_type, c.auth_user, repo_name,
-                requested_page, page_limit, search_sort)
+                search_query, search_type, c.auth_user, repo_name, repo_group_name,
+                requested_page=requested_page, page_limit=page_limit, sort=search_sort)
 
             formatted_results = Page(
                 search_result['results'], page=requested_page,
@@ -79,6 +85,8 b' def search(request, tmpl_context, repo_n'
         finally:
             searcher.cleanup()
 
+        search_tags = searcher.extract_search_tags(search_query)
+
         if not search_result['error']:
             execution_time = '%s results (%.3f seconds)' % (
                 search_result['count'],
@@ -90,6 +98,7 b' def search(request, tmpl_context, repo_n'
 
     c.perm_user = c.auth_user
     c.repo_name = repo_name
+    c.repo_group_name = repo_group_name
     c.sort = search_sort
     c.url_generator = url_generator
     c.errors = errors
@@ -98,12 +107,12 b' def search(request, tmpl_context, repo_n'
     c.cur_query = search_query
     c.search_type = search_type
     c.searcher = searcher
+    c.search_tags = search_tags
 
 
 class SearchView(BaseAppView):
     def load_default_context(self):
         c = self._get_local_tmpl_context()
-
         return c
 
     @LoginRequired()
@@ -112,14 +121,14 b' class SearchView(BaseAppView):'
         renderer='rhodecode:templates/search/search.mako')
     def search(self):
         c = self.load_default_context()
-        search(self.request, c, repo_name=None)
+        perform_search(self.request, c)
         return self._get_template_context(c)
 
 
 class SearchRepoView(RepoAppView):
     def load_default_context(self):
         c = self._get_local_tmpl_context()
-
+        c.active = 'search'
         return c
 
     @LoginRequired()
@@ -128,7 +137,28 b' class SearchRepoView(RepoAppView):'
     @view_config(
         route_name='search_repo', request_method='GET',
         renderer='rhodecode:templates/search/search.mako')
+    @view_config(
+        route_name='search_repo_alt', request_method='GET',
+        renderer='rhodecode:templates/search/search.mako')
     def search_repo(self):
         c = self.load_default_context()
-        search(self.request, c, repo_name=self.db_repo_name)
+        perform_search(self.request, c, repo_name=self.db_repo_name)
         return self._get_template_context(c)
+
+
+class SearchRepoGroupView(RepoGroupAppView):
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+        c.active = 'search'
+        return c
+
+    @LoginRequired()
+    @HasRepoGroupPermissionAnyDecorator(
+        'group.read', 'group.write', 'group.admin')
+    @view_config(
+        route_name='search_repo_group', request_method='GET',
+        renderer='rhodecode:templates/search/search.mako')
+    def search_repo_group(self):
+        c = self.load_default_context()
+        perform_search(self.request, c, repo_group_name=self.db_repo_group_name)
+        return self._get_template_context(c)

rhodecode/apps/ssh_support/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -36,6 +36,7 b' def _sanitize_settings_and_apply_default'
     _bool_setting(settings, config_keys.generate_authorized_keyfile, 'false')
     _bool_setting(settings, config_keys.wrapper_allow_shell, 'false')
     _bool_setting(settings, config_keys.enable_debug_logging, 'false')
+    _bool_setting(settings, config_keys.ssh_key_generator_enabled, 'true')
 
     _string_setting(settings, config_keys.authorized_keys_file_path,
                     '~/.ssh/authorized_keys_rhodecode',

rhodecode/apps/ssh_support/config_keys.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -24,6 +24,7 b''
 generate_authorized_keyfile = 'ssh.generate_authorized_keyfile'
 authorized_keys_file_path = 'ssh.authorized_keys_file_path'
 authorized_keys_line_ssh_opts = 'ssh.authorized_keys_ssh_opts'
+ssh_key_generator_enabled = 'ssh.enable_ui_key_generator'
 wrapper_cmd = 'ssh.wrapper_cmd'
 wrapper_allow_shell = 'ssh.wrapper_cmd_allow_shell'
 enable_debug_logging = 'ssh.enable_debug_logging'

rhodecode/apps/ssh_support/events.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/lib/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/lib/backends/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/lib/backends/base.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/lib/backends/git.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/lib/backends/hg.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/lib/backends/svn.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/lib/ssh_wrapper.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/subscribers.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/tests/conftest.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/tests/test_server_git.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/tests/test_server_hg.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/tests/test_server_svn.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/tests/test_ssh_authorized_keys_gen.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/tests/test_ssh_wrapper.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/ssh_support/utils.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/svn_support/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -17,10 +17,10 b''
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
-
+import os
 import logging
-import os
 import shlex
+from pyramid import compat
 
 # Do not use `from rhodecode import events` here, it will be overridden by the
 # events module in this package due to pythons import mechanism.
@@ -85,6 +85,6 b' def _append_path_sep(path):'
     """
     Append the path separator if missing.
     """
-    if isinstance(path, basestring) and not path.endswith(os.path.sep):
+    if isinstance(path, compat.string_types) and not path.endswith(os.path.sep):
         path += os.path.sep
     return path

rhodecode/apps/svn_support/config_keys.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/svn_support/events.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/svn_support/subscribers.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/svn_support/tests/test_mod_dav_svn_config.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/svn_support/utils.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group/tests/test_user_groups.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group/tests/test_user_groups_permissions.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group/views/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group_profile/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group_profile/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group_profile/tests/test_user_group.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_group_profile/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_profile/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_profile/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_profile/tests/test_users.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/apps/user_profile/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/base.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/interface.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/plugins/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/plugins/auth_crowd.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/plugins/auth_headers.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/plugins/auth_jasig_cas.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/plugins/auth_ldap.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/plugins/auth_pam.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/plugins/auth_rhodecode.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -24,12 +24,15 b' RhodeCode authentication plugin for buil'
 
 import logging
 
-from rhodecode.translation import _
+import colander
 
-from rhodecode.authentication.base import RhodeCodeAuthPluginBase, hybrid_property
-from rhodecode.authentication.routes import AuthnPluginResourceBase
+from rhodecode.translation import _
 from rhodecode.lib.utils2 import safe_str
 from rhodecode.model.db import User
+from rhodecode.authentication.schema import AuthnPluginSettingsSchemaBase
+from rhodecode.authentication.base import (
+    RhodeCodeAuthPluginBase, hybrid_property, HTTP_TYPE, VCS_TYPE)
+from rhodecode.authentication.routes import AuthnPluginResourceBase
 
 log = logging.getLogger(__name__)
 
@@ -45,6 +48,11 b' class RhodecodeAuthnResource(AuthnPlugin'
 
 class RhodeCodeAuthPlugin(RhodeCodeAuthPluginBase):
     uid = 'rhodecode'
+    AUTH_RESTRICTION_NONE = 'user_all'
+    AUTH_RESTRICTION_SUPER_ADMIN = 'user_super_admin'
+    AUTH_RESTRICTION_SCOPE_ALL = 'scope_all'
+    AUTH_RESTRICTION_SCOPE_HTTP = 'scope_http'
+    AUTH_RESTRICTION_SCOPE_VCS = 'scope_vcs'
 
     def includeme(self, config):
         config.add_authn_plugin(self)
@@ -64,6 +72,9 b' class RhodeCodeAuthPlugin(RhodeCodeAuthP'
             route_name='auth_home',
             context=RhodecodeAuthnResource)
 
+    def get_settings_schema(self):
+        return RhodeCodeSettingsSchema()
+
     def get_display_name(self):
         return _('RhodeCode Internal')
 
@@ -94,12 +105,36 b' class RhodeCodeAuthPlugin(RhodeCodeAuthP'
         if not userobj:
             log.debug('userobj was:%s skipping', userobj)
             return None
+
         if userobj.extern_type != self.name:
-            log.warning(
-                "userobj:%s extern_type mismatch got:`%s` expected:`%s`",
+            log.warning("userobj:%s extern_type mismatch got:`%s` expected:`%s`",
                 userobj, userobj.extern_type, self.name)
             return None
 
+        # check scope of auth
+        scope_restriction = settings.get('scope_restriction', '')
+
+        if scope_restriction == self.AUTH_RESTRICTION_SCOPE_HTTP \
+                and self.auth_type != HTTP_TYPE:
+            log.warning("userobj:%s tried scope type %s and scope restriction is set to %s",
+                        userobj, self.auth_type, scope_restriction)
+            return None
+
+        if scope_restriction == self.AUTH_RESTRICTION_SCOPE_VCS \
+                and self.auth_type != VCS_TYPE:
+            log.warning("userobj:%s tried scope type %s and scope restriction is set to %s",
+                        userobj, self.auth_type, scope_restriction)
+            return None
+
+        # check super-admin restriction
+        auth_restriction = settings.get('auth_restriction', '')
+
+        if auth_restriction == self.AUTH_RESTRICTION_SUPER_ADMIN \
+                and userobj.admin is False:
+            log.warning("userobj:%s is not super-admin and auth restriction is set to %s",
+                        userobj, auth_restriction)
+            return None
+
         user_attrs = {
             "username": userobj.username,
             "firstname": userobj.firstname,
@@ -131,23 +166,55 b' class RhodeCodeAuthPlugin(RhodeCodeAuthP'
                 user_attrs['_hash_migrate'] = new_hash
 
             if userobj.username == User.DEFAULT_USER and userobj.active:
-                log.info(
-                    'user `%s` authenticated correctly as anonymous user', userobj.username)
+                log.info('user `%s` authenticated correctly as anonymous user',
+                         userobj.username)
                 return user_attrs
 
             elif userobj.username == username and password_match:
                 log.info('user `%s` authenticated correctly', userobj.username)
                 return user_attrs
-            log.warn("user `%s` used a wrong password when "
+            log.warning("user `%s` used a wrong password when "
                      "authenticating on this plugin", userobj.username)
             return None
         else:
-            log.warning(
-                'user `%s` failed to authenticate via %s, reason: account not '
+            log.warning('user `%s` failed to authenticate via %s, reason: account not '
                 'active.', username, self.name)
             return None
 
 
+class RhodeCodeSettingsSchema(AuthnPluginSettingsSchemaBase):
+
+    auth_restriction_choices = [
+        (RhodeCodeAuthPlugin.AUTH_RESTRICTION_NONE, 'All users'),
+        (RhodeCodeAuthPlugin.AUTH_RESTRICTION_SUPER_ADMIN, 'Super admins only'),
+    ]
+
+    auth_scope_choices = [
+        (RhodeCodeAuthPlugin.AUTH_RESTRICTION_SCOPE_ALL, 'HTTP and VCS'),
+        (RhodeCodeAuthPlugin.AUTH_RESTRICTION_SCOPE_HTTP, 'HTTP only'),
+    ]
+
+    auth_restriction = colander.SchemaNode(
+        colander.String(),
+        default=auth_restriction_choices[0],
+        description=_('Allowed user types for authentication using this plugin.'),
+        title=_('User restriction'),
+        validator=colander.OneOf([x[0] for x in auth_restriction_choices]),
+        widget='select_with_labels',
+        choices=auth_restriction_choices
+    )
+    scope_restriction = colander.SchemaNode(
+        colander.String(),
+        default=auth_scope_choices[0],
+        description=_('Allowed protocols for authentication using this plugin. '
+                      'VCS means GIT/HG/SVN. HTTP is web based login.'),
+        title=_('Scope restriction'),
+        validator=colander.OneOf([x[0] for x in auth_scope_choices]),
+        widget='select_with_labels',
+        choices=auth_scope_choices
+    )
+
+
 def includeme(config):
     plugin_id = 'egg:rhodecode-enterprise-ce#{}'.format(RhodeCodeAuthPlugin.uid)
     plugin_factory(plugin_id).includeme(config)

rhodecode/authentication/plugins/auth_token.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -23,7 +23,9 b' RhodeCode authentication token plugin fo'
 """
 
 import logging
+import colander
 
+from rhodecode.authentication.schema import AuthnPluginSettingsSchemaBase
 from rhodecode.translation import _
 from rhodecode.authentication.base import (
     RhodeCodeAuthPluginBase, VCS_TYPE, hybrid_property)
@@ -48,6 +50,7 b' class RhodeCodeAuthPlugin(RhodeCodeAuthP'
     Enables usage of authentication tokens for vcs operations.
     """
     uid = 'token'
+    AUTH_RESTRICTION_SCOPE_VCS = 'scope_vcs'
 
     def includeme(self, config):
         config.add_authn_plugin(self)
@@ -67,6 +70,9 b' class RhodeCodeAuthPlugin(RhodeCodeAuthP'
             route_name='auth_home',
             context=RhodecodeAuthnResource)
 
+    def get_settings_schema(self):
+        return RhodeCodeSettingsSchema()
+
     def get_display_name(self):
         return _('Rhodecode Token')
 
@@ -142,12 +148,10 b' class RhodeCodeAuthPlugin(RhodeCodeAuthP'
                     'user `%s` successfully authenticated via %s',
                     user_attrs['username'], self.name)
                 return user_attrs
-            log.warn(
-                'user `%s` failed to authenticate via %s, reason: bad or '
+            log.warning('user `%s` failed to authenticate via %s, reason: bad or '
                 'inactive token.', username, self.name)
         else:
-            log.warning(
-                'user `%s` failed to authenticate via %s, reason: account not '
+            log.warning('user `%s` failed to authenticate via %s, reason: account not '
                 'active.', username, self.name)
         return None
 
@@ -155,3 +159,19 b' class RhodeCodeAuthPlugin(RhodeCodeAuthP'
 def includeme(config):
     plugin_id = 'egg:rhodecode-enterprise-ce#{}'.format(RhodeCodeAuthPlugin.uid)
     plugin_factory(plugin_id).includeme(config)
+
+
+class RhodeCodeSettingsSchema(AuthnPluginSettingsSchemaBase):
+    auth_scope_choices = [
+        (RhodeCodeAuthPlugin.AUTH_RESTRICTION_SCOPE_VCS, 'VCS only'),
+    ]
+
+    scope_restriction = colander.SchemaNode(
+        colander.String(),
+        default=auth_scope_choices[0],
+        description=_('Choose operation scope restriction when authenticating.'),
+        title=_('Scope restriction'),
+        validator=colander.OneOf([x[0] for x in auth_scope_choices]),
+        widget='select_with_labels',
+        choices=auth_scope_choices
+    )

rhodecode/authentication/registry.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/routes.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/schema.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/tests/conftest.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/tests/functional/test_settings.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/authentication/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/conf.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2013-2018 RhodeCode GmbH
+# Copyright (C) 2013-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/environment.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/jsroutes.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/middleware.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -98,7 +98,7 b' def make_pyramid_app(global_config, **se'
     global_config = _substitute_values(global_config, environ)
     settings = _substitute_values(settings, environ)
 
-    sanitize_settings_and_apply_defaults(settings)
+    sanitize_settings_and_apply_defaults(global_config, settings)
 
     config = Configurator(settings=settings)
 
@@ -165,7 +165,7 b' def error_handler(exception, request):'
 
     error_explanation = base_response.explanation or str(base_response)
     if base_response.status_code == 404:
-        error_explanation += " Or you don't have permission to access it."
+        error_explanation += " Optionally you don't have permission to access this page."
     c = AttributeDict()
     c.error_message = base_response.status
     c.error_explanation = error_explanation
@@ -281,6 +281,7 b' def includeme(config):'
         config.include('rhodecode.apps.ops')
         config.include('rhodecode.apps.admin')
         config.include('rhodecode.apps.channelstream')
+        config.include('rhodecode.apps.file_store')
         config.include('rhodecode.apps.login')
         config.include('rhodecode.apps.home')
         config.include('rhodecode.apps.journal')
@@ -381,7 +382,7 b' def wrap_app_in_wsgi_middlewares(pyramid'
     return pyramid_app_with_cleanup
 
 
-def sanitize_settings_and_apply_defaults(settings):
+def sanitize_settings_and_apply_defaults(global_config, settings):
     """
     Applies settings defaults and does all type conversion.
 
@@ -420,6 +421,7 b' def sanitize_settings_and_apply_defaults'
     # TODO: johbo: Re-think this, usually the call to config.include
     # should allow to pass in a prefix.
     settings.setdefault('rhodecode.api.url', '/_admin/api')
+    settings.setdefault('__file__', global_config.get('__file__'))
 
     # Sanitize generic settings.
     _list_setting(settings, 'default_encoding', 'UTF-8')
@@ -708,18 +710,29 b' def _string_setting(settings, name, defa'
 
 
 def _substitute_values(mapping, substitutions):
+    result = {}
 
     try:
-        result = {
+        for key, value in mapping.items():
+            # initialize without substitution first
+            result[key] = value
+
             # Note: Cannot use regular replacements, since they would clash
             # with the implementation of ConfigParser. Using "format" instead.
-            key: value.format(**substitutions)
-            for key, value in mapping.items()
-        }
+            try:
+                result[key] = value.format(**substitutions)
     except KeyError as e:
-        raise ValueError(
-            'Failed to substitute env variable: {}. '
-            'Make sure you have specified this env variable without ENV_ prefix'.format(e))
+                env_var = '{}'.format(e.args[0])
+
+                msg = 'Failed to substitute: `{key}={{{var}}}` with environment entry. ' \
+                      'Make sure your environment has {var} set, or remove this ' \
+                      'variable from config file'.format(key=key, var=env_var)
+
+                if env_var.startswith('ENV_'):
+                    raise ValueError(msg)
+                else:
+                    log.warning(msg)
+
     except ValueError as e:
         log.warning('Failed to substitute ENV variable: %s', e)
         result = mapping

rhodecode/config/patches.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/__init__.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/examples/validate_commit_message_author.py

@@ -14,19 +14,6 b''
     'hook_type': '',
     'user_agent': 'Client user agent, e.g git or mercurial CLI version',
 })
-@has_kwargs({
-    'server_url': 'url of instance that triggered this hook',
-    'config': 'path to .ini config used',
-    'scm': 'type of version control "git", "hg", "svn"',
-    'username': 'username of actor who triggered this event',
-    'ip': 'ip address of actor who triggered this hook',
-    'action': '',
-    'repository': 'repository name',
-    'repo_store_path': 'full path to where repositories are stored',
-    'commit_ids': 'pre transaction metadata for commit ids',
-    'hook_type': '',
-    'user_agent': 'Client user agent, e.g git or mercurial CLI version',
-})
 def _pre_push_hook(*args, **kwargs):
     """
     Post push hook

rhodecode/config/rcextensions/helpers/__init__.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/helpers/extra_fields.py

@@ -1,5 +1,5 b''
 # -*- coding: utf-8 -*-
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/helpers/extract_post_commits.py

@@ -1,5 +1,5 b''
 # -*- coding: utf-8 -*-
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/helpers/extract_pre_commits.py

@@ -1,5 +1,5 b''
 # -*- coding: utf-8 -*-
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/helpers/http_call.py

@@ -1,5 +1,5 b''
 # -*- coding: utf-8 -*-
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/hooks.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/rcextensions/utils.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/routing_links.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/config/utils.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/events/__init__.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/events/base.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/events/interfaces.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/events/pullrequest.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/events/repo.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/events/repo_group.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/events/user.py

@@ -1,4 +1,4 b''
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/forms/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/i18n/de/LC_MESSAGES/rhodecode.po

@@ -7457,7 +7457,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/es/LC_MESSAGES/rhodecode.po

@@ -7455,7 +7455,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/fr/LC_MESSAGES/rhodecode.po

@@ -7456,7 +7456,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/it/LC_MESSAGES/rhodecode.po

@@ -7456,7 +7456,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/ja/LC_MESSAGES/rhodecode.po

@@ -7463,7 +7463,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/pl/LC_MESSAGES/rhodecode.po

@@ -7461,7 +7461,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/pt/LC_MESSAGES/rhodecode.po

@@ -7457,7 +7457,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/rhodecode.pot

@@ -7497,7 +7497,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/ru/LC_MESSAGES/rhodecode.po

@@ -7474,7 +7474,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/i18n/zh/LC_MESSAGES/rhodecode.po

@@ -7458,7 +7458,7 b' msgid ""'
 "Schema of clone url construction eg. '{scheme}://{user}@{netloc}/{repo}', available vars:\n"
 "                       {scheme} 'http' or 'https' sent from running RhodeCode server,\n"
 "                       {user}   current user username,\n"
-"                       {sys_user} current system user running this process, usefull for ssh,\n"
+"                       {sys_user} current system user running this process, Useful for ssh,\n"
 "                       {hostname} hostname of this server running RhodeCode,\n"
 "                       {netloc} network location/server host of running RhodeCode server,\n"
 "                       {repo}   full repository name,\n"

rhodecode/integrations/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/registry.py

@@ -1,5 +1,5 b''
 # -*- coding: utf-8 -*-
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/routes.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/schema.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/tests/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2016-2018 RhodeCode GmbH
+# Copyright (C) 2016-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/tests/test_integrations.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/types/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/types/base.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -23,17 +23,27 b' import string'
 import collections
 import logging
 import requests
+import urllib
 from requests.adapters import HTTPAdapter
 from requests.packages.urllib3.util.retry import Retry
 
 from mako import exceptions
 
+from rhodecode.lib.utils2 import safe_str
 from rhodecode.translation import _
 
 
 log = logging.getLogger(__name__)
 
 
+class UrlTmpl(string.Template):
+
+    def safe_substitute(self, **kws):
+        # url encode the kw for usage in url
+        kws = {k: urllib.quote(safe_str(v)) for k, v in kws.items()}
+        return super(UrlTmpl, self).safe_substitute(**kws)
+
+
 class IntegrationTypeBase(object):
     """ Base class for IntegrationType plugins """
     is_dummy = False
@@ -217,7 +227,9 b' class WebhookDataHandler(CommitParsingDa'
         common_vars.update(extra_vars)
 
         template_url = self.template_url.replace('${extra:', '${extra__')
-        return string.Template(template_url).safe_substitute(**common_vars)
+        for k, v in common_vars.items():
+            template_url = UrlTmpl(template_url).safe_substitute(**{k: v})
+        return template_url
 
     def repo_push_event_handler(self, event, data):
         url = self.get_base_parsed_template(data)
@@ -228,20 +240,18 b' class WebhookDataHandler(CommitParsingDa'
         if '${branch}' in url or '${branch_head}' in url or '${commit_id}' in url:
             # call it multiple times, for each branch if used in variables
             for branch, commit_ids in branches_commits.items():
-                branch_url = string.Template(url).safe_substitute(branch=branch)
+                branch_url = UrlTmpl(url).safe_substitute(branch=branch)
 
                 if '${branch_head}' in branch_url:
                     # last commit in the aggregate is the head of the branch
                     branch_head = commit_ids['branch_head']
-                    branch_url = string.Template(branch_url).safe_substitute(
-                        branch_head=branch_head)
+                    branch_url = UrlTmpl(branch_url).safe_substitute(branch_head=branch_head)
 
                 # call further down for each commit if used
                 if '${commit_id}' in branch_url:
                     for commit_data in commit_ids['commits']:
                         commit_id = commit_data['raw_id']
-                        commit_url = string.Template(branch_url).safe_substitute(
-                            commit_id=commit_id)
+                        commit_url = UrlTmpl(branch_url).safe_substitute(commit_id=commit_id)
                         # register per-commit call
                         log.debug(
                             'register %s call(%s) to url %s',
@@ -251,36 +261,34 b' class WebhookDataHandler(CommitParsingDa'
 
                 else:
                     # register per-branch call
-                    log.debug(
-                        'register %s call(%s) to url %s',
+                    log.debug('register %s call(%s) to url %s',
                         self.name, event, branch_url)
-                    url_calls.append(
-                        (branch_url, self.headers, data))
+                    url_calls.append((branch_url, self.headers, data))
 
         else:
-            log.debug(
-                'register %s call(%s) to url %s', self.name, event, url)
+            log.debug('register %s call(%s) to url %s', self.name, event, url)
             url_calls.append((url, self.headers, data))
 
         return url_calls
 
     def repo_create_event_handler(self, event, data):
         url = self.get_base_parsed_template(data)
-        log.debug(
-            'register %s call(%s) to url %s', self.name, event, url)
+        log.debug('register %s call(%s) to url %s', self.name, event, url)
         return [(url, self.headers, data)]
 
     def pull_request_event_handler(self, event, data):
         url = self.get_base_parsed_template(data)
-        log.debug(
-            'register %s call(%s) to url %s', self.name, event, url)
-        url = string.Template(url).safe_substitute(
-            pull_request_id=data['pullrequest']['pull_request_id'],
-            pull_request_title=data['pullrequest']['title'],
-            pull_request_url=data['pullrequest']['url'],
-            pull_request_shadow_url=data['pullrequest']['shadow_url'],
-            pull_request_commits_uid=data['pullrequest']['commits_uid'],
-        )
+        log.debug('register %s call(%s) to url %s', self.name, event, url)
+        pr_vars = [
+            ('pull_request_id', data['pullrequest']['pull_request_id']),
+            ('pull_request_title', data['pullrequest']['title']),
+            ('pull_request_url', data['pullrequest']['url']),
+            ('pull_request_shadow_url', data['pullrequest']['shadow_url']),
+            ('pull_request_commits_uid', data['pullrequest']['commits_uid']),
+        ]
+        for k, v in pr_vars:
+            url = UrlTmpl(url).safe_substitute(**{k: v})
+
         return [(url, self.headers, data)]
 
     def __call__(self, event, data):

rhodecode/integrations/types/email.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/integrations/types/hipchat.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -122,8 +122,10 b' class HipchatIntegrationType(Integration'
             log.debug('event not valid: %r', event)
             return
 
-        if event.name not in self.settings['events']:
-            log.debug('event ignored: %r', event)
+        allowed_events = self.settings['events']
+        if event.name not in allowed_events:
+            log.debug('event ignored: %r event %s not in allowed events %s',
+                      event, event.name, allowed_events)
             return
 
         data = event.as_dict()

rhodecode/integrations/types/slack.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -138,8 +138,10 b' class SlackIntegrationType(IntegrationTy'
             log.debug('event not valid: %r', event)
             return
 
-        if event.name not in self.settings['events']:
-            log.debug('event ignored: %r', event)
+        allowed_events = self.settings['events']
+        if event.name not in allowed_events:
+            log.debug('event ignored: %r event %s not in allowed events %s',
+                      event, event.name, allowed_events)
             return
 
         data = event.as_dict()

rhodecode/integrations/types/webhook.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -118,12 +118,11 b' class WebhookSettingsSchema(colander.Sch'
     method_type = colander.SchemaNode(
         colander.String(),
         title=_('Call Method'),
-        description=_('Select if the Webhook call should be made '
-                      'with POST or GET.'),
+        description=_('Select a HTTP method to use when calling the Webhook.'),
         default='post',
         missing='',
         widget=deform.widget.RadioChoiceWidget(
-            values=[('get', 'GET'), ('post', 'POST')],
+            values=[('get', 'GET'), ('post', 'POST'), ('put', 'PUT')],
             inline=True
         ),
     )
@@ -171,8 +170,10 b' class WebhookIntegrationType(Integration'
             log.debug('event not valid: %r', event)
             return
 
-        if event.name not in self.settings['events']:
-            log.debug('event ignored: %r', event)
+        allowed_events = self.settings['events']
+        if event.name not in allowed_events:
+            log.debug('event ignored: %r event %s not in allowed events %s',
+                      event, event.name, allowed_events)
             return
 
         data = event.as_dict()
@@ -187,8 +188,7 b' class WebhookIntegrationType(Integration'
         handler = WebhookDataHandler(template_url, headers)
 
         url_calls = handler(event, data)
-        log.debug('webhook: calling following urls: %s',
-                  [x[0] for x in url_calls])
+        log.debug('Webhook: calling following urls: %s', [x[0] for x in url_calls])
 
         run_task(post_to_webhook, url_calls, self.settings)
 
@@ -214,7 +214,9 b' def post_to_webhook(url_calls, settings)'
             'parents': [{'raw_id': '431b772a5353dad9974b810dd3707d79e3a7f6e0'}],
             'permalink_url': u'http://rc.local:8080/_7/changeset/a815cc738b9651eb5ffbcfb1ce6ccd7c701a5ddf',
             'raw_id': 'a815cc738b9651eb5ffbcfb1ce6ccd7c701a5ddf',
-            'refs': {'bookmarks': [], 'branches': [u'default'], 'tags': [u'tip']},
+                               'refs': {'bookmarks': [],
+                                        'branches': [u'default'],
+                                        'tags': [u'tip']},
             'reviewers': [],
             'revision': 9L,
             'short_id': 'a815cc738b96',
@@ -228,11 +230,11 b' def post_to_webhook(url_calls, settings)'
           'url': u'http://rc.local:8080/hg-repo'},
          'server_url': u'http://rc.local:8080',
          'utc_timestamp': datetime.datetime(2017, 11, 30, 13, 0, 1, 569276)
+         }
+    """
 
-    """
     call_headers = {
-        'User-Agent': 'RhodeCode-webhook-caller/{}'.format(
-            rhodecode.__version__)
+        'User-Agent': 'RhodeCode-webhook-caller/{}'.format(rhodecode.__version__)
     }  # updated below with custom ones, allows override
 
     auth = get_auth(settings)
@@ -247,8 +249,7 b' def post_to_webhook(url_calls, settings)'
         headers = headers or {}
         call_headers.update(headers)
 
-        log.debug('calling Webhook with method: %s, and auth:%s',
-                  call_method, auth)
+        log.debug('calling Webhook with method: %s, and auth:%s', call_method, auth)
         if settings.get('log_data'):
             log.debug('calling webhook with data: %s', data)
         resp = call_method(url, json={

rhodecode/integrations/views.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2012-2018 RhodeCode GmbH
+# Copyright (C) 2012-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/lib/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/lib/action_parser.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/lib/audit_logger.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2017-2018 RhodeCode GmbH
+# Copyright (C) 2017-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/lib/auth.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/lib/base.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
@@ -35,6 +35,7 b' from paste.httpexceptions import HTTPUna'
 from paste.httpheaders import WWW_AUTHENTICATE, AUTHORIZATION
 
 import rhodecode
+from rhodecode.apps._base import TemplateArgs
 from rhodecode.authentication.base import VCS_TYPE
 from rhodecode.lib import auth, utils2
 from rhodecode.lib import helpers as h
@@ -43,7 +44,7 b' from rhodecode.lib.exceptions import Use'
 from rhodecode.lib.utils import (password_changed, get_enabled_hook_classes)
 from rhodecode.lib.utils2 import (
     str2bool, safe_unicode, AttributeDict, safe_int, sha1, aslist, safe_str)
-from rhodecode.model.db import Repository, User, ChangesetComment
+from rhodecode.model.db import Repository, User, ChangesetComment, UserBookmark
 from rhodecode.model.notification import NotificationModel
 from rhodecode.model.settings import VcsSettingsModel, SettingsModel
 
@@ -281,7 +282,7 b' def get_current_lang(request):'
     return getattr(request, '_LOCALE_', request.locale_name)
 
 
-def attach_context_attributes(context, request, user_id):
+def attach_context_attributes(context, request, user_id=None):
     """
     Attach variables into template context called `c`.
     """
@@ -312,6 +313,10 b' def attach_context_attributes(context, r'
         rc_config.get('rhodecode_dashboard_items', 100))
     context.visual.admin_grid_items = safe_int(
         rc_config.get('rhodecode_admin_grid_items', 100))
+    context.visual.show_revision_number = str2bool(
+        rc_config.get('rhodecode_show_revision_number', True))
+    context.visual.show_sha_length = safe_int(
+        rc_config.get('rhodecode_show_sha_length', 100))
     context.visual.repository_fields = str2bool(
         rc_config.get('rhodecode_repository_fields'))
     context.visual.show_version = str2bool(
@@ -343,6 +348,8 b' def attach_context_attributes(context, r'
         config.get('labs_settings_active', 'false'))
     context.ssh_enabled = str2bool(
         config.get('ssh.generate_authorized_keyfile', 'false'))
+    context.ssh_key_generator_enabled = str2bool(
+        config.get('ssh.enable_ui_key_generator', 'true'))
 
     context.visual.allow_repo_location_change = str2bool(
         config.get('allow_repo_location_change', True))
@@ -417,7 +424,13 b' def attach_context_attributes(context, r'
     context.csrf_token = auth.get_csrf_token(session=request.session)
     context.backends = rhodecode.BACKENDS.keys()
     context.backends.sort()
-    context.unread_notifications = NotificationModel().get_unread_cnt_for_user(user_id)
+    unread_count = 0
+    user_bookmark_list = []
+    if user_id:
+        unread_count = NotificationModel().get_unread_cnt_for_user(user_id)
+        user_bookmark_list = UserBookmark.get_bookmarks_for_user(user_id)
+    context.unread_notifications = unread_count
+    context.bookmark_items = user_bookmark_list
 
     # web case
     if hasattr(request, 'user'):
@@ -551,7 +564,11 b' def bootstrap_request(**kwargs):'
             from rhodecode.lib.partial_renderer import get_partial_renderer
             return get_partial_renderer(request=self, tmpl_name=tmpl_name)
 
-        _call_context = {}
+        _call_context = TemplateArgs()
+        _call_context.visual = TemplateArgs()
+        _call_context.visual.show_sha_length = 12
+        _call_context.visual.show_revision_number = True
+
         @property
         def call_context(self):
             return self._call_context

rhodecode/lib/caching_query.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3

rhodecode/lib/celerylib/__init__.py

@@ -1,6 +1,6 b''
 # -*- coding: utf-8 -*-
 
-# Copyright (C) 2010-2018 RhodeCode GmbH
+# Copyright (C) 2010-2019 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3