release: Merge default into stable for release preparation

docs/admin/tuning-change-lfs-dir.rst

@@ -0,0 +1,14 b''
+.. _git-lfs-loc:
+
+Change the |git| LFS storage Location
+-------------------------------------
+
+|RCE| manages |git| LFS files from the following default location
+:file:`/home/{user}/repos/.cache/lfs_store`. If you wish to change this, use
+the following steps:
+
+1. Open :menuselection:`Admin --> Settings --> VCS` as super-admin.
+
+In section called `Git Settings` you can change where the LFS
+objects should be stored.
+

docs/release-notes/release-notes-4.7.0.rst

@@ -0,0 +1,162 b''
+|RCE| 4.7.0 |RNS|
+-----------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2017-04-08
+
+
+New Features
+^^^^^^^^^^^^
+
+- Git: added support for Git LFS v2 protocol. RhodeCode now supports both
+  Mercurial Largefiles, and Git LFS for storing large binaries.
+- Largefiles: detect Git LFS or Mercurial Largefiles objects in UI.
+  Those are now available for downloading together with showing their size.
+- Files: Jupyter notebooks will be now rendered inside the file view. Including
+  MatJax support, and relative images.
+- Files: render images inside the file view.
+  Instead of displaying binary message, render images icons and gifs
+  inside the file view page.
+- Files: relative ULR support inside rendered files. It's now possible to
+  write Markup files and relative links will be handled from the RhodeCode
+  instance itself. Adds basic wiki functionality.
+- Files: allow to show inline pdf in browser using embedded files from source code.
+- Annotation: added shortcut links to browse the annotation view with previous
+  commits. Allows browsing history for each line from annotation view.
+- Pull Requests: add explicit close action instead of close with status from
+  status selector. This allows closing of approved or rejected
+  pull requests, without performing a merge action.
+- Authentication: LDAP now has an option to sync LDA groups using two
+  distinct ways. Either using rfc2307 or rfc2307bis. Increases compatibility
+  with different OpenLDAP and AD servers.
+- Slack: updated slack integration to use the attachments for nicer formatting.
+  Added number of commits inside the message, changed UI for all Slack events.
+- Authentication (EE edition only): added repository scope for VCS type auth
+  tokens. Each token can be now bound to particular repository for added security.
+- User administration: added audit page to allow showing single user actions.
+- API: implemented `get_user_audit_logs` method to fetch audit logs via API endpoint.
+- User administration: It's now possible to edit user group membership from
+  user view.
+- User groups administration: added managing and showing the group
+  synchronization in UI. It's now possible to enable manual group syncing on
+  already existing user groups from external sources such as LDAP/AD.
+- Repositories: added new strip view allowing removing commits from repositories
+  via web interface for repository administrators.
+- System Info: added info about workers and worker type.
+  Added more details about CPU. Expose workers of VCSServer in system info data.
+  Detect database migration errors.
+
+
+General
+^^^^^^^
+
+- Core: ported many views into pure pyramid code with python3.6 compatibility.
+- Core: removed deprecated Pyro4 backend from Enterprise code.
+- Maintenance: implemented maintenance view for Mercurial and GIT repositories.
+  For HG it will run `hg verify`, and for GIT a `git gc` command.
+- Notifications: different approach with fixed/standard container. Floating
+  notifications no longer hide the menu when browsed on top of the page.
+  Also added option to remove single elements from stacked notifications.
+- VCS server: exception-handling: better handling of remote exception and logging.
+- VCS server: propagate hooks tracebacks to VCS server for easier debugging.
+- Core: prevent `httplib3` logs to spam internal RhodeCode logs.
+  It often confuses people looking at those entries, misleading during debug.
+- Mercurial: allow editing Largefiles store location from web interface.
+- Git: allow editing GIT LFS store location from web interface.
+- API: add get_method API call. This allows showing the method and it's parameter
+  from the CLI without reading the documentation.
+  In addition use it's mechanics to propose users other methods with close names
+  if the calling method is not found.
+- UI: add timezone info into tooltips.
+- Dependencies: bumped pyramid to 1.7.4
+- Dependencies: bumped Mercurial version to 4.1.2
+
+
+Security
+^^^^^^^^
+
+- Hooks: added changes to propagate commit metadata on pre-push.
+  This allows easier implementation of checking hooks such as branch protection.
+- Hooks: added new pretx hook to allow mercurial checks such as protected
+  branches, or force push.
+- Auth: give owner of user group proper admin permissions to the user group.
+  This makes the behaviour consistent with repositories and repository groups.
+  And allows delegation of administration of those to other users.
+- Password reset: strengthen security on password reset logic.
+  Generate token that has special password reset role.
+  Set 10 minute expiration for the token.
+  Add some logic to prevent brute forcing attacks.
+  Use more implicit messages to prevent user email discovery attacks.
+- Core: added checks for password change for authenticated users in pure
+  Pyramid views. 2 views were still available and not forcing users to change
+  their passwords.
+- Auth tokens: removed builtin auth-token for users.
+  Builtin token were non-removable, and always generated for new users. This
+  wasn't best practice for security as some users are strictly not allowed to
+  use tokens. From now on new users needs a new token generation in case they
+  want to use token based authentication.
+- Auth tokens: don't generate builtin token for new users.
+  Also don't change them when password reset is made.
+- Api: added last-activity into returned data of get_user api.
+
+
+Performance
+^^^^^^^^^^^
+
+- Mercurial: enabled new `Zstandard` compression algorithm available with
+  Mercurial 4.1.X. This allows faster, more CPU efficient clones when used
+  with new Mercurial clients.
+
+- Users Admin: moved user admin to pyramid, and made it load users in chunks.
+  Fixed loading data to be lazy fetched, drastically improves speed of user
+  administration page in case of large amount of users.
+
+
+Fixes
+^^^^^
+
+- Search: goto commit search will now use a safe search option and never
+  throw any exceptions even if search is misconfigured
+  e.g. Elastic Search cluster is down.
+- Events: fix a case for events called from API that couldn't fetch
+  registered user object.
+- Comments: unlock submit if we use slash commands to set status.
+- UI: fixed an issue with date of last change was not displayed correctly.
+- Emails: added comment types (TODO/NOTE) into emails.
+- Events: fix wrongly returned author data.
+- Error middleware: read the instance title from cached object.
+  Reading from settings inside error handler can cause error hiding when
+  error_handler was caused by database errors.
+- Pull requests: show version age component should use local dates instead of UTC.
+- Pull requests: lock button when updating reviewers to forbid multi-submit
+  problems. Additionally fixed some small UI issues found in that view.
+- Pull requests: forbid browsing versions on closed pull request.
+- Pull requests: allow super-admins to delete pull requests instead of only owners.
+- Diffs: support mercurial copy operation in diffs details.
+- SVN: escape special chars to allow interactions with non-standard svn paths.
+  Path with special characters such as '#' will no longer trigger 404 errors.
+- Data grids: fix some styling and processing text display.
+- API: use consistent way to extract users, repos, repo groups and user groups
+  by id or name. Makes usage of Number vs String to differentiate if we pick
+  object ID or it's name this will allow editing of objects by either id or
+  it's name, including numeric string names.
+- API: validate commit_id when using commit_comment API
+- API: cleanup sessions enforce older_then must be a valid INT.
+
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+- Auth-tokens: a builtin token will be migrated for all users into a custom
+  external token. We advise to inform users that the current builtin tokens
+  will now show as external ones. Builtin tokens were removed to allow expiring
+  ,or removing them. It's now possible to create users without any tokens.
+
+  From now on new users needs a new token generation in case they want to use
+  token based authentication.
+
+- Hooks: we added via migration a pre transaction hook for Mercurial. If you're
+  using a custom code inside pre-push function of rcextensions make sure it
+  will not block your pushes.

docs/tutorials/git-lfs-ext.rst

@@ -0,0 +1,93 b''
+.. _git-lfs-files:
+
+|git| LFS Extension
+===================
+
+
+Git Large File Storage (or LFS) is a new, open-source extension to Git that
+aims to improve handling of large files. It does this by replacing large files
+in your repository—such as graphics and videos—with simple text pointers.
+|RC| Server includes an embedded LFS object store server, allowing storage of
+large files without the need for an external object store.
+Git LFS is disabled by default, globally, and for each individual repository.
+
+.. note::
+
+    |RC| implements V2 API of Git LFS. Please make sure your git client is
+    using the latest version (2.0.X recommended) to leverage full feature set
+    of the V2 API.
+
+
+
+Enabling Git LFS
+++++++++++++++++
+
+Git LFS is disabled by default within |RC| Server.
+
+To enable Git LFS Globally:
+
+    - Go to :menuselection:`Admin --> Settings --> VCS`
+
+    - Scroll down into `Git settings`
+
+    - Tick `Enable lfs extension`
+
+    - Save your settings.
+
+Those settings apply globally to each repository that inherits from the defaults
+You can leave `lfs extension` disabled globally, and only enable it per
+repository that would use the lfs.
+
+
+.. note::
+
+    You might want to adjust the global storage location at that point, however
+    we recommend leaving the default one created.
+
+
+Installing and using the Git LFS command line client
+++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Git LFS aims to integrate with the standard Git workflow as seamlessly
+as possible. To push your first Git LFS files to an existing repository
+Download and install the git-lfs command line client
+Install the Git LFS filters::
+
+    git lfs install
+
+This adds the following lines to the .gitconfig file located in your home directory::
+
+    [filter "lfs"]
+        clean = git-lfs clean %f
+        smudge = git-lfs smudge %f
+        required = true
+
+The above change applies globally, so it is not necessary to run this for
+each repository you work with. Choose the file types you would like LFS to
+handle by executing the git lfs track command. The git lfs track command
+creates or updates the .gitattributes file in your repository.
+Change to your cloned repository, then execute git add to ensure updates
+to the .gitattributes are later committed::
+
+    git lfs track "*.jpg"
+    git add .gitattributes
+
+Add, commit, and push your changes as you normally would::
+
+    git add image.jpg
+    git commit -m "Added an image"
+    git push
+
+When pushed, the Git tree updates to include a pointer to the file actual
+file content. This pointer will include the SHA256 hash of the object and its
+size in bytes. For example::
+
+    oid sha256:4fa32d6f9b1461c4a53618a47324ee43e36ce7ceaea2ad440cc811a7e6881be1
+    size 2580390
+
+
+The object itself will be uploaded to a separate location via the Git LFS Batch API.
+The transfer is validated and authorized by |RC| server itself.
+
+If give repository has Git LFS disabled, a proper message will be sent back to
+the client and upload of LFS objects will be forbidden.

pkgs/patch-celery-dateutil.diff

@@ -0,0 +1,12 b''
+diff -rup celery-2.2.10-orig/setup.py celery-2.2.10/setup.py
+--- celery-2.2.10-orig/setup.py	2017-02-25 15:30:34.000000000 +0100
++++ celery-2.2.10/setup.py	2017-02-25 15:30:34.000000000 +0100
+@@ -48,7 +48,7 @@ try:
+ except ImportError:
+     install_requires.append("importlib")
+ install_requires.extend([
+-    "python-dateutil>=1.5.0,<2.0.0",
++    "python-dateutil>=1.5.0,<2.2.0",
+     "anyjson>=0.3.1",
+     "kombu>=1.1.2,<2.0.0",
+     "pyparsing>=1.5.0,<2.0.0",

rhodecode/api/tests/test_get_method.py

@@ -0,0 +1,58 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+import pytest
+
+from rhodecode.api.tests.utils import build_data, api_call, assert_ok
+
+
[e-mail]("testuser_api", "app")
+class TestGetMethod(object):
+    def test_get_methods_no_matches(self):
+        id_, params = build_data(self.apikey, 'get_method', pattern='hello')
+        response = api_call(self.app, params)
+
+        expected = []
+        assert_ok(id_, expected, given=response.body)
+
+    def test_get_methods(self):
+        id_, params = build_data(self.apikey, 'get_method', pattern='*comment*')
+        response = api_call(self.app, params)
+
+        expected = ['changeset_comment', 'comment_pull_request',
+                    'comment_commit']
+        assert_ok(id_, expected, given=response.body)
+
+    def test_get_methods_on_single_match(self):
+        id_, params = build_data(self.apikey, 'get_method', pattern='*comment_commit*')
+        response = api_call(self.app, params)
+
+        expected = ['comment_commit',
+                    {'apiuser': '<RequiredType>',
+                     'comment_type': "<Optional:u'note'>",
+                     'commit_id': '<RequiredType>',
+                     'message': '<RequiredType>',
+                     'repoid': '<RequiredType>',
+                     'request': '<RequiredType>',
+                     'resolves_comment_id': '<Optional:None>',
+                     'status': '<Optional:None>',
+                     'userid': '<Optional:<OptionalAttr:apiuser>>'}]
+        assert_ok(id_, expected, given=response.body)

rhodecode/apps/_base/__init__.py

@@ -0,0 +1,163 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import time
+import logging
+from pylons import tmpl_context as c
+from pyramid.httpexceptions import HTTPFound
+
+from rhodecode.lib import helpers as h
+from rhodecode.lib.utils2 import StrictAttributeDict, safe_int
+from rhodecode.model import repo
+from rhodecode.model.db import User
+from rhodecode.model.scm import ScmModel
+
+log = logging.getLogger(__name__)
+
+
+ADMIN_PREFIX = '/_admin'
+STATIC_FILE_PREFIX = '/_static'
+
+
+class TemplateArgs(StrictAttributeDict):
+    pass
+
+
+class BaseAppView(object):
+
+    def __init__(self, context, request):
+        self.request = request
+        self.context = context
+        self.session = request.session
+        self._rhodecode_user = request.user  # auth user
+        self._rhodecode_db_user = self._rhodecode_user.get_instance()
+        self._maybe_needs_password_change(
+            request.matched_route.name, self._rhodecode_db_user)
+
+    def _maybe_needs_password_change(self, view_name, user_obj):
+        log.debug('Checking if user %s needs password change on view %s',
+                  user_obj, view_name)
+        skip_user_views = [
+            'logout', 'login',
+            'my_account_password', 'my_account_password_update'
+        ]
+
+        if not user_obj:
+            return
+
+        if user_obj.username == User.DEFAULT_USER:
+            return
+
+        now = time.time()
+        should_change = user_obj.user_data.get('force_password_change')
+        change_after = safe_int(should_change) or 0
+        if should_change and now > change_after:
+            log.debug('User %s requires password change', user_obj)
+            h.flash('You are required to change your password', 'warning',
+                    ignore_duplicate=True)
+
+            if view_name not in skip_user_views:
+                raise HTTPFound(
+                    self.request.route_path('my_account_password'))
+
+    def _get_local_tmpl_context(self):
+        c = TemplateArgs()
+        c.auth_user = self.request.user
+        return c
+
+    def _register_global_c(self, tmpl_args):
+        """
+        Registers attributes to pylons global `c`
+        """
+        # TODO(marcink): remove once pyramid migration is finished
+        for k, v in tmpl_args.items():
+            setattr(c, k, v)
+
+    def _get_template_context(self, tmpl_args):
+        self._register_global_c(tmpl_args)
+
+        local_tmpl_args = {
+            'defaults': {},
+            'errors': {},
+        }
+        local_tmpl_args.update(tmpl_args)
+        return local_tmpl_args
+
+    def load_default_context(self):
+        """
+        example:
+
+        def load_default_context(self):
+            c = self._get_local_tmpl_context()
+            c.custom_var = 'foobar'
+            self._register_global_c(c)
+            return c
+        """
+        raise NotImplementedError('Needs implementation in view class')
+
+
+class RepoAppView(BaseAppView):
+
+    def __init__(self, context, request):
+        super(RepoAppView, self).__init__(context, request)
+        self.db_repo = request.db_repo
+        self.db_repo_name = self.db_repo.repo_name
+        self.db_repo_pull_requests = ScmModel().get_pull_requests(self.db_repo)
+
+    def _get_local_tmpl_context(self):
+        c = super(RepoAppView, self)._get_local_tmpl_context()
+        # register common vars for this type of view
+        c.rhodecode_db_repo = self.db_repo
+        c.repo_name = self.db_repo_name
+        c.repository_pull_requests = self.db_repo_pull_requests
+        return c
+
+
+class RepoRoutePredicate(object):
+    def __init__(self, val, config):
+        self.val = val
+
+    def text(self):
+        return 'repo_route = %s' % self.val
+
+    phash = text
+
+    def __call__(self, info, request):
+        repo_name = info['match']['repo_name']
+        repo_model = repo.RepoModel()
+        by_name_match = repo_model.get_by_repo_name(repo_name, cache=True)
+        # if we match quickly from database, short circuit the operation,
+        # and validate repo based on the type.
+        if by_name_match:
+            # register this as request object we can re-use later
+            request.db_repo = by_name_match
+            return True
+
+        by_id_match = repo_model.get_repo_by_id(repo_name)
+        if by_id_match:
+            request.db_repo = by_id_match
+            return True
+
+        return False
+
+
+def includeme(config):
+    config.add_route_predicate(
+        'repo_route', RepoRoutePredicate)

rhodecode/apps/admin/tests/test_admin_users.py

@@ -0,0 +1,142 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import pytest
+
+from rhodecode.model.db import User, UserApiKeys
+
+from rhodecode.tests import (
+    TestController, TEST_USER_REGULAR_LOGIN, assert_session_flash)
+from rhodecode.tests.fixture import Fixture
+
+fixture = Fixture()
+
+
+def route_path(name, params=None, **kwargs):
+    import urllib
+    from rhodecode.apps._base import ADMIN_PREFIX
+
+    base_url = {
+        'users':
+            ADMIN_PREFIX + '/users',
+        'users_data':
+            ADMIN_PREFIX + '/users_data',
+        'edit_user_auth_tokens':
+            ADMIN_PREFIX + '/users/{user_id}/edit/auth_tokens',
+        'edit_user_auth_tokens_add':
+            ADMIN_PREFIX + '/users/{user_id}/edit/auth_tokens/new',
+        'edit_user_auth_tokens_delete':
+            ADMIN_PREFIX + '/users/{user_id}/edit/auth_tokens/delete',
+    }[name].format(**kwargs)
+
+    if params:
+        base_url = '{}?{}'.format(base_url, urllib.urlencode(params))
+    return base_url
+
+
+class TestAdminUsersView(TestController):
+
+    def test_show_users(self):
+        self.log_user()
+        self.app.get(route_path('users'))
+
+    def test_show_users_data(self, xhr_header):
+        self.log_user()
+        response = self.app.get(route_path(
+            'users_data'), extra_environ=xhr_header)
+
+        all_users = User.query().filter(
+            User.username != User.DEFAULT_USER).count()
+        assert response.json['recordsTotal'] == all_users
+
+    def test_show_users_data_filtered(self, xhr_header):
+        self.log_user()
+        response = self.app.get(route_path(
+            'users_data', params={'search[value]': 'empty_search'}),
+            extra_environ=xhr_header)
+
+        all_users = User.query().filter(
+            User.username != User.DEFAULT_USER).count()
+        assert response.json['recordsTotal'] == all_users
+        assert response.json['recordsFiltered'] == 0
+
+    def test_auth_tokens_default_user(self):
+        self.log_user()
+        user = User.get_default_user()
+        response = self.app.get(
+            route_path('edit_user_auth_tokens', user_id=user.user_id),
+            status=302)
+
+    def test_auth_tokens(self):
+        self.log_user()
+
+        user = User.get_by_username(TEST_USER_REGULAR_LOGIN)
+        response = self.app.get(
+            route_path('edit_user_auth_tokens', user_id=user.user_id))
+        for token in user.auth_tokens:
+            response.mustcontain(token)
+            response.mustcontain('never')
+
+    @pytest.mark.parametrize("desc, lifetime", [
+        ('forever', -1),
+        ('5mins', 60*5),
+        ('30days', 60*60*24*30),
+    ])
+    def test_add_auth_token(self, desc, lifetime, user_util):
+        self.log_user()
+        user = user_util.create_user()
+        user_id = user.user_id
+
+        response = self.app.post(
+            route_path('edit_user_auth_tokens_add', user_id=user_id),
+            {'description': desc, 'lifetime': lifetime,
+             'csrf_token': self.csrf_token})
+        assert_session_flash(response, 'Auth token successfully created')
+
+        response = response.follow()
+        user = User.get(user_id)
+        for auth_token in user.auth_tokens:
+            response.mustcontain(auth_token)
+
+    def test_delete_auth_token(self, user_util):
+        self.log_user()
+        user = user_util.create_user()
+        user_id = user.user_id
+        keys = user.extra_auth_tokens
+        assert 2 == len(keys)
+
+        response = self.app.post(
+            route_path('edit_user_auth_tokens_add', user_id=user_id),
+            {'description': 'desc', 'lifetime': -1,
+             'csrf_token': self.csrf_token})
+        assert_session_flash(response, 'Auth token successfully created')
+        response.follow()
+
+        # now delete our key
+        keys = UserApiKeys.query().filter(UserApiKeys.user_id == user_id).all()
+        assert 3 == len(keys)
+
+        response = self.app.post(
+            route_path('edit_user_auth_tokens_delete', user_id=user_id),
+            {'del_auth_token': keys[0].api_key, 'csrf_token': self.csrf_token})
+
+        assert_session_flash(response, 'Auth token successfully deleted')
+        keys = UserApiKeys.query().filter(UserApiKeys.user_id == user_id).all()
+        assert 2 == len(keys)

rhodecode/apps/admin/views/users.py

@@ -0,0 +1,317 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import logging
+
+from pyramid.httpexceptions import HTTPFound
+from pyramid.view import view_config
+
+from rhodecode.lib.helpers import Page
+from rhodecode_tools.lib.ext_json import json
+
+from rhodecode.apps._base import BaseAppView
+from rhodecode.lib.auth import (
+    LoginRequired, HasPermissionAllDecorator, CSRFRequired)
+from rhodecode.lib import helpers as h
+from rhodecode.lib.utils import PartialRenderer
+from rhodecode.lib.utils2 import safe_int, safe_unicode
+from rhodecode.model.auth_token import AuthTokenModel
+from rhodecode.model.user import UserModel
+from rhodecode.model.user_group import UserGroupModel
+from rhodecode.model.db import User, or_
+from rhodecode.model.meta import Session
+
+log = logging.getLogger(__name__)
+
+
+class AdminUsersView(BaseAppView):
+    ALLOW_SCOPED_TOKENS = False
+    """
+    This view has alternative version inside EE, if modified please take a look
+    in there as well.
+    """
+
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+        c.allow_scoped_tokens = self.ALLOW_SCOPED_TOKENS
+        self._register_global_c(c)
+        return c
+
+    def _redirect_for_default_user(self, username):
+        _ = self.request.translate
+        if username == User.DEFAULT_USER:
+            h.flash(_("You can't edit this user"), category='warning')
+            # TODO(marcink): redirect to 'users' admin panel once this
+            # is a pyramid view
+            raise HTTPFound('/')
+
+    def _extract_ordering(self, request):
+        column_index = safe_int(request.GET.get('order[0][column]'))
+        order_dir = request.GET.get(
+            'order[0][dir]', 'desc')
+        order_by = request.GET.get(
+            'columns[%s][data][sort]' % column_index, 'name_raw')
+
+        # translate datatable to DB columns
+        order_by = {
+            'first_name': 'name',
+            'last_name': 'lastname',
+        }.get(order_by) or order_by
+
+        search_q = request.GET.get('search[value]')
+        return search_q, order_by, order_dir
+
+    def _extract_chunk(self, request):
+        start = safe_int(request.GET.get('start'), 0)
+        length = safe_int(request.GET.get('length'), 25)
+        draw = safe_int(request.GET.get('draw'))
+        return draw, start, length
+
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        route_name='users', request_method='GET',
+        renderer='rhodecode:templates/admin/users/users.mako')
+    def users_list(self):
+        c = self.load_default_context()
+        return self._get_template_context(c)
+
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        # renderer defined below
+        route_name='users_data', request_method='GET', renderer='json',
+        xhr=True)
+    def users_list_data(self):
+        draw, start, limit = self._extract_chunk(self.request)
+        search_q, order_by, order_dir = self._extract_ordering(self.request)
+
+        _render = PartialRenderer('data_table/_dt_elements.mako')
+
+        def user_actions(user_id, username):
+            return _render("user_actions", user_id, username)
+
+        users_data_total_count = User.query()\
+            .filter(User.username != User.DEFAULT_USER) \
+            .count()
+
+        # json generate
+        base_q = User.query().filter(User.username != User.DEFAULT_USER)
+
+        if search_q:
+            like_expression = u'%{}%'.format(safe_unicode(search_q))
+            base_q = base_q.filter(or_(
+                User.username.ilike(like_expression),
+                User._email.ilike(like_expression),
+                User.name.ilike(like_expression),
+                User.lastname.ilike(like_expression),
+            ))
+
+        users_data_total_filtered_count = base_q.count()
+
+        sort_col = getattr(User, order_by, None)
+        if sort_col and order_dir == 'asc':
+            base_q = base_q.order_by(sort_col.asc().nullslast())
+        elif sort_col:
+            base_q = base_q.order_by(sort_col.desc().nullslast())
+
+        base_q = base_q.offset(start).limit(limit)
+        users_list = base_q.all()
+
+        users_data = []
+        for user in users_list:
+            users_data.append({
+                "username": h.gravatar_with_user(user.username),
+                "email": user.email,
+                "first_name": h.escape(user.name),
+                "last_name": h.escape(user.lastname),
+                "last_login": h.format_date(user.last_login),
+                "last_activity": h.format_date(user.last_activity),
+                "active": h.bool2icon(user.active),
+                "active_raw": user.active,
+                "admin": h.bool2icon(user.admin),
+                "extern_type": user.extern_type,
+                "extern_name": user.extern_name,
+                "action": user_actions(user.user_id, user.username),
+            })
+
+        data = ({
+            'draw': draw,
+            'data': users_data,
+            'recordsTotal': users_data_total_count,
+            'recordsFiltered': users_data_total_filtered_count,
+        })
+
+        return data
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        route_name='edit_user_auth_tokens', request_method='GET',
+        renderer='rhodecode:templates/admin/users/user_edit.mako')
+    def auth_tokens(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        user_id = self.request.matchdict.get('user_id')
+        c.user = User.get_or_404(user_id, pyramid_exc=True)
+        self._redirect_for_default_user(c.user.username)
+
+        c.active = 'auth_tokens'
+
+        c.lifetime_values = [
+            (str(-1), _('forever')),
+            (str(5), _('5 minutes')),
+            (str(60), _('1 hour')),
+            (str(60 * 24), _('1 day')),
+            (str(60 * 24 * 30), _('1 month')),
+        ]
+        c.lifetime_options = [(c.lifetime_values, _("Lifetime"))]
+        c.role_values = [
+            (x, AuthTokenModel.cls._get_role_name(x))
+            for x in AuthTokenModel.cls.ROLES]
+        c.role_options = [(c.role_values, _("Role"))]
+        c.user_auth_tokens = AuthTokenModel().get_auth_tokens(
+            c.user.user_id, show_expired=True)
+        return self._get_template_context(c)
+
+    def maybe_attach_token_scope(self, token):
+        # implemented in EE edition
+        pass
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @CSRFRequired()
+    @view_config(
+        route_name='edit_user_auth_tokens_add', request_method='POST')
+    def auth_tokens_add(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        user_id = self.request.matchdict.get('user_id')
+        c.user = User.get_or_404(user_id, pyramid_exc=True)
+        self._redirect_for_default_user(c.user.username)
+
+        lifetime = safe_int(self.request.POST.get('lifetime'), -1)
+        description = self.request.POST.get('description')
+        role = self.request.POST.get('role')
+
+        token = AuthTokenModel().create(
+            c.user.user_id, description, lifetime, role)
+        self.maybe_attach_token_scope(token)
+        Session().commit()
+
+        h.flash(_("Auth token successfully created"), category='success')
+        return HTTPFound(h.route_path('edit_user_auth_tokens', user_id=user_id))
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @CSRFRequired()
+    @view_config(
+        route_name='edit_user_auth_tokens_delete', request_method='POST')
+    def auth_tokens_delete(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        user_id = self.request.matchdict.get('user_id')
+        c.user = User.get_or_404(user_id, pyramid_exc=True)
+        self._redirect_for_default_user(c.user.username)
+
+        del_auth_token = self.request.POST.get('del_auth_token')
+
+        if del_auth_token:
+            AuthTokenModel().delete(del_auth_token, c.user.user_id)
+            Session().commit()
+            h.flash(_("Auth token successfully deleted"), category='success')
+
+        return HTTPFound(h.route_path('edit_user_auth_tokens', user_id=user_id))
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        route_name='edit_user_groups_management', request_method='GET',
+        renderer='rhodecode:templates/admin/users/user_edit.mako')
+    def groups_management(self):
+        c = self.load_default_context()
+
+        user_id = self.request.matchdict.get('user_id')
+        c.user = User.get_or_404(user_id, pyramid_exc=True)
+        c.data = c.user.group_member
+        self._redirect_for_default_user(c.user.username)
+        groups = [UserGroupModel.get_user_groups_as_dict(group.users_group) for group in c.user.group_member]
+        c.groups = json.dumps(groups)
+        c.active = 'groups'
+
+        return self._get_template_context(c)
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        route_name='edit_user_groups_management_updates', request_method='POST')
+    def groups_management_updates(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        user_id = self.request.matchdict.get('user_id')
+        c.user = User.get_or_404(user_id, pyramid_exc=True)
+        self._redirect_for_default_user(c.user.username)
+
+        users_groups = set(self.request.POST.getall('users_group_id'))
+        users_groups_model = []
+
+        for ugid in users_groups:
+            users_groups_model.append(UserGroupModel().get_group(safe_int(ugid)))
+        user_group_model = UserGroupModel()
+        user_group_model.change_groups(c.user, users_groups_model)
+
+        Session().commit()
+        c.active = 'user_groups_management'
+        h.flash(_("Groups successfully changed"), category='success')
+
+        return HTTPFound(h.route_path(
+            'edit_user_groups_management', user_id=user_id))
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        route_name='edit_user_audit_logs', request_method='GET',
+        renderer='rhodecode:templates/admin/users/user_edit.mako')
+    def user_audit_logs(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        user_id = self.request.matchdict.get('user_id')
+        c.user = User.get_or_404(user_id, pyramid_exc=True)
+        self._redirect_for_default_user(c.user.username)
+        c.active = 'audit'
+
+        p = safe_int(self.request.GET.get('page', 1), 1)
+
+        filter_term = self.request.GET.get('filter')
+        c.user_log = UserModel().get_user_log(c.user, filter_term)
+
+        def url_generator(**kw):
+            if filter_term:
+                kw['filter'] = filter_term
+            return self.request.current_route_path(_query=kw)
+
+        c.user_log = Page(c.user_log, page=p, items_per_page=10,
+                          url=url_generator)
+        c.filter_term = filter_term
+        return self._get_template_context(c)
+

rhodecode/apps/my_account/__init__.py

@@ -0,0 +1,52 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+from rhodecode.apps._base import ADMIN_PREFIX
+
+
+def includeme(config):
+
+    config.add_route(
+        name='my_account_profile',
+        pattern=ADMIN_PREFIX + '/my_account/profile')
+
+    config.add_route(
+        name='my_account_password',
+        pattern=ADMIN_PREFIX + '/my_account/password')
+
+    config.add_route(
+        name='my_account_password_update',
+        pattern=ADMIN_PREFIX + '/my_account/password')
+
+    config.add_route(
+        name='my_account_auth_tokens',
+        pattern=ADMIN_PREFIX + '/my_account/auth_tokens')
+    config.add_route(
+        name='my_account_auth_tokens_add',
+        pattern=ADMIN_PREFIX + '/my_account/auth_tokens/new',
+        )
+    config.add_route(
+        name='my_account_auth_tokens_delete',
+        pattern=ADMIN_PREFIX + '/my_account/auth_tokens/delete',
+        )
+
+    # Scan module for configuration decorators.
+    config.scan()

rhodecode/apps/my_account/tests/__init__.py

@@ -0,0 +1,19 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/

rhodecode/apps/my_account/tests/test_my_account_auth_tokens.py

@@ -0,0 +1,111 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import pytest
+
+from rhodecode.apps._base import ADMIN_PREFIX
+from rhodecode.model.db import User
+from rhodecode.tests import (
+    TestController, TEST_USER_ADMIN_LOGIN, TEST_USER_ADMIN_PASS,
+    TEST_USER_REGULAR_LOGIN, TEST_USER_REGULAR_PASS, assert_session_flash)
+from rhodecode.tests.fixture import Fixture
+from rhodecode.tests.utils import AssertResponse
+
+fixture = Fixture()
+
+
+def route_path(name, **kwargs):
+    return {
+        'my_account_auth_tokens':
+            ADMIN_PREFIX + '/my_account/auth_tokens',
+        'my_account_auth_tokens_add':
+            ADMIN_PREFIX + '/my_account/auth_tokens/new',
+        'my_account_auth_tokens_delete':
+            ADMIN_PREFIX + '/my_account/auth_tokens/delete',
+    }[name].format(**kwargs)
+
+
+class TestMyAccountAuthTokens(TestController):
+
+    def test_my_account_auth_tokens(self):
+        usr = self.log_user('test_regular2', 'test12')
+        user = User.get(usr['user_id'])
+        response = self.app.get(route_path('my_account_auth_tokens'))
+        for token in user.auth_tokens:
+            response.mustcontain(token)
+            response.mustcontain('never')
+
+    def test_my_account_add_auth_tokens_wrong_csrf(self, user_util):
+        user = user_util.create_user(password='qweqwe')
+        self.log_user(user.username, 'qweqwe')
+
+        self.app.post(
+            route_path('my_account_auth_tokens_add'),
+            {'description': 'desc', 'lifetime': -1}, status=403)
+
+    @pytest.mark.parametrize("desc, lifetime", [
+        ('forever', -1),
+        ('5mins', 60*5),
+        ('30days', 60*60*24*30),
+    ])
+    def test_my_account_add_auth_tokens(self, desc, lifetime, user_util):
+        user = user_util.create_user(password='qweqwe')
+        user_id = user.user_id
+        self.log_user(user.username, 'qweqwe')
+
+        response = self.app.post(
+            route_path('my_account_auth_tokens_add'),
+            {'description': desc, 'lifetime': lifetime,
+             'csrf_token': self.csrf_token})
+        assert_session_flash(response, 'Auth token successfully created')
+
+        response = response.follow()
+        user = User.get(user_id)
+        for auth_token in user.auth_tokens:
+            response.mustcontain(auth_token)
+
+    def test_my_account_delete_auth_token(self, user_util):
+        user = user_util.create_user(password='qweqwe')
+        user_id = user.user_id
+        self.log_user(user.username, 'qweqwe')
+
+        user = User.get(user_id)
+        keys = user.extra_auth_tokens
+        assert 2 == len(keys)
+
+        response = self.app.post(
+            route_path('my_account_auth_tokens_add'),
+            {'description': 'desc', 'lifetime': -1,
+             'csrf_token': self.csrf_token})
+        assert_session_flash(response, 'Auth token successfully created')
+        response.follow()
+
+        user = User.get(user_id)
+        keys = user.extra_auth_tokens
+        assert 3 == len(keys)
+
+        response = self.app.post(
+            route_path('my_account_auth_tokens_delete'),
+            {'del_auth_token': keys[0].api_key, 'csrf_token': self.csrf_token})
+        assert_session_flash(response, 'Auth token successfully deleted')
+
+        user = User.get(user_id)
+        keys = user.extra_auth_tokens
+        assert 2 == len(keys)

rhodecode/apps/my_account/tests/test_my_account_password.py

@@ -0,0 +1,137 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import pytest
+import mock
+
+from rhodecode.apps._base import ADMIN_PREFIX
+from rhodecode.lib import helpers as h
+from rhodecode.lib.auth import check_password
+from rhodecode.model.meta import Session
+from rhodecode.model.user import UserModel
+from rhodecode.tests import assert_session_flash
+from rhodecode.tests.fixture import Fixture, TestController, error_function
+
+fixture = Fixture()
+
+
+def route_path(name, **kwargs):
+    return {
+        'home': '/',
+        'my_account_password':
+            ADMIN_PREFIX + '/my_account/password',
+    }[name].format(**kwargs)
+
+
+test_user_1 = 'testme'
+test_user_1_password = '0jd83nHNS/d23n'
+
+
+class TestMyAccountPassword(TestController):
+    def test_valid_change_password(self, user_util):
+        new_password = 'my_new_valid_password'
+        user = user_util.create_user(password=test_user_1_password)
+        self.log_user(user.username, test_user_1_password)
+
+        form_data = [
+            ('current_password', test_user_1_password),
+            ('__start__', 'new_password:mapping'),
+            ('new_password', new_password),
+            ('new_password-confirm', new_password),
+            ('__end__', 'new_password:mapping'),
+            ('csrf_token', self.csrf_token),
+        ]
+        response = self.app.post(route_path('my_account_password'), form_data).follow()
+        assert 'Successfully updated password' in response
+
+        # check_password depends on user being in session
+        Session().add(user)
+        try:
+            assert check_password(new_password, user.password)
+        finally:
+            Session().expunge(user)
+
+    @pytest.mark.parametrize('current_pw, new_pw, confirm_pw', [
+        ('', 'abcdef123', 'abcdef123'),
+        ('wrong_pw', 'abcdef123', 'abcdef123'),
+        (test_user_1_password, test_user_1_password, test_user_1_password),
+        (test_user_1_password, '', ''),
+        (test_user_1_password, 'abcdef123', ''),
+        (test_user_1_password, '', 'abcdef123'),
+        (test_user_1_password, 'not_the', 'same_pw'),
+        (test_user_1_password, 'short', 'short'),
+    ])
+    def test_invalid_change_password(self, current_pw, new_pw, confirm_pw,
+                                     user_util):
+        user = user_util.create_user(password=test_user_1_password)
+        self.log_user(user.username, test_user_1_password)
+
+        form_data = [
+            ('current_password', current_pw),
+            ('__start__', 'new_password:mapping'),
+            ('new_password', new_pw),
+            ('new_password-confirm', confirm_pw),
+            ('__end__', 'new_password:mapping'),
+            ('csrf_token', self.csrf_token),
+        ]
+        response = self.app.post(route_path('my_account_password'), form_data)
+
+        assert_response = response.assert_response()
+        assert assert_response.get_elements('.error-block')
+
+    @mock.patch.object(UserModel, 'update_user', error_function)
+    def test_invalid_change_password_exception(self, user_util):
+        user = user_util.create_user(password=test_user_1_password)
+        self.log_user(user.username, test_user_1_password)
+
+        form_data = [
+            ('current_password', test_user_1_password),
+            ('__start__', 'new_password:mapping'),
+            ('new_password', '123456'),
+            ('new_password-confirm', '123456'),
+            ('__end__', 'new_password:mapping'),
+            ('csrf_token', self.csrf_token),
+        ]
+        response = self.app.post(route_path('my_account_password'), form_data)
+        assert_session_flash(
+            response, 'Error occurred during update of user password')
+
+    def test_password_is_updated_in_session_on_password_change(self, user_util):
+        old_password = 'abcdef123'
+        new_password = 'abcdef124'
+
+        user = user_util.create_user(password=old_password)
+        session = self.log_user(user.username, old_password)
+        old_password_hash = session['password']
+
+        form_data = [
+            ('current_password', old_password),
+            ('__start__', 'new_password:mapping'),
+            ('new_password', new_password),
+            ('new_password-confirm', new_password),
+            ('__end__', 'new_password:mapping'),
+            ('csrf_token', self.csrf_token),
+        ]
+        self.app.post(route_path('my_account_password'), form_data)
+
+        response = self.app.get(route_path('home'))
+        new_password_hash = response.session['rhodecode_user']['password']
+
+        assert old_password_hash != new_password_hash
  No newline at end of file

rhodecode/apps/my_account/tests/test_my_account_profile.py

@@ -0,0 +1,55 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import pytest
+
+from rhodecode.apps._base import ADMIN_PREFIX
+from rhodecode.tests import (
+    TestController, TEST_USER_ADMIN_LOGIN,
+    TEST_USER_REGULAR_LOGIN, TEST_USER_REGULAR_PASS)
+from rhodecode.tests.fixture import Fixture
+
+fixture = Fixture()
+
+
+def route_path(name, **kwargs):
+    return {
+        'my_account':
+            ADMIN_PREFIX + '/my_account/profile',
+    }[name].format(**kwargs)
+
+
+class TestMyAccountProfile(TestController):
+
+    def test_my_account(self):
+        self.log_user()
+        response = self.app.get(route_path('my_account'))
+
+        response.mustcontain(TEST_USER_ADMIN_LOGIN)
+        response.mustcontain('href="/_admin/my_account/edit"')
+        response.mustcontain('Photo')
+
+    def test_my_account_regular_user(self):
+        self.log_user(TEST_USER_REGULAR_LOGIN, TEST_USER_REGULAR_PASS)
+        response = self.app.get(route_path('my_account'))
+
+        response.mustcontain(TEST_USER_REGULAR_LOGIN)
+        response.mustcontain('href="/_admin/my_account/edit"')
+        response.mustcontain('Photo')

rhodecode/apps/my_account/views.py

@@ -0,0 +1,194 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import logging
+
+from pyramid.httpexceptions import HTTPFound
+from pyramid.view import view_config
+
+from rhodecode.apps._base import BaseAppView
+from rhodecode import forms
+from rhodecode.lib.auth import LoginRequired, NotAnonymous, CSRFRequired
+from rhodecode.lib import helpers as h
+from rhodecode.lib.utils2 import safe_int, md5
+from rhodecode.model.auth_token import AuthTokenModel
+from rhodecode.model.meta import Session
+from rhodecode.model.user import UserModel
+from rhodecode.model.validation_schema.schemas import user_schema
+
+log = logging.getLogger(__name__)
+
+
+class MyAccountView(BaseAppView):
+    ALLOW_SCOPED_TOKENS = False
+    """
+    This view has alternative version inside EE, if modified please take a look
+    in there as well.
+    """
+
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+        c.user = c.auth_user.get_instance()
+        c.allow_scoped_tokens = self.ALLOW_SCOPED_TOKENS
+        self._register_global_c(c)
+        return c
+
+    @LoginRequired()
+    @NotAnonymous()
+    @view_config(
+        route_name='my_account_profile', request_method='GET',
+        renderer='rhodecode:templates/admin/my_account/my_account.mako')
+    def my_account_profile(self):
+        c = self.load_default_context()
+        c.active = 'profile'
+        return self._get_template_context(c)
+
+    @LoginRequired()
+    @NotAnonymous()
+    @view_config(
+        route_name='my_account_password', request_method='GET',
+        renderer='rhodecode:templates/admin/my_account/my_account.mako')
+    def my_account_password(self):
+        c = self.load_default_context()
+        c.active = 'password'
+        c.extern_type = c.user.extern_type
+
+        schema = user_schema.ChangePasswordSchema().bind(
+            username=c.user.username)
+
+        form = forms.Form(
+            schema, buttons=(forms.buttons.save, forms.buttons.reset))
+
+        c.form = form
+        return self._get_template_context(c)
+
+    @LoginRequired()
+    @NotAnonymous()
+    @CSRFRequired()
+    @view_config(
+        route_name='my_account_password', request_method='POST',
+        renderer='rhodecode:templates/admin/my_account/my_account.mako')
+    def my_account_password_update(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+        c.active = 'password'
+        c.extern_type = c.user.extern_type
+
+        schema = user_schema.ChangePasswordSchema().bind(
+            username=c.user.username)
+
+        form = forms.Form(
+            schema, buttons=(forms.buttons.save, forms.buttons.reset))
+
+        if c.extern_type != 'rhodecode':
+            raise HTTPFound(self.request.route_path('my_account_password'))
+
+        controls = self.request.POST.items()
+        try:
+            valid_data = form.validate(controls)
+            UserModel().update_user(c.user.user_id, **valid_data)
+            c.user.update_userdata(force_password_change=False)
+            Session().commit()
+        except forms.ValidationFailure as e:
+            c.form = e
+            return self._get_template_context(c)
+
+        except Exception:
+            log.exception("Exception updating password")
+            h.flash(_('Error occurred during update of user password'),
+                    category='error')
+        else:
+            instance = c.auth_user.get_instance()
+            self.session.setdefault('rhodecode_user', {}).update(
+                {'password': md5(instance.password)})
+            self.session.save()
+            h.flash(_("Successfully updated password"), category='success')
+
+        raise HTTPFound(self.request.route_path('my_account_password'))
+
+    @LoginRequired()
+    @NotAnonymous()
+    @view_config(
+        route_name='my_account_auth_tokens', request_method='GET',
+        renderer='rhodecode:templates/admin/my_account/my_account.mako')
+    def my_account_auth_tokens(self):
+        _ = self.request.translate
+
+        c = self.load_default_context()
+        c.active = 'auth_tokens'
+
+        c.lifetime_values = [
+            (str(-1), _('forever')),
+            (str(5), _('5 minutes')),
+            (str(60), _('1 hour')),
+            (str(60 * 24), _('1 day')),
+            (str(60 * 24 * 30), _('1 month')),
+        ]
+        c.lifetime_options = [(c.lifetime_values, _("Lifetime"))]
+        c.role_values = [
+            (x, AuthTokenModel.cls._get_role_name(x))
+            for x in AuthTokenModel.cls.ROLES]
+        c.role_options = [(c.role_values, _("Role"))]
+        c.user_auth_tokens = AuthTokenModel().get_auth_tokens(
+            c.user.user_id, show_expired=True)
+        return self._get_template_context(c)
+
+    def maybe_attach_token_scope(self, token):
+        # implemented in EE edition
+        pass
+
+    @LoginRequired()
+    @NotAnonymous()
+    @CSRFRequired()
+    @view_config(
+        route_name='my_account_auth_tokens_add', request_method='POST')
+    def my_account_auth_tokens_add(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        lifetime = safe_int(self.request.POST.get('lifetime'), -1)
+        description = self.request.POST.get('description')
+        role = self.request.POST.get('role')
+
+        token = AuthTokenModel().create(
+            c.user.user_id, description, lifetime, role)
+        self.maybe_attach_token_scope(token)
+        Session().commit()
+
+        h.flash(_("Auth token successfully created"), category='success')
+        return HTTPFound(h.route_path('my_account_auth_tokens'))
+
+    @LoginRequired()
+    @NotAnonymous()
+    @CSRFRequired()
+    @view_config(
+        route_name='my_account_auth_tokens_delete', request_method='POST')
+    def my_account_auth_tokens_delete(self):
+        _ = self.request.translate
+        c = self.load_default_context()
+
+        del_auth_token = self.request.POST.get('del_auth_token')
+
+        if del_auth_token:
+            AuthTokenModel().delete(del_auth_token, c.user.user_id)
+            Session().commit()
+            h.flash(_("Auth token successfully deleted"), category='success')
+
+        return HTTPFound(h.route_path('my_account_auth_tokens'))

rhodecode/apps/repository/__init__.py

@@ -0,0 +1,46 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+def includeme(config):
+
+    config.add_route(
+        name='repo_maintenance',
+        pattern='/{repo_name:.*?[^/]}/maintenance', repo_route=True)
+
+    config.add_route(
+        name='repo_maintenance_execute',
+        pattern='/{repo_name:.*?[^/]}/maintenance/execute', repo_route=True)
+
+
+    # Strip
+    config.add_route(
+        name='strip',
+        pattern='/{repo_name:.*?[^/]}/strip', repo_route=True)
+
+    config.add_route(
+        name='strip_check',
+        pattern='/{repo_name:.*?[^/]}/strip_check', repo_route=True)
+
+    config.add_route(
+        name='strip_execute',
+        pattern='/{repo_name:.*?[^/]}/strip_execute', repo_route=True)
+    # Scan module for configuration decorators.
+    config.scan()

rhodecode/apps/repository/views/__init__.py

@@ -0,0 +1,19 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2011-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/

rhodecode/apps/repository/views/repo_maintainance.py

@@ -0,0 +1,70 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2011-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import logging
+
+from pyramid.view import view_config
+
+from rhodecode.apps._base import RepoAppView
+from rhodecode.lib.auth import (LoginRequired, HasRepoPermissionAnyDecorator,
+    NotAnonymous)
+from rhodecode.lib import repo_maintenance
+
+log = logging.getLogger(__name__)
+
+
+class RepoMaintenanceView(RepoAppView):
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+
+        # TODO(marcink): remove repo_info and use c.rhodecode_db_repo instead
+        c.repo_info = self.db_repo
+
+        self._register_global_c(c)
+        return c
+
+    @LoginRequired()
+    @NotAnonymous()
+    @HasRepoPermissionAnyDecorator('repository.admin')
+    @view_config(
+        route_name='repo_maintenance', request_method='GET',
+        renderer='rhodecode:templates/admin/repos/repo_edit.mako')
+    def repo_maintenance(self):
+        c = self.load_default_context()
+        c.active = 'maintenance'
+        maintenance = repo_maintenance.RepoMaintenance()
+        c.executable_tasks = maintenance.get_tasks_for_repo(self.db_repo)
+        return self._get_template_context(c)
+
+    @LoginRequired()
+    @NotAnonymous()
+    @HasRepoPermissionAnyDecorator('repository.admin')
+    @view_config(
+        route_name='repo_maintenance_execute', request_method='GET',
+        renderer='json', xhr=True)
+    def repo_maintenance_execute(self):
+        c = self.load_default_context()
+        c.active = 'maintenance'
+        _ = self.request.translate
+
+        maintenance = repo_maintenance.RepoMaintenance()
+        executed_types = maintenance.execute(self.db_repo)
+
+        return executed_types

rhodecode/apps/repository/views/strip.py

@@ -0,0 +1,110 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2011-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import logging
+from pyramid.view import view_config
+
+from rhodecode.apps._base import RepoAppView
+from rhodecode.lib.auth import (LoginRequired, HasRepoPermissionAnyDecorator,
+    NotAnonymous)
+
+
+log = logging.getLogger(__name__)
+
+
+class StripView(RepoAppView):
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+
+        # TODO(marcink): remove repo_info and use c.rhodecode_db_repo instead
+        c.repo_info = self.db_repo
+
+        self._register_global_c(c)
+        return c
+
+    @LoginRequired()
+    @NotAnonymous()
+    @HasRepoPermissionAnyDecorator('repository.admin')
+    @view_config(
+        route_name='strip', request_method='GET',
+        renderer='rhodecode:templates/admin/repos/repo_edit.mako')
+    def strip(self):
+        c = self.load_default_context()
+        c.active = 'strip'
+        c.strip_limit = 10
+
+        return self._get_template_context(c)
+
+    @LoginRequired()
+    @NotAnonymous()
+    @HasRepoPermissionAnyDecorator('repository.admin')
+    @view_config(
+        route_name='strip_check', request_method='POST',
+        renderer='json', xhr=True
+    )
+    def strip_check(self):
+        from rhodecode.lib.vcs.backends.base import EmptyCommit
+        data = {}
+        rp = self.request.POST
+        for i in range(1, 11):
+            chset = 'changeset_id-%d'%(i,)
+            check = rp.get(chset)
+            if check:
+                data[i] = self.db_repo.get_changeset(rp[chset])
+                if isinstance(data[i], EmptyCommit):
+                    data[i] = {'rev': None, 'commit': rp[chset]}
+                else:
+                    data[i] = {'rev': data[i].raw_id, 'branch': data[i].branch, 'author': data[i].author,
+                               'comment': data[i].message}
+            else:
+                break
+        return data
+
+    @LoginRequired()
+    @NotAnonymous()
+    @HasRepoPermissionAnyDecorator('repository.admin')
+    @view_config(
+        route_name='strip_execute', request_method='POST',
+        renderer='json', xhr=True
+    )
+    def strip_execute(self):
+
+        from rhodecode.model.scm import ScmModel
+        from rhodecode.lib.ext_json import json
+
+        c = self.load_default_context()
+        user = self._rhodecode_user
+        rp = self.request.POST
+        data = {}
+        for idx in rp:
+            commit = json.loads(rp[idx])
+            #If someone put two times the same branch
+            if commit['branch'] in data.keys():
+                continue
+            try:
+                ScmModel().strip(repo=c.repo_info,
+                          commit_id=commit['rev'], branch=commit['branch'])
+                log.info('Stripped commit %s from repo `%s` by %s' % (commit['rev'], c.repo_info.repo_name, user))
+                data[commit['rev']] = True
+            except Exception, e:
+                data[commit['rev']] = False
+                log.debug('Stripped commit %s from repo `%s` failed by %s, exeption %s' % (commit['rev'],
+                                                                            c.repo_info.repo_name, user, e.message))
+        return data

rhodecode/apps/user_profile/__init__.py

@@ -0,0 +1,28 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+def includeme(config):
+    config.add_route(
+        name='user_profile',
+        pattern='/_profiles/{username}')
+
+    # Scan module for configuration decorators.
+    config.scan()

rhodecode/apps/user_profile/tests/__init__.py

@@ -0,0 +1,19 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2017 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/

.bumpversion.cfg

@@ -1,6 +1,6 b''
 [bumpversion]
-current_version = 4.6.1
+current_version = 4.7.0
 message = release: Bump version {current_version} to {new_version}
 
 [bumpversion:file:rhodecode/VERSION]
 

.coveragerc

@@ -1,18 +1,20 b''
 [run]
 
 branch = True
 
 include =
     rhodecode/lib/*
     rhodecode/model/*
     rhodecode/controllers/*
 
 omit =
     rhodecode/lib/vcs/remote/*
     rhodecode/lib/dbmigrate/*
+    rhodecode/lib/paster_commands/*
+    rhodecode/tests/*
 
 [report]
 
 exclude_lines =
     raise NotImplementedError
 

.release.cfg

@@ -1,33 +1,28 b''
 [DEFAULT]
 done = false
 
 [task:bump_version]
 done = true
 
-[task:rc_tools_pinned]
-done = true
-
 [task:fixes_on_stable]
-done = true
 
 [task:pip2nix_generated]
-done = true
 
 [task:changelog_updated]
-done = true
 
 [task:generate_api_docs]
-done = true
+
+[task:updated_translation]
 
 [release]
-state = prepared
-version = 4.6.1
+state = in_progress
+version = 4.7.0
 
-[task:updated_translation]
+[task:rc_tools_pinned]
 
 [task:generate_js_routes]
 
 [task:updated_trial_license]
 
 [task:generate_oss_licenses]
 

MANIFEST.in

@@ -1,53 +1,53 b''
 # top level files
-include test.ini
+
 include MANIFEST.in
 include README.rst
 include CHANGES.rst
 include LICENSE.txt
 
 include rhodecode/VERSION
 
 # docs
 recursive-include docs *
 
 # all config files
 recursive-include configs *
 
 # translations
 recursive-include rhodecode/i18n *
 
 # hook templates
 recursive-include rhodecode/config/hook_templates *
 
 # non-python core stuff
 recursive-include rhodecode *.cfg
 recursive-include rhodecode *.json
 recursive-include rhodecode *.ini_tmpl
 recursive-include rhodecode *.sh
 recursive-include rhodecode *.mako
 
 # 502 page
 include rhodecode/public/502.html
 
 
 # images, css
 include rhodecode/public/css/*.css
 include rhodecode/public/images/*.*
 
 # sound files
 include rhodecode/public/sounds/*.mp3
 include rhodecode/public/sounds/*.wav
 
 # fonts
 recursive-include rhodecode/public/fonts/ProximaNova *
 recursive-include rhodecode/public/fonts/RCIcons *
 
 # js
 recursive-include rhodecode/public/js *
 
 # templates
 recursive-include rhodecode/templates *
 
 # skip any tests files
 recursive-exclude rhodecode/tests *
 

configs/development.ini

@@ -1,681 +1,678 b''
 
 
 ################################################################################
 ##                 RHODECODE COMMUNITY EDITION CONFIGURATION                  ##
 # The %(here)s variable will be replaced with the parent directory of this file#
 ################################################################################
 
 [DEFAULT]
 debug = true
 
 ################################################################################
 ##                            EMAIL CONFIGURATION                             ##
 ## Uncomment and replace with the email address which should receive          ##
 ## any error reports after an application crash                               ##
 ## Additionally these settings will be used by the RhodeCode mailing system   ##
 ################################################################################
 
 ## prefix all emails subjects with given prefix, helps filtering out emails
 #email_prefix = [RhodeCode]
 
 ## email FROM address all mails will be sent
 #app_email_from = rhodecode-noreply@localhost
 
 ## Uncomment and replace with the address which should receive any error report
 ## note: using appenlight for error handling doesn't need this to be uncommented
 #email_to = admin@localhost
 
 ## in case of Application errors, sent an error email form
 #error_email_from = rhodecode_error@localhost
 
 ## additional error message to be send in case of server crash
 #error_message =
 
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 ## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
 #smtp_auth =
 
 [server:main]
 ## COMMON ##
 host = 127.0.0.1
 port = 5000
 
 ##################################
 ##     WAITRESS WSGI SERVER     ##
 ## Recommended for Development  ##
 ##################################
 
 use = egg:waitress#main
 ## number of worker threads
 threads = 5
 ## MAX BODY SIZE 100GB
 max_request_body_size = 107374182400
 ## Use poll instead of select, fixes file descriptors limits problems.
 ## May not work on old windows systems.
 asyncore_use_poll = true
 
 
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
 ## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 #use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
 ## when this option is set to more than one worker, recommended
 ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
 ## The `instance_id = *` must be set in the [app:main] section below
 #workers = 2
 ## number of threads for each of the worker, must be set to 1 for gevent
 ## generally recommened to be at 1
 #threads = 1
 ## process name
 #proc_name = rhodecode
 ## type of worker class, one of sync, gevent
 ## recommended for bigger setup is using of of other than sync one
 #worker_class = sync
 ## The maximum number of simultaneous clients. Valid only for Gevent
 #worker_connections = 10
 ## max number of requests that worker will handle before being gracefully
 ## restarted, could prevent memory leaks
 #max_requests = 1000
 #max_requests_jitter = 30
 ## amount of time a worker can spend with handling a request before it
 ## gets killed and restarted. Set to 6hrs
 #timeout = 21600
 
 
 ## prefix middleware for RhodeCode.
 ## recommended when using proxy setup.
 ## allows to set RhodeCode under a prefix in server.
 ## eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
 ## And set your prefix like: `prefix = /custom_prefix`
 ## be sure to also set beaker.session.cookie_path = /custom_prefix if you need
 ## to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
 
 [app:main]
 use = egg:rhodecode-enterprise-ce
 
 ## enable proxy prefix middleware, defined above
 #filter-with = proxy-prefix
 
 # During development the we want to have the debug toolbar enabled
 pyramid.includes =
     pyramid_debugtoolbar
     rhodecode.utils.debugtoolbar
     rhodecode.lib.middleware.request_wrapper
 
 pyramid.reload_templates = true
 
 debugtoolbar.hosts = 0.0.0.0/0
 debugtoolbar.exclude_prefixes =
     /css
     /fonts
     /images
     /js
 
 ## RHODECODE PLUGINS ##
 rhodecode.includes =
     rhodecode.api
 
 
 # api prefix url
 rhodecode.api.url = /_admin/api
 
 
 ## END RHODECODE PLUGINS ##
 
 ## encryption key used to encrypt social plugin tokens,
 ## remote_urls with credentials etc, if not set it defaults to
 ## `beaker.session.secret`
 #rhodecode.encrypted_values.secret =
 
 ## decryption strict mode (enabled by default). It controls if decryption raises
 ## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
 ## return gzipped responses from Rhodecode (static files/application)
 gzip_responses = false
 
 ## autogenerate javascript routes file on startup
 generate_js_files = false
 
 ## Optional Languages
 ## en(default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ## perform a full repository scan on each server start, this should be
 ## set to false after first startup, to allow faster server restarts.
 startup.import_repos = false
 
 ## Uncomment and set this path to use archive download cache.
 ## Once enabled, generated archives will be cached at this location
 ## and served from the cache during subsequent requests for the same archive of
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
 ## change this to unique ID for security
 app_instance_uuid = rc-production
 
 ## cut off limit for large diffs (size in bytes)
 cut_off_limit_diff = 1024000
 cut_off_limit_file = 256000
 
 ## use cache version of scm repo everywhere
 vcs_full_cache = true
 
 ## force https in RhodeCode, fixes https redirects, assumes it's always https
 ## Normally this is controlled by proper http flags sent from http server
 force_https = false
 
 ## use Strict-Transport-Security headers
 use_htsts = false
 
 ## number of commits stats will parse on each iteration
 commit_parse_limit = 25
 
 ## git rev filter option, --all is the default filter, if you need to
 ## hide all refs in changelog switch this to --branches --tags
 git_rev_filter = --branches --tags
 
 # Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ## RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
 ## url that does rewrites to _admin/gists/{gistid}.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ## List of controllers (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
 ## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ##
 ## Syntax is ControllerClass:function_pattern.
 ## To enable access to raw_files put `FilesController:raw`.
 ## To enable access to patches add `ChangesetController:changeset_patch`.
 ## The list should be "," separated and on a single line.
 ##
 ## Recommended controllers to enable:
 #    ChangesetController:changeset_patch,
 #    ChangesetController:changeset_raw,
 #    FilesController:raw,
 #    FilesController:archivefile,
 #    GistsController:*,
 api_access_controllers_whitelist =
 
 ## default encoding used to convert from and to unicode
 ## can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
 ## multiple instances of rhodecode, make sure it's globally unique for
 ## all running rhodecode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
 ## authentication settings. Then you can enable e.g. the internal rhodecode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
 ## egg:rhodecode-enterprise-ce#rhodecode
 ## egg:rhodecode-enterprise-ce#pam
 ## egg:rhodecode-enterprise-ce#ldap
 ## egg:rhodecode-enterprise-ce#jasig_cas
 ## egg:rhodecode-enterprise-ce#headers
 ## egg:rhodecode-enterprise-ce#crowd
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ## alternative return HTTP header for failed authentication. Default HTTP
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ## This will be served instead of default 401 on bad authnetication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
 ## ret_code directly, use 401 initially (Which triggers credentials prompt)
 ## and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ## locking return code. When repository is locked return this HTTP code. 2XX
 ## codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ## allows to change the repository location in settings page
 allow_repo_location_change = true
 
 ## allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ## generated license token, goto license page in RhodeCode settings to obtain
 ## new token
 license_token =
 
 ## supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 ## supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = dev
 
 ## Display extended labs settings
 labs_settings_active = true
 
 ####################################
 ###        CELERY CONFIG        ####
 ####################################
 use_celery = false
 broker.host = localhost
 broker.vhost = rabbitmqhost
 broker.port = 5672
 broker.user = rabbitmq
 broker.password = qweqwe
 
 celery.imports = rhodecode.lib.celerylib.tasks
 
 celery.result.backend = amqp
 celery.result.dburi = amqp://
 celery.result.serialier = json
 
 #celery.send.task.error.emails = true
 #celery.amqp.task.result.expires = 18000
 
 celeryd.concurrency = 2
 #celeryd.log.file = celeryd.log
 celeryd.log.level = debug
 celeryd.max.tasks.per.child = 1
 
 ## tasks will never be sent to the queue, but executed locally instead.
 celery.always.eager = false
 
 ####################################
 ###         BEAKER CACHE        ####
 ####################################
 # default cache dir for templates.  Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk
 cache_dir = %(here)s/data
 
 ## locking and default file storage for Beaker. Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk/cache/beaker_data
 beaker.cache.data_dir = %(here)s/data/cache/beaker_data
 beaker.cache.lock_dir = %(here)s/data/cache/beaker_lock
 
 beaker.cache.regions = super_short_term, short_term, long_term, sql_cache_short, auth_plugins, repo_cache_long
 
 beaker.cache.super_short_term.type = memory
 beaker.cache.super_short_term.expire = 10
 beaker.cache.super_short_term.key_length = 256
 
 beaker.cache.short_term.type = memory
 beaker.cache.short_term.expire = 60
 beaker.cache.short_term.key_length = 256
 
 beaker.cache.long_term.type = memory
 beaker.cache.long_term.expire = 36000
 beaker.cache.long_term.key_length = 256
 
 beaker.cache.sql_cache_short.type = memory
 beaker.cache.sql_cache_short.expire = 10
 beaker.cache.sql_cache_short.key_length = 256
 
 ## default is memory cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.auth_plugins.type = ext:database
 #beaker.cache.auth_plugins.lock_dir = %(here)s/data/cache/auth_plugin_lock
 #beaker.cache.auth_plugins.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.cache.auth_plugins.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.cache.auth_plugins.sa.pool_recycle = 3600
 #beaker.cache.auth_plugins.sa.pool_size = 10
 #beaker.cache.auth_plugins.sa.max_overflow = 0
 
 beaker.cache.repo_cache_long.type = memorylru_base
 beaker.cache.repo_cache_long.max_items = 4096
 beaker.cache.repo_cache_long.expire = 2592000
 
 ## default is memorylru_base cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.repo_cache_long.type = ext:memcached
 #beaker.cache.repo_cache_long.url = localhost:11211
 #beaker.cache.repo_cache_long.expire = 1209600
 #beaker.cache.repo_cache_long.key_length = 256
 
 ####################################
 ###       BEAKER SESSION        ####
 ####################################
 
 ## .session.type is type of storage options for the session, current allowed
 ## types are file, ext:memcached, ext:database, and memory (default).
 beaker.session.type = file
 beaker.session.data_dir = %(here)s/data/sessions/data
 
 ## db based session, fast, and allows easy management over logged in users
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = develop-rc-uytcxaz
 beaker.session.lock_dir = %(here)s/data/sessions/lock
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = key_for_encryption
 #beaker.session.validate_key = validation_key
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 ## Path to use for the cookie. Set to prefix if you use prefix middleware
 #beaker.session.cookie_path = /custom_prefix
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
 
 ## auto save the session to not to use .save()
 beaker.session.auto = false
 
 ## default cookie expiration time in seconds, set to `true` to set expire
 ## at browser close
 #beaker.session.cookie_expires = 3600
 
 ###################################
 ## SEARCH INDEXING CONFIGURATION ##
 ###################################
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
 ## WHOOSH Backend, doesn't require additional services to run
 ## it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
 ########################################
 ###    CHANNELSTREAM CONFIG         ####
 ########################################
 ## channelstream enables persistent connections and live notification
 ## in the system. It's also used by the chat system
 channelstream.enabled = false
 
 ## server address for channelstream server on the backend
 channelstream.server = 127.0.0.1:9800
 
 ## location of the channelstream server from outside world
 ## use ws:// for http or wss:// for https. This address needs to be handled
 ## by external HTTP server such as Nginx or Apache
 ## see nginx/apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = secret
 channelstream.history.location = %(here)s/channelstream_history
 
 ## Internal application path that Javascript uses to connect into.
 ## If you use proxy-prefix the prefix should be added before /_channelstream
 channelstream.proxy_path = /_channelstream
 
 
 ###################################
 ##       APPENLIGHT CONFIG       ##
 ###################################
 
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
 ## appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
 # used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
 
 ## enables 404 error logging (default False)
 appenlight.report_404 = false
 
 ## time in seconds after request is considered being slow (default 1)
 appenlight.slow_request_time = 1
 
 ## record slow requests in application
 ## (needs to be enabled for slow datastore recording and time tracking)
 appenlight.slow_requests = true
 
 ## enable hooking to application loggers
 appenlight.logging = true
 
 ## minimum log level for log capture
 appenlight.logging.level = WARNING
 
 ## send logs only from erroneous/slow requests
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
 ## list of additonal keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
 ## start with HTTP* this list be extended with additional keywords here
 appenlight.environ_keys_whitelist =
 
 ## list of keywords that should be blanked from request object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always blank keys that contain following words
 ## 'password', 'passwd', 'pwd', 'auth_tkt', 'secret', 'csrf'
 ## this list be extended with additional keywords set here
 appenlight.request_keys_blacklist =
 
 ## list of namespaces that should be ignores when gathering log entries
 ## can be string with comma separated list of namespaces
 ## (by default the client ignores own entries: appenlight_client.client)
 appenlight.log_namespace_blacklist =
 
 
 ################################################################################
 ## WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*  ##
 ## Debug mode will enable the interactive debugging tool, allowing ANYONE to  ##
 ## execute malicious code after an exception is raised.                       ##
 ################################################################################
 #set debug = false
 
 
 ##############
 ## STYLING  ##
 ##############
 debug_style = true
 
 ###########################################
 ###   MAIN RHODECODE DATABASE CONFIG    ###
 ###########################################
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
 sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 
 # see sqlalchemy docs for other advanced settings
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
 ## recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
 ## the number of connections to keep open inside the connection pool.
 ## 0 indicates no limit
 #sqlalchemy.db1.pool_size = 5
 
 ## the number of connections to allow in connection pool "overflow", that is
 ## connections that can be opened above and beyond the pool_size setting,
 ## which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 
 ##################
 ### VCS CONFIG ###
 ##################
 vcs.server.enable = true
 vcs.server = localhost:9900
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
-## `pyro4` - use pyro4 server
 ## `http` - use http-rpc backend (default)
 vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
-## `pyro4` - use pyro4 server
 ## `http` - use http-rpc backend (default)
 ##
 vcs.scm_app_implementation = http
 
 ## Push/Pull operations hooks protocol, available options are:
-## `pyro4` - use pyro4 server
 ## `http` - use http-rpc backend (default)
 vcs.hooks.protocol = http
 
 vcs.server.log_level = debug
 ## Start VCSServer with this instance as a subprocess, usefull for development
 vcs.start_server = true
 
 ## List of enabled VCS backends, available options are:
 ## `hg`  - mercurial
 ## `git` - git
 ## `svn` - subversion
 vcs.backends = hg, git, svn
 
 vcs.connection_timeout = 3600
 ## Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ## Available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible
 #vcs.svn.compatible_version = pre-1.8-compatible
 
 
 ############################################################
 ### Subversion proxy support (mod_dav_svn)               ###
 ### Maps RhodeCode repo groups into SVN paths for Apache ###
 ############################################################
 ## Enable or disable the config file generation.
 svn.proxy.generate_config = false
 ## Generate config file with `SVNListParentPath` set to `On`.
 svn.proxy.list_parent_path = true
 ## Set location and file name of generated config file.
 svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
 ## Used as a prefix to the `Location` block in the generated config file.
 ## In most cases it should be set to `/`.
 svn.proxy.location_root = /
 ## Command to reload the mod dav svn configuration on change.
 ## Example: `/etc/init.d/apache2 reload`
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 ## If the timeout expires before the reload command finishes, the command will
 ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
 #svn.proxy.reload_timeout = 10
 
 ## Dummy marker to add new entries after.
 ## Add any custom entries below. Please don't remove.
 custom.conf = 1
 
 
 ################################
 ### LOGGING CONFIGURATION   ####
 ################################
 [loggers]
 keys = root, routes, rhodecode, sqlalchemy, beaker, templates
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, color_formatter, color_formatter_sql
 
 #############
 ## LOGGERS ##
 #############
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_routes]
 level = DEBUG
 handlers =
 qualname = routes.middleware
 ## "level = DEBUG" logs the route matched and routing variables.
 propagate = 1
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_templates]
 level = INFO
 handlers =
 qualname = pylons.templating
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
 ##############
 ## HANDLERS ##
 ##############
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr, )
 level = DEBUG
 formatter = color_formatter
 
 [handler_console_sql]
 class = StreamHandler
 args = (sys.stderr, )
 level = DEBUG
 formatter = color_formatter_sql
 
 ################
 ## FORMATTERS ##
 ################
 
 [formatter_generic]
-class = rhodecode.lib.logging_formatter.Pyro4AwareFormatter
+class = rhodecode.lib.logging_formatter.ExceptionAwareFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S

configs/gunicorn_config.py

@@ -1,90 +1,90 b''
 """
 gunicorn config extension and hooks. Sets additional configuration that is
 available post the .ini config.
 
 - workers = ${cpu_number}
 - threads = 1
 - proc_name = ${gunicorn_proc_name}
 - worker_class = sync
 - worker_connections = 10
 - max_requests = 1000
 - max_requests_jitter = 30
 - timeout = 21600
 
 """
 
 import multiprocessing
 import sys
 import threading
 import traceback
 
 
 # GLOBAL
 errorlog = '-'
 accesslog = '-'
 loglevel = 'debug'
 
 # SECURITY
 limit_request_line = 4094
 limit_request_fields = 100
 limit_request_field_size = 8190
 
 # SERVER MECHANICS
 # None == system temp dir
 worker_tmp_dir = None
 tmp_upload_dir = None
 
 # Custom log format
 access_log_format = (
-    '%(t)s GNCRN %(p)-8s %(h)-15s rqt:%(L)s %(s)s %(b)s "%(m)s:%(U)s %(q)s" usr:%(u)s "%(f)s" "%(a)s"')
+    '%(t)s GNCRN %(p)-8s %(h)-15s rqt:%(L)s %(s)s %(b)-6s "%(m)s:%(U)s %(q)s" usr:%(u)s "%(f)s" "%(a)s"')
 
 # self adjust workers based on CPU count
 # workers = multiprocessing.cpu_count() * 2 + 1
 
 
 def post_fork(server, worker):
     server.log.info("[<%-10s>] WORKER spawned", worker.pid)
 
 
 def pre_fork(server, worker):
     pass
 
 
 def pre_exec(server):
     server.log.info("Forked child, re-executing.")
 
 
 def when_ready(server):
     server.log.info("Server is ready. Spawning workers")
 
 
 def worker_int(worker):
     worker.log.info("[<%-10s>] worker received INT or QUIT signal", worker.pid)
 
     # get traceback info, on worker crash
     id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
     code = []
     for thread_id, stack in sys._current_frames().items():
         code.append(
             "\n# Thread: %s(%d)" % (id2name.get(thread_id, ""), thread_id))
         for fname, lineno, name, line in traceback.extract_stack(stack):
             code.append('File: "%s", line %d, in %s' % (fname, lineno, name))
             if line:
                 code.append("  %s" % (line.strip()))
     worker.log.debug("\n".join(code))
 
 
 def worker_abort(worker):
     worker.log.info("[<%-10s>] worker received SIGABRT signal", worker.pid)
 
 
 def pre_request(worker, req):
     return
     worker.log.debug("[<%-10s>] PRE WORKER: %s %s",
                      worker.pid, req.method, req.path)
 
 
 def post_request(worker, req, environ, resp):
     return
     worker.log.debug("[<%-10s>] POST WORKER: %s %s resp: %s", worker.pid,
                      req.method, req.path, resp.status_code)
  No newline at end of file

configs/production.ini

@@ -1,650 +1,647 b''
 
 
 ################################################################################
 ##                 RHODECODE COMMUNITY EDITION CONFIGURATION                  ##
 # The %(here)s variable will be replaced with the parent directory of this file#
 ################################################################################
 
 [DEFAULT]
 debug = true
 
 ################################################################################
 ##                            EMAIL CONFIGURATION                             ##
 ## Uncomment and replace with the email address which should receive          ##
 ## any error reports after an application crash                               ##
 ## Additionally these settings will be used by the RhodeCode mailing system   ##
 ################################################################################
 
 ## prefix all emails subjects with given prefix, helps filtering out emails
 #email_prefix = [RhodeCode]
 
 ## email FROM address all mails will be sent
 #app_email_from = rhodecode-noreply@localhost
 
 ## Uncomment and replace with the address which should receive any error report
 ## note: using appenlight for error handling doesn't need this to be uncommented
 #email_to = admin@localhost
 
 ## in case of Application errors, sent an error email form
 #error_email_from = rhodecode_error@localhost
 
 ## additional error message to be send in case of server crash
 #error_message =
 
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 ## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
 #smtp_auth =
 
 [server:main]
 ## COMMON ##
 host = 127.0.0.1
 port = 5000
 
 ##################################
 ##     WAITRESS WSGI SERVER     ##
 ## Recommended for Development  ##
 ##################################
 
 #use = egg:waitress#main
 ## number of worker threads
 #threads = 5
 ## MAX BODY SIZE 100GB
 #max_request_body_size = 107374182400
 ## Use poll instead of select, fixes file descriptors limits problems.
 ## May not work on old windows systems.
 #asyncore_use_poll = true
 
 
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
 ## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
 ## when this option is set to more than one worker, recommended
 ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
 ## The `instance_id = *` must be set in the [app:main] section below
 workers = 2
 ## number of threads for each of the worker, must be set to 1 for gevent
 ## generally recommened to be at 1
 #threads = 1
 ## process name
 proc_name = rhodecode
 ## type of worker class, one of sync, gevent
 ## recommended for bigger setup is using of of other than sync one
 worker_class = sync
 ## The maximum number of simultaneous clients. Valid only for Gevent
 #worker_connections = 10
 ## max number of requests that worker will handle before being gracefully
 ## restarted, could prevent memory leaks
 max_requests = 1000
 max_requests_jitter = 30
 ## amount of time a worker can spend with handling a request before it
 ## gets killed and restarted. Set to 6hrs
 timeout = 21600
 
 
 ## prefix middleware for RhodeCode.
 ## recommended when using proxy setup.
 ## allows to set RhodeCode under a prefix in server.
 ## eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
 ## And set your prefix like: `prefix = /custom_prefix`
 ## be sure to also set beaker.session.cookie_path = /custom_prefix if you need
 ## to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
 
 [app:main]
 use = egg:rhodecode-enterprise-ce
 
 ## enable proxy prefix middleware, defined above
 #filter-with = proxy-prefix
 
 ## encryption key used to encrypt social plugin tokens,
 ## remote_urls with credentials etc, if not set it defaults to
 ## `beaker.session.secret`
 #rhodecode.encrypted_values.secret =
 
 ## decryption strict mode (enabled by default). It controls if decryption raises
 ## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
 ## return gzipped responses from Rhodecode (static files/application)
 gzip_responses = false
 
 ## autogenerate javascript routes file on startup
 generate_js_files = false
 
 ## Optional Languages
 ## en(default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ## perform a full repository scan on each server start, this should be
 ## set to false after first startup, to allow faster server restarts.
 startup.import_repos = false
 
 ## Uncomment and set this path to use archive download cache.
 ## Once enabled, generated archives will be cached at this location
 ## and served from the cache during subsequent requests for the same archive of
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
 ## change this to unique ID for security
 app_instance_uuid = rc-production
 
 ## cut off limit for large diffs (size in bytes)
 cut_off_limit_diff = 1024000
 cut_off_limit_file = 256000
 
 ## use cache version of scm repo everywhere
 vcs_full_cache = true
 
 ## force https in RhodeCode, fixes https redirects, assumes it's always https
 ## Normally this is controlled by proper http flags sent from http server
 force_https = false
 
 ## use Strict-Transport-Security headers
 use_htsts = false
 
 ## number of commits stats will parse on each iteration
 commit_parse_limit = 25
 
 ## git rev filter option, --all is the default filter, if you need to
 ## hide all refs in changelog switch this to --branches --tags
 git_rev_filter = --branches --tags
 
 # Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ## RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
 ## url that does rewrites to _admin/gists/{gistid}.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ## List of controllers (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
 ## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ##
 ## Syntax is ControllerClass:function_pattern.
 ## To enable access to raw_files put `FilesController:raw`.
 ## To enable access to patches add `ChangesetController:changeset_patch`.
 ## The list should be "," separated and on a single line.
 ##
 ## Recommended controllers to enable:
 #    ChangesetController:changeset_patch,
 #    ChangesetController:changeset_raw,
 #    FilesController:raw,
 #    FilesController:archivefile,
 #    GistsController:*,
 api_access_controllers_whitelist =
 
 ## default encoding used to convert from and to unicode
 ## can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
 ## multiple instances of rhodecode, make sure it's globally unique for
 ## all running rhodecode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
 ## authentication settings. Then you can enable e.g. the internal rhodecode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
 ## egg:rhodecode-enterprise-ce#rhodecode
 ## egg:rhodecode-enterprise-ce#pam
 ## egg:rhodecode-enterprise-ce#ldap
 ## egg:rhodecode-enterprise-ce#jasig_cas
 ## egg:rhodecode-enterprise-ce#headers
 ## egg:rhodecode-enterprise-ce#crowd
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ## alternative return HTTP header for failed authentication. Default HTTP
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ## This will be served instead of default 401 on bad authnetication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
 ## ret_code directly, use 401 initially (Which triggers credentials prompt)
 ## and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ## locking return code. When repository is locked return this HTTP code. 2XX
 ## codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ## allows to change the repository location in settings page
 allow_repo_location_change = true
 
 ## allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ## generated license token, goto license page in RhodeCode settings to obtain
 ## new token
 license_token =
 
 ## supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 ## supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = prod
 
 ## Display extended labs settings
 labs_settings_active = true
 
 ####################################
 ###        CELERY CONFIG        ####
 ####################################
 use_celery = false
 broker.host = localhost
 broker.vhost = rabbitmqhost
 broker.port = 5672
 broker.user = rabbitmq
 broker.password = qweqwe
 
 celery.imports = rhodecode.lib.celerylib.tasks
 
 celery.result.backend = amqp
 celery.result.dburi = amqp://
 celery.result.serialier = json
 
 #celery.send.task.error.emails = true
 #celery.amqp.task.result.expires = 18000
 
 celeryd.concurrency = 2
 #celeryd.log.file = celeryd.log
 celeryd.log.level = debug
 celeryd.max.tasks.per.child = 1
 
 ## tasks will never be sent to the queue, but executed locally instead.
 celery.always.eager = false
 
 ####################################
 ###         BEAKER CACHE        ####
 ####################################
 # default cache dir for templates.  Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk
 cache_dir = %(here)s/data
 
 ## locking and default file storage for Beaker. Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk/cache/beaker_data
 beaker.cache.data_dir = %(here)s/data/cache/beaker_data
 beaker.cache.lock_dir = %(here)s/data/cache/beaker_lock
 
 beaker.cache.regions = super_short_term, short_term, long_term, sql_cache_short, auth_plugins, repo_cache_long
 
 beaker.cache.super_short_term.type = memory
 beaker.cache.super_short_term.expire = 10
 beaker.cache.super_short_term.key_length = 256
 
 beaker.cache.short_term.type = memory
 beaker.cache.short_term.expire = 60
 beaker.cache.short_term.key_length = 256
 
 beaker.cache.long_term.type = memory
 beaker.cache.long_term.expire = 36000
 beaker.cache.long_term.key_length = 256
 
 beaker.cache.sql_cache_short.type = memory
 beaker.cache.sql_cache_short.expire = 10
 beaker.cache.sql_cache_short.key_length = 256
 
 ## default is memory cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.auth_plugins.type = ext:database
 #beaker.cache.auth_plugins.lock_dir = %(here)s/data/cache/auth_plugin_lock
 #beaker.cache.auth_plugins.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.cache.auth_plugins.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.cache.auth_plugins.sa.pool_recycle = 3600
 #beaker.cache.auth_plugins.sa.pool_size = 10
 #beaker.cache.auth_plugins.sa.max_overflow = 0
 
 beaker.cache.repo_cache_long.type = memorylru_base
 beaker.cache.repo_cache_long.max_items = 4096
 beaker.cache.repo_cache_long.expire = 2592000
 
 ## default is memorylru_base cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.repo_cache_long.type = ext:memcached
 #beaker.cache.repo_cache_long.url = localhost:11211
 #beaker.cache.repo_cache_long.expire = 1209600
 #beaker.cache.repo_cache_long.key_length = 256
 
 ####################################
 ###       BEAKER SESSION        ####
 ####################################
 
 ## .session.type is type of storage options for the session, current allowed
 ## types are file, ext:memcached, ext:database, and memory (default).
 beaker.session.type = file
 beaker.session.data_dir = %(here)s/data/sessions/data
 
 ## db based session, fast, and allows easy management over logged in users
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = production-rc-uytcxaz
 beaker.session.lock_dir = %(here)s/data/sessions/lock
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = key_for_encryption
 #beaker.session.validate_key = validation_key
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 ## Path to use for the cookie. Set to prefix if you use prefix middleware
 #beaker.session.cookie_path = /custom_prefix
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
 
 ## auto save the session to not to use .save()
 beaker.session.auto = false
 
 ## default cookie expiration time in seconds, set to `true` to set expire
 ## at browser close
 #beaker.session.cookie_expires = 3600
 
 ###################################
 ## SEARCH INDEXING CONFIGURATION ##
 ###################################
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
 ## WHOOSH Backend, doesn't require additional services to run
 ## it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
 ########################################
 ###    CHANNELSTREAM CONFIG         ####
 ########################################
 ## channelstream enables persistent connections and live notification
 ## in the system. It's also used by the chat system
 channelstream.enabled = false
 
 ## server address for channelstream server on the backend
 channelstream.server = 127.0.0.1:9800
 
 ## location of the channelstream server from outside world
 ## use ws:// for http or wss:// for https. This address needs to be handled
 ## by external HTTP server such as Nginx or Apache
 ## see nginx/apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = secret
 channelstream.history.location = %(here)s/channelstream_history
 
 ## Internal application path that Javascript uses to connect into.
 ## If you use proxy-prefix the prefix should be added before /_channelstream
 channelstream.proxy_path = /_channelstream
 
 
 ###################################
 ##       APPENLIGHT CONFIG       ##
 ###################################
 
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
 ## appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
 # used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
 
 ## enables 404 error logging (default False)
 appenlight.report_404 = false
 
 ## time in seconds after request is considered being slow (default 1)
 appenlight.slow_request_time = 1
 
 ## record slow requests in application
 ## (needs to be enabled for slow datastore recording and time tracking)
 appenlight.slow_requests = true
 
 ## enable hooking to application loggers
 appenlight.logging = true
 
 ## minimum log level for log capture
 appenlight.logging.level = WARNING
 
 ## send logs only from erroneous/slow requests
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
 ## list of additonal keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
 ## start with HTTP* this list be extended with additional keywords here
 appenlight.environ_keys_whitelist =
 
 ## list of keywords that should be blanked from request object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always blank keys that contain following words
 ## 'password', 'passwd', 'pwd', 'auth_tkt', 'secret', 'csrf'
 ## this list be extended with additional keywords set here
 appenlight.request_keys_blacklist =
 
 ## list of namespaces that should be ignores when gathering log entries
 ## can be string with comma separated list of namespaces
 ## (by default the client ignores own entries: appenlight_client.client)
 appenlight.log_namespace_blacklist =
 
 
 ################################################################################
 ## WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*  ##
 ## Debug mode will enable the interactive debugging tool, allowing ANYONE to  ##
 ## execute malicious code after an exception is raised.                       ##
 ################################################################################
 set debug = false
 
 
 ###########################################
 ###   MAIN RHODECODE DATABASE CONFIG    ###
 ###########################################
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
 sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 
 # see sqlalchemy docs for other advanced settings
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
 ## recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
 ## the number of connections to keep open inside the connection pool.
 ## 0 indicates no limit
 #sqlalchemy.db1.pool_size = 5
 
 ## the number of connections to allow in connection pool "overflow", that is
 ## connections that can be opened above and beyond the pool_size setting,
 ## which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 
 ##################
 ### VCS CONFIG ###
 ##################
 vcs.server.enable = true
 vcs.server = localhost:9900
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
-## `pyro4` - use pyro4 server
 ## `http` - use http-rpc backend (default)
 vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
-## `pyro4` - use pyro4 server
 ## `http` - use http-rpc backend (default)
 ##
 vcs.scm_app_implementation = http
 
 ## Push/Pull operations hooks protocol, available options are:
-## `pyro4` - use pyro4 server
 ## `http` - use http-rpc backend (default)
 vcs.hooks.protocol = http
 
 vcs.server.log_level = info
 ## Start VCSServer with this instance as a subprocess, usefull for development
 vcs.start_server = false
 
 ## List of enabled VCS backends, available options are:
 ## `hg`  - mercurial
 ## `git` - git
 ## `svn` - subversion
 vcs.backends = hg, git, svn
 
 vcs.connection_timeout = 3600
 ## Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ## Available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible
 #vcs.svn.compatible_version = pre-1.8-compatible
 
 
 ############################################################
 ### Subversion proxy support (mod_dav_svn)               ###
 ### Maps RhodeCode repo groups into SVN paths for Apache ###
 ############################################################
 ## Enable or disable the config file generation.
 svn.proxy.generate_config = false
 ## Generate config file with `SVNListParentPath` set to `On`.
 svn.proxy.list_parent_path = true
 ## Set location and file name of generated config file.
 svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
 ## Used as a prefix to the `Location` block in the generated config file.
 ## In most cases it should be set to `/`.
 svn.proxy.location_root = /
 ## Command to reload the mod dav svn configuration on change.
 ## Example: `/etc/init.d/apache2 reload`
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 ## If the timeout expires before the reload command finishes, the command will
 ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
 #svn.proxy.reload_timeout = 10
 
 ## Dummy marker to add new entries after.
 ## Add any custom entries below. Please don't remove.
 custom.conf = 1
 
 
 ################################
 ### LOGGING CONFIGURATION   ####
 ################################
 [loggers]
 keys = root, routes, rhodecode, sqlalchemy, beaker, templates
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, color_formatter, color_formatter_sql
 
 #############
 ## LOGGERS ##
 #############
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_routes]
 level = DEBUG
 handlers =
 qualname = routes.middleware
 ## "level = DEBUG" logs the route matched and routing variables.
 propagate = 1
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_templates]
 level = INFO
 handlers =
 qualname = pylons.templating
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
 ##############
 ## HANDLERS ##
 ##############
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr, )
 level = INFO
 formatter = generic
 
 [handler_console_sql]
 class = StreamHandler
 args = (sys.stderr, )
 level = WARN
 formatter = generic
 
 ################
 ## FORMATTERS ##
 ################
 
 [formatter_generic]
-class = rhodecode.lib.logging_formatter.Pyro4AwareFormatter
+class = rhodecode.lib.logging_formatter.ExceptionAwareFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S

default.nix

@@ -1,240 +1,241 b''
 # Nix environment for the community edition
 #
 # This shall be as lean as possible, just producing the Enterprise
 # derivation. For advanced tweaks to pimp up the development environment we use
 # "shell.nix" so that it does not have to clutter this file.
 
 args@
 { pythonPackages ? "python27Packages"
 , pythonExternalOverrides ? self: super: {}
 , doCheck ? true
 , ...
 }:
 
 let
 
   # Use nixpkgs from args or import them. We use this indirect approach
   # through args to be able to use the name `pkgs` for our customized packages.
   # Otherwise we will end up with an infinite recursion.
   nixpkgs = args.pkgs or (import <nixpkgs> { });
 
   # johbo: Interim bridge which allows us to build with the upcoming
   # nixos.16.09 branch (unstable at the moment of writing this note) and the
   # current stable nixos-16.03.
   backwardsCompatibleFetchgit = { ... }@args:
     let
       origSources = nixpkgs.fetchgit args;
     in
     nixpkgs.lib.overrideDerivation origSources (oldAttrs: {
       NIX_PREFETCH_GIT_CHECKOUT_HOOK = ''
         find $out -name '.git*' -print0 | xargs -0 rm -rf
       '';
     });
 
   # Create a customized version of nixpkgs which should be used throughout the
   # rest of this file.
   pkgs = nixpkgs.overridePackages (self: super: {
     fetchgit = backwardsCompatibleFetchgit;
   });
 
   # Evaluates to the last segment of a file system path.
   basename = path: with pkgs.lib; last (splitString "/" path);
 
   # source code filter used as arugment to builtins.filterSource.
   src-filter = path: type: with pkgs.lib;
     let
       ext = last (splitString "." path);
     in
       !builtins.elem (basename path) [
         ".git" ".hg" "__pycache__" ".eggs"
         "bower_components" "node_modules"
         "build" "data" "result" "tmp"] &&
       !builtins.elem ext ["egg-info" "pyc"] &&
       # TODO: johbo: This check is wrong, since "path" contains an absolute path,
       # it would still be good to restore it since we want to ignore "result-*".
       !hasPrefix "result" path;
 
   basePythonPackages = with builtins; if isAttrs pythonPackages
     then pythonPackages
     else getAttr pythonPackages pkgs;
 
   buildBowerComponents =
     pkgs.buildBowerComponents or
     (import ./pkgs/backport-16.03-build-bower-components.nix { inherit pkgs; });
 
   sources = pkgs.config.rc.sources or {};
   version = builtins.readFile ./rhodecode/VERSION;
   rhodecode-enterprise-ce-src = builtins.filterSource src-filter ./.;
 
   nodeEnv = import ./pkgs/node-default.nix {
     inherit pkgs;
   };
   nodeDependencies = nodeEnv.shell.nodeDependencies;
 
   bowerComponents = buildBowerComponents {
     name = "enterprise-ce-${version}";
     generated = ./pkgs/bower-packages.nix;
     src = rhodecode-enterprise-ce-src;
   };
 
   pythonGeneratedPackages = self: basePythonPackages.override (a: {
     inherit self;
   })
   // (scopedImport {
     self = self;
     super = basePythonPackages;
     inherit pkgs;
     inherit (pkgs) fetchurl fetchgit;
   } ./pkgs/python-packages.nix);
 
   pythonOverrides = import ./pkgs/python-packages-overrides.nix {
     inherit
       basePythonPackages
       pkgs;
   };
 
   pythonLocalOverrides = self: super: {
     rhodecode-enterprise-ce =
       let
         linkNodeAndBowerPackages = ''
           echo "Export RhodeCode CE path"
           export RHODECODE_CE_PATH=${rhodecode-enterprise-ce-src}
           echo "Link node packages"
           rm -fr node_modules
           mkdir node_modules
           # johbo: Linking individual packages allows us to run "npm install"
           # inside of a shell to try things out. Re-entering the shell will
           # restore a clean environment.
           ln -s ${nodeDependencies}/lib/node_modules/* node_modules/
 
           echo "DONE: Link node packages"
 
           echo "Link bower packages"
           rm -fr bower_components
           mkdir bower_components
 
           ln -s ${bowerComponents}/bower_components/* bower_components/
           echo "DONE: Link bower packages"
         '';
       in super.rhodecode-enterprise-ce.override (attrs: {
 
       inherit
         doCheck
         version;
       name = "rhodecode-enterprise-ce-${version}";
       releaseName = "RhodeCodeEnterpriseCE-${version}";
       src = rhodecode-enterprise-ce-src;
+      dontStrip = true; # prevent strip, we don't need it.
 
       buildInputs =
         attrs.buildInputs ++
         (with self; [
           pkgs.nodePackages.bower
           pkgs.nodePackages.grunt-cli
           pkgs.subversion
           pytest-catchlog
           rhodecode-testdata
         ]);
 
       #TODO: either move this into overrides, OR use the new machanics from
       # pip2nix and requiremtn.txt file
       propagatedBuildInputs = attrs.propagatedBuildInputs ++ (with self; [
         rhodecode-tools
       ]);
 
       # TODO: johbo: Make a nicer way to expose the parts. Maybe
       # pkgs/default.nix?
       passthru = {
         inherit
           bowerComponents
           linkNodeAndBowerPackages
           myPythonPackagesUnfix
           pythonLocalOverrides;
         pythonPackages = self;
       };
 
       LC_ALL = "en_US.UTF-8";
       LOCALE_ARCHIVE =
         if pkgs.stdenv ? glibc
         then "${pkgs.glibcLocales}/lib/locale/locale-archive"
         else "";
 
       preCheck = ''
         export PATH="$out/bin:$PATH"
       '';
 
       postCheck = ''
         rm -rf $out/lib/${self.python.libPrefix}/site-packages/pytest_pylons
         rm -rf $out/lib/${self.python.libPrefix}/site-packages/rhodecode/tests
       '';
 
       preBuild = linkNodeAndBowerPackages + ''
         grunt
         rm -fr node_modules
       '';
 
       postInstall = ''
         # python based programs need to be wrapped
         ln -s ${self.supervisor}/bin/supervisor* $out/bin/
         ln -s ${self.gunicorn}/bin/gunicorn $out/bin/
         ln -s ${self.PasteScript}/bin/paster $out/bin/
         ln -s ${self.channelstream}/bin/channelstream $out/bin/
         ln -s ${self.pyramid}/bin/* $out/bin/  #*/
 
         # rhodecode-tools
         # TODO: johbo: re-think this. Do the tools import anything from enterprise?
         ln -s ${self.rhodecode-tools}/bin/rhodecode-* $out/bin/
 
         # note that condition should be restricted when adding further tools
         for file in $out/bin/*; do  #*/
           wrapProgram $file \
               --prefix PYTHONPATH : $PYTHONPATH \
               --prefix PATH : $PATH \
               --set PYTHONHASHSEED random
         done
 
         mkdir $out/etc
         cp configs/production.ini $out/etc
 
         echo "Writing meta information for rccontrol to nix-support/rccontrol"
         mkdir -p $out/nix-support/rccontrol
         cp -v rhodecode/VERSION $out/nix-support/rccontrol/version
         echo "DONE: Meta information for rccontrol written"
 
         # TODO: johbo: Make part of ac-tests
         if [ ! -f rhodecode/public/js/scripts.js ]; then
           echo "Missing scripts.js"
           exit 1
         fi
         if [ ! -f rhodecode/public/css/style.css ]; then
           echo "Missing style.css"
           exit 1
         fi
       '';
 
     });
 
     rhodecode-testdata = import "${rhodecode-testdata-src}/default.nix" {
     inherit
       doCheck
       pkgs
       pythonPackages;
     };
 
   };
 
   rhodecode-testdata-src = sources.rhodecode-testdata or (
     pkgs.fetchhg {
       url = "https://code.rhodecode.com/upstream/rc_testdata";
-      rev = "v0.9.0";
-      sha256 = "0k0ccb7cncd6mmzwckfbr6l7fsymcympwcm948qc3i0f0m6bbg1y";
+      rev = "v0.10.0";
+      sha256 = "0zn9swwvx4vgw4qn8q3ri26vvzgrxn15x6xnjrysi1bwmz01qjl0";
   });
 
   # Apply all overrides and fix the final package set
   myPythonPackagesUnfix = with pkgs.lib;
     (extends pythonExternalOverrides
     (extends pythonLocalOverrides
     (extends pythonOverrides
              pythonGeneratedPackages)));
   myPythonPackages = (pkgs.lib.fix myPythonPackagesUnfix);
 
 in myPythonPackages.rhodecode-enterprise-ce

docs-internal/vcs/index.rst

@@ -1,23 +1,22 b''
 
 ======================================
  VCS client and VCSServer integration
 ======================================
 
 Enterprise uses the VCSServer as a backend to provide version control
 functionalities. This section describes the components in Enterprise which talk
 to the VCSServer.
 
 The client library is implemented in :mod:`rhodecode.lib.vcs`. For HTTP based
 access of the command line clients special middlewares and utilities are
 implemented in :mod:`rhodecode.lib.middleware`.
 
 
 
 
 .. toctree::
    :maxdepth: 2
 
-   http-transition
    middleware
    vcsserver
    subversion

docs/admin/apache-conf-example.rst

@@ -1,87 +1,87 b''
 .. _apache-conf-eg:
 
 Apache Configuration Example
 ----------------------------
 
 Use the following example to configure Apache as a your web server.
 Below config if for an Apache Reverse Proxy configuration.
 
 .. note::
 
    Apache requires the following modules to be enabled. Below is an example
    how to enable them on Ubuntu Server
 
 
 .. code-block:: bash
 
     $ sudo a2enmod proxy
     $ sudo a2enmod proxy_http
     $ sudo a2enmod proxy_balancer
     $ sudo a2enmod headers
     $ sudo a2enmod ssl
     $ sudo a2enmod rewrite
 
     # requires Apache 2.4+, required to handle websockets/channelstream
     $ sudo a2enmod proxy_wstunnel
 
 
 .. code-block:: apache
 
     ## HTTP to HTTPS rewrite
     <VirtualHost *:80>
        ServerName rhodecode.myserver.com
        DocumentRoot /var/www/html
        Redirect permanent / https://rhodecode.myserver.com/
     </VirtualHost>
 
     ## MAIN SSL enabled server
     <VirtualHost *:443>
 
         ServerName rhodecode.myserver.com
         ServerAlias rhodecode.myserver.com
 
         ## serve static files by Apache, recommended for performance
         #Alias /_static /home/ubuntu/.rccontrol/community-1/static
 
         RequestHeader set X-Forwarded-Proto "https"
 
         ## channelstream websocket handling
         ProxyPass /_channelstream ws://localhost:9800
         ProxyPassReverse /_channelstream ws://localhost:9800
 
         <Proxy *>
           Order allow,deny
           Allow from all
         </Proxy>
 
         # Directive to properly generate url (clone url) for RhodeCode
         ProxyPreserveHost On
 
         # Url to running RhodeCode instance. This is shown as `- URL:` when
         # running rccontrol status.
-        ProxyPass / http://127.0.0.1:10002/
+        ProxyPass / http://127.0.0.1:10002/ timeout=7200 Keepalive=On
         ProxyPassReverse / http://127.0.0.1:10002/
 
         # strict http prevents from https -> http downgrade
         Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
 
         # Set x-frame options
         Header always append X-Frame-Options SAMEORIGIN
 
         # To enable https use line below
         # SetEnvIf X-Url-Scheme https HTTPS=1
 
         # SSL setup
         SSLEngine On
         SSLCertificateFile /etc/apache2/ssl/rhodecode.myserver.pem
         SSLCertificateKeyFile /etc/apache2/ssl/rhodecode.myserver.key
 
         SSLProtocol             all -SSLv2 -SSLv3
         SSLCipherSuite          ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
         SSLHonorCipherOrder     on
 
         # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
         #SSLOpenSSLConfCmd DHParameters "/etc/apache2/dhparam.pem"
 
     </VirtualHost>
 

docs/admin/enable-debug.rst

@@ -1,130 +1,124 b''
 .. _debug-mode:
 
 Enabling Debug Mode
 -------------------
 
 To enable debug mode on a |RCE| instance you need to set the debug property
 in the :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file. To
 do this, use the following steps
 
 1. Open the file and set the ``debug`` line to ``true``
 2. Restart you instance using the ``rccontrol restart`` command,
    see the following example:
 
 You can also set the log level, the follow are the valid options;
 ``debug``, ``info``, ``warning``, or ``fatal``.
 
 .. code-block:: ini
 
     [DEFAULT]
     debug = true
     pdebug = false
 
 .. code-block:: bash
 
     # Restart your instance
     $ rccontrol restart enterprise-1
     Instance "enterprise-1" successfully stopped.
     Instance "enterprise-1" successfully started.
 
 Debug and Logging Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Further debugging and logging settings can also be set in the
 :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
 
 In the logging section, the various packages that run with |RCE| can have
 different debug levels set. If you want to increase the logging level change
 ``level = DEBUG`` line to one of the valid options.
 
 You also need to change the log level for handlers. See the example
 ``##handler`` section below. The ``handler`` level takes the same options as
 the ``debug`` level.
 
 .. code-block:: ini
 
     ################################
     ### LOGGING CONFIGURATION   ####
     ################################
     [loggers]
-    keys = root, routes, rhodecode, sqlalchemy, beaker, pyro4, templates
+    keys = root, routes, rhodecode, sqlalchemy, beaker, templates
 
     [handlers]
     keys = console, console_sql, file, file_rotating
 
     [formatters]
     keys = generic, color_formatter, color_formatter_sql
 
     #############
     ## LOGGERS ##
     #############
     [logger_root]
     level = NOTSET
     handlers = console
 
     [logger_routes]
     level = DEBUG
     handlers =
     qualname = routes.middleware
     ## "level = DEBUG" logs the route matched and routing variables.
     propagate = 1
 
     [logger_beaker]
     level = DEBUG
     handlers =
     qualname = beaker.container
     propagate = 1
 
-    [logger_pyro4]
-    level = DEBUG
-    handlers =
-    qualname = Pyro4
-    propagate = 1
-
     [logger_templates]
     level = INFO
     handlers =
     qualname = pylons.templating
     propagate = 1
 
     [logger_rhodecode]
     level = DEBUG
     handlers =
     qualname = rhodecode
     propagate = 1
 
     [logger_sqlalchemy]
     level = INFO
     handlers = console_sql
     qualname = sqlalchemy.engine
     propagate = 0
 
     ##############
     ## HANDLERS ##
     ##############
 
     [handler_console]
     class = StreamHandler
     args = (sys.stderr,)
     level = INFO
     formatter = generic
 
     [handler_console_sql]
     class = StreamHandler
     args = (sys.stderr,)
     level = WARN
     formatter = generic
 
     [handler_file]
     class = FileHandler
     args = ('rhodecode.log', 'a',)
     level = INFO
     formatter = generic
 
     [handler_file_rotating]
     class = logging.handlers.TimedRotatingFileHandler
     # 'D', 5 - rotate every 5days
     # you can set 'h', 'midnight'
     args = ('rhodecode.log', 'D', 5, 10,)
     level = INFO
     formatter = generic

docs/admin/svn-http.rst

@@ -1,138 +1,150 b''
 .. _svn-http:
 
 |svn| With Write Over HTTP
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To use |svn| with read/write support over the |svn| HTTP protocol, you have to
 configure the HTTP |svn| backend.
 
 Prerequisites
 =============
 
 - Enable HTTP support inside the admin VCS settings on your |RCE| instance
 - You need to install the following tools on the machine that is running an
   instance of |RCE|:
   ``Apache HTTP Server`` and ``mod_dav_svn``.
 
 
 Using Ubuntu 14.04 Distribution as an example execute the following:
 
 .. code-block:: bash
 
     $ sudo apt-get install apache2 libapache2-mod-svn
 
 Once installed you need to enable ``dav_svn``:
 
 .. code-block:: bash
 
     $ sudo a2enmod dav_svn
     $ sudo a2enmod headers
     $ sudo a2enmod authn_anon
 
 
 Configuring Apache Setup
 ========================
 
 .. tip::
 
    It is recommended to run Apache on a port other than 80, due to possible
    conflicts with other HTTP servers like nginx. To do this, set the
    ``Listen`` parameter in the ``/etc/apache2/ports.conf`` file, for example
    ``Listen 8090``.
 
 
 .. warning::
 
    Make sure your Apache instance which runs the mod_dav_svn module is
    only accessible by |RCE|. Otherwise everyone is able to browse
    the repositories or run subversion operations (checkout/commit/etc.).
 
 It is also recommended to run apache as the same user as |RCE|, otherwise
 permission issues could occur. To do this edit the ``/etc/apache2/envvars``
 
    .. code-block:: apache
 
       export APACHE_RUN_USER=rhodecode
       export APACHE_RUN_GROUP=rhodecode
 
 1. To configure Apache, create and edit a virtual hosts file, for example
    :file:`/etc/apache2/sites-enabled/default.conf`. Below is an example
    how to use one with auto-generated config ```mod_dav_svn.conf```
    from configured |RCE| instance.
 
 .. code-block:: apache
 
     <VirtualHost *:8090>
         ServerAdmin rhodecode-admin@localhost
         DocumentRoot /var/www/html
         ErrorLog ${'${APACHE_LOG_DIR}'}/error.log
         CustomLog ${'${APACHE_LOG_DIR}'}/access.log combined
         Include /home/user/.rccontrol/enterprise-1/mod_dav_svn.conf
     </VirtualHost>
 
 
 2. Go to the :menuselection:`Admin --> Settings --> VCS` page, and
    enable :guilabel:`Proxy Subversion HTTP requests`, and specify the
    :guilabel:`Subversion HTTP Server URL`.
 
 3. Open the |RCE| configuration file,
    :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
 
 4. Add the following configuration option in the ``[app:main]``
    section if you don't have it yet.
 
    This enables mapping of the created |RCE| repo groups into special
    |svn| paths. Each time a new repository group is created, the system will
    update the template file and create new mapping. Apache web server needs to
    be reloaded to pick up the changes on this file.
    To do this, simply configure `svn.proxy.reload_cmd` inside the .ini file.
    Example configuration:
 
 
 .. code-block:: ini
 
     ############################################################
     ### Subversion proxy support (mod_dav_svn)               ###
     ### Maps RhodeCode repo groups into SVN paths for Apache ###
     ############################################################
     ## Enable or disable the config file generation.
     svn.proxy.generate_config = true
     ## Generate config file with `SVNListParentPath` set to `On`.
     svn.proxy.list_parent_path = true
     ## Set location and file name of generated config file.
     svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
     ## Used as a prefix to the <Location> block in the generated config file.
     ## In most cases it should be set to `/`.
     svn.proxy.location_root = /
     ## Command to reload the mod dav svn configuration on change.
     ## Example: `/etc/init.d/apache2 reload`
     svn.proxy.reload_cmd = /etc/init.d/apache2 reload
     ## If the timeout expires before the reload command finishes, the command will
     ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
     #svn.proxy.reload_timeout = 10
 
 
 This would create a special template file called ```mod_dav_svn.conf```. We
 used that file path in the apache config above inside the Include statement.
-It's also possible to generate the config from the
-:menuselection:`Admin --> Settings --> VCS` page.
+It's also possible to manually generate the config from the
+:menuselection:`Admin --> Settings --> VCS` page by clicking a
+`Generate Apache Config` button.
+
+5. Now only things left is to enable svn support, and generate the initial
+   configuration.
+
+   - Select `Proxy subversion HTTP requests` checkbox
+   - Enter http://localhost:8090 into `Subversion HTTP Server URL`
+   - Click the `Generate Apache Config` button.
+
+This config will be automatically re-generated once an user-groups is added
+to properly map the additional paths generated.
+
 
 
 Using |svn|
 ===========
 
 Once |svn| has been enabled on your instance, you can use it with the
 following examples. For more |svn| information, see the `Subversion Red Book`_
 
 .. code-block:: bash
 
     # To clone a repository
     svn checkout http://my-svn-server.example.com/my-svn-repo
 
     # svn commit
     svn commit
 
 
 .. _Subversion Red Book: http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn
 
 .. _Ask Ubuntu: http://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue
  No newline at end of file

docs/admin/tuning-change-large-file-dir.rst

@@ -1,32 +1,14 b''
 .. _hg-lrg-loc:
 
 Change the |hg| Large Files Location
 ------------------------------------
 
 |RCE| manages |hg| larges files from the following default location
 :file:`/home/{user}/repos/.cache/largefiles`. If you wish to change this, use
 the following steps:
 
-1. Open ishell from the terminal and use it to log into the |RCE| database by
-   specifying the instance :file:`rhodecode.ini` file.
-
-.. code-block:: bash
-
-    # Open iShell from the terminal and set ini file
-    $ rccontrol ishell enterprise-1
-
-2. Run the following commands, and ensure that |RCE| has write access to the
-   new directory:
+1. Open :menuselection:`Admin --> Settings --> VCS` as super-admin.
 
-.. code-block:: bash
+In section called `Mercurial Settings` you can change where the largefiles
+objects should be stored.
 
-    # Once logged into the database, use SQL to redirect
-    # the large files location
-    In [1]: from rhodecode.model.settings import SettingsModel
-    In [2]: SettingsModel().get_ui_by_key('usercache')
-    Out[2]: <RhodeCodeUi[largefiles]usercache=>/mnt/hgfs/shared/workspace/xxxx/.cache/largefiles]>
-
-    In [3]: largefiles_cache = SettingsModel().get_ui_by_key('usercache')
-    In [4]: largefiles_cache.ui_value = '/new/path’
-    In [5]: Session().add(largefiles_cache);Session().commit()
-

docs/admin/tuning-rhodecode.rst

@@ -1,21 +1,22 b''
 .. _rhodecode-tuning-ref:
 
 Tuning |RCE|
 ============
 
 To customize your |RCE| |version| installation for maximum performance you
 may find some of the following methods useful.
 
 .. toctree::
 
    tuning-gunicorn
    tuning-vcs-memory-cache
    tuning-user-sessions-performance
    tuning-increase-db-performance
    tuning-scale-horizontally
    tuning-increase-cache-size
    tuning-mount-cache-memory
    tuning-change-encoding
    tuning-change-large-file-dir
+   tuning-change-lfs-dir
    tuning-hg-auth-loop
 

docs/admin/vcs-server.rst

@@ -1,304 +1,298 b''
 .. _vcs-server:
 
 VCS Server Management
 ---------------------
 
 The VCS Server handles |RCM| backend functionality. You need to configure
 a VCS Server to run with a |RCM| instance. If you do not, you will be missing
 the connection between |RCM| and its |repos|. This will cause error messages
 on the web interface. You can run your setup in the following configurations,
 currently the best performance is one VCS Server per |RCM| instance:
 
 * One VCS Server per |RCM| instance.
 * One VCS Server handling multiple instances.
 
 .. important::
 
    If your server locale settings are not correctly configured,
    |RCE| and the VCS Server can run into issues. See this `Ask Ubuntu`_ post
    which explains the problem and gives a solution.
 
 For more information, see the following sections:
 
 * :ref:`install-vcs`
 * :ref:`config-vcs`
 * :ref:`vcs-server-options`
 * :ref:`vcs-server-versions`
 * :ref:`vcs-server-maintain`
 * :ref:`vcs-server-config-file`
 * :ref:`svn-http`
 
 .. _install-vcs:
 
 VCS Server Installation
 ^^^^^^^^^^^^^^^^^^^^^^^
 
 To install a VCS Server, see
 :ref:`Installing a VCS server <control:install-vcsserver>`.
 
 .. _config-vcs:
 
 Hooking |RCE| to its VCS Server
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To configure a |RCE| instance to use a VCS server, see
 :ref:`Configuring the VCS Server connection <control:manually-vcsserver-ini>`.
 
 .. _vcs-server-options:
 
 |RCE| VCS Server Options
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 The following list shows the available options on the |RCM| side of the
 connection to the VCS Server. The settings are configured per
 instance in the
 :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
 
 .. rst-class:: dl-horizontal
 
     \vcs.backends <available-vcs-systems>
         Set a comma-separated list of the |repo| options available from the
         web interface. The default is ``hg, git, svn``,
         which is all |repo| types available.
 
     \vcs.connection_timeout <seconds>
         Set the length of time in seconds that the VCS Server waits for
         requests to process. After the timeout expires,
         the request is closed. The default is ``3600``. Set to a higher
         number if you experience network latency, or timeout issues with very
         large push/pull requests.
 
     \vcs.server.enable <boolean>
         Enable or disable the VCS Server. The available options are ``true`` or
         ``false``. The default is ``true``.
 
     \vcs.server <host:port>
         Set the host, either hostname or IP Address, and port of the VCS server
         you wish to run with your |RCM| instance.
 
 .. code-block:: ini
 
     ##################
     ### VCS CONFIG ###
     ##################
     # set this line to match your VCS Server
     vcs.server = 127.0.0.1:10004
     # Set to False to disable the VCS Server
     vcs.server.enable = True
     vcs.backends = hg, git, svn
     vcs.connection_timeout = 3600
 
 
 .. _vcs-server-versions:
 
 VCS Server Versions
 ^^^^^^^^^^^^^^^^^^^
 
 An updated version of the VCS Server is released with each |RCE| version. Use
 the VCS Server number that matches with the |RCE| version to pair the
 appropriate ones together. For |RCE| versions pre 3.3.0,
 VCS Server 1.X.Y works with |RCE| 3.X.Y, for example:
 
 * VCS Server 1.0.0 works with |RCE| 3.0.0
 * VCS Server 1.2.2 works with |RCE| 3.2.2
 
 For |RCE| versions post 3.3.0, the VCS Server and |RCE| version numbers
 match, for example:
 
 * VCS Server |release| works with |RCE| |release|
 
 .. _vcs-server-maintain:
 
 VCS Server Memory Optimization
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To configure the VCS server to manage the cache efficiently, you need to
 configure the following options in the
 :file:`/home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini` file. Once
 configured, restart the VCS Server.
 
 .. rst-class:: dl-horizontal
 
     \beaker.cache.repo_object.type = memorylru
         Configures the cache to discard the least recently used items.
         This setting takes the following valid options:
 
         * ``memorylru``: The default setting, which removes the least recently
           used items from the cache.
         * ``memory``: Runs the VCS Server without clearing the cache.
         * ``nocache``: Runs the VCS Server without a cache. This will
           dramatically reduce the VCS Server performance.
 
     \beaker.cache.repo_object.max_items = 100
         Sets the maximum number of items stored in the cache, before the cache
         starts to be cleared.
 
         As a general rule of thumb, running this value at 120 resulted in a
         5GB cache. Running it at 240 resulted in a 9GB cache. Your results
         will differ based on usage patterns and |repo| sizes.
 
         Tweaking this value to run at a fairly constant memory load on your
         server will help performance.
 
 To clear the cache completely, you can restart the VCS Server.
 
 .. important::
 
    While the VCS Server handles a restart gracefully on the web interface,
    it will drop connections during push/pull requests. So it is recommended
    you only perform this when there is very little traffic on the instance.
 
 Use the following example to restart your VCS Server,
 for full details see the :ref:`RhodeCode Control CLI <control:rcc-cli>`.
 
 .. code-block:: bash
 
     $ rccontrol status
 
 .. code-block:: vim
 
     - NAME: vcsserver-1
     - STATUS: RUNNING
     - TYPE: VCSServer
     - VERSION: 1.0.0
     - URL: http://127.0.0.1:10001
 
     $ rccontrol restart vcsserver-1
     Instance "vcsserver-1" successfully stopped.
     Instance "vcsserver-1" successfully started.
 
 .. _vcs-server-config-file:
 
 VCS Server Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 You can configure settings for multiple VCS Servers on your
 system using their individual configuration files. Use the following
 properties inside the configuration file to set up your system. The default
 location is :file:`home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini`.
 For a more detailed explanation of the logger levers, see :ref:`debug-mode`.
 
 .. rst-class:: dl-horizontal
 
     \host <ip-address>
         Set the host on which the VCS Server will run.
 
     \port <int>
         Set the port number on which the VCS Server will be available.
 
     \locale <locale_utf>
         Set the locale the VCS Server expects.
 
     \threadpool_size <int>
         Set the size of the threadpool used to communicate
         with the WSGI workers. This should be at least 6 times the number of
         WSGI worker processes.
 
     \timeout <seconds>
         Set the timeout for RPC communication in seconds.
 
 .. note::
 
    After making changes, you need to restart your VCS Server to pick them up.
 
 .. code-block:: ini
 
     ################################################################################
     # RhodeCode VCSServer - configuration                                          #
     #                                                                              #
     ################################################################################
 
     [DEFAULT]
     host = 127.0.0.1
     port = 9900
     locale = en_US.UTF-8
     # number of worker threads, this should be set based on a formula threadpool=N*6
     # where N is number of RhodeCode Enterprise workers, eg. running 2 instances
     # 8 gunicorn workers each would be 2 * 8 * 6 = 96, threadpool_size = 96
     threadpool_size = 16
     timeout = 0
 
     # cache regions, please don't change
     beaker.cache.regions = repo_object
     beaker.cache.repo_object.type = memorylru
     beaker.cache.repo_object.max_items = 1000
 
     # cache auto-expires after N seconds
     beaker.cache.repo_object.expire = 10
     beaker.cache.repo_object.enabled = true
 
 
     ################################
     ### LOGGING CONFIGURATION   ####
     ################################
     [loggers]
-    keys = root, vcsserver, pyro4, beaker
+    keys = root, vcsserver, beaker
 
     [handlers]
     keys = console
 
     [formatters]
     keys = generic
 
     #############
     ## LOGGERS ##
     #############
     [logger_root]
     level = NOTSET
     handlers = console
 
     [logger_vcsserver]
     level = DEBUG
     handlers =
     qualname = vcsserver
     propagate = 1
 
     [logger_beaker]
     level = DEBUG
     handlers =
     qualname = beaker
     propagate = 1
 
-    [logger_pyro4]
-    level = DEBUG
-    handlers =
-    qualname = Pyro4
-    propagate = 1
-
 
     ##############
     ## HANDLERS ##
     ##############
 
     [handler_console]
     class = StreamHandler
     args = (sys.stderr,)
     level = DEBUG
     formatter = generic
 
     [handler_file]
     class = FileHandler
     args = ('vcsserver.log', 'a',)
     level = DEBUG
     formatter = generic
 
     [handler_file_rotating]
     class = logging.handlers.TimedRotatingFileHandler
     # 'D', 5 - rotate every 5days
     # you can set 'h', 'midnight'
     args = ('vcsserver.log', 'D', 5, 10,)
     level = DEBUG
     formatter = generic
 
     ################
     ## FORMATTERS ##
     ################
 
     [formatter_generic]
     format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
     datefmt = %Y-%m-%d %H:%M:%S
 
 
 .. _Subversion Red Book: http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn
 
 .. _Ask Ubuntu: http://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue

docs/api/api.rst

@@ -1,207 +1,207 b''
 .. _api:
 
 API Documentation
 =================
 
 The |RCE| API uses a single scheme for calling all API methods. The API is
 implemented with JSON protocol in both directions. To send API requests to
 your instance of |RCE|, use the following URL format
 ``<your_server>/_admin``
 
 .. note::
 
    To use the API, you should configure the :file:`~/.rhoderc` file with
    access details per instance. For more information, see
    :ref:`config-rhoderc`.
 
 
 API ACCESS FOR WEB VIEWS
 ------------------------
 
 API access can also be turned on for each web view in |RCE| that is
 decorated with a `@LoginRequired` decorator. To enable API access, change
 the standard login decorator to `@LoginRequired(api_access=True)`.
 
 From |RCM| version 1.7.0 you can configure a white list
 of views that have API access enabled by default. To enable these,
 edit the |RCM| configuration ``.ini`` file. The default location is:
 
 * |RCM| Pre-2.2.7 :file:`root/rhodecode/data/production.ini`
 * |RCM| 3.0 :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
 
 To configure the white list, edit this section of the file. In this
 configuration example, API access is granted to the patch/diff raw file and
 archive.
 
 .. code-block:: ini
 
     ## List of controllers (using glob syntax) that AUTH TOKENS could be used for access.
     ## Adding ?auth_token = <token> to the url authenticates this request as if it
     ## came from the the logged in user who own this authentication token.
     ##
     ## Syntax is <ControllerClass>:<function_pattern>.
     ## The list should be "," separated and on a single line.
     ##
     api_access_controllers_whitelist = ChangesetController:changeset_patch,ChangesetController:changeset_raw,ilesController:raw,FilesController:archivefile,
 
 After this change, a |RCE| view can be accessed without login by adding a
 GET parameter ``?auth_token=<auth_token>`` to a url. For example to
 access the raw diff.
 
 .. code-block:: html
 
    http://<server>/<repo>/changeset-diff/<sha>?auth_token=<auth_token>
 
 By default this is only enabled on RSS/ATOM feed views. Exposing raw diffs is a
 good way to integrate with 3rd party services like code review, or build farms
 that could download archives.
 
 API ACCESS
 ----------
 
 All clients are required to send JSON-RPC spec JSON data.
 
 .. code-block:: bash
 
     {
         "id:"<id>",
         "auth_token":"<auth_token>",
         "method":"<method_name>",
         "args":{"<arg_key>":"<arg_val>"}
     }
 
 Example call for auto pulling from remote repositories using curl:
 
 .. code-block:: bash
 
     curl https://server.com/_admin/api -X POST -H 'content-type:text/plain' --data-binary '{"id":1,
-    "auth_token":"xe7cdb2v278e4evbdf5vs04v832v0efvcbcve4a3","method":"pull", "args":{"repo":"CPython"}}'
+    "auth_token":"xe7cdb2v278e4evbdf5vs04v832v0efvcbcve4a3","method":"pull", "args":{"repoid":"CPython"}}'
 
 Provide those parameters:
  - **id** A value of any type, which is used to match the response with the
    request that it is replying to.
  - **auth_token** for access and permission validation.
  - **method** is name of method to call
  - **args** is an ``key:value`` list of arguments to pass to method
 
 .. note::
 
     To get your |authtoken|, from the |RCE| interface,
     go to:
     :menuselection:`username --> My account --> Auth tokens`
 
     For security reasons you should always create a dedicated |authtoken| for
     API use only.
 
 
 The |RCE| API will always return a JSON-RPC response:
 
 .. code-block:: bash
 
     {
         "id": <id>, # matching id sent by request
         "result": "<result>"|null, # JSON formatted result, null if any errors
         "error": "null"|<error_message> # JSON formatted error (if any)
     }
 
 All responses from API will be with `HTTP/1.0 200 OK` status code.
 If there is an error when calling the API, the *error* key will contain a
 failure description and the *result* will be `null`.
 
 API CLIENT
 ----------
 
 To install the |RCE| API, see :ref:`install-tools`. To configure the API per
 instance, see the :ref:`rc-tools` section as you need to configure a
 :file:`~/.rhoderc` file with your |authtokens|.
 
 Once you have set up your instance API access, use the following examples to
 get started.
 
 .. code-block:: bash
 
     # Getting the 'rhodecode' repository
     # from a RhodeCode Enterprise instance
     rhodecode-api --instance-name=enterprise-1 get_repo repoid:rhodecode
 
     Calling method get_repo => http://127.0.0.1:5000
     Server response
     {
         <json data>
     }
 
     # Creating a new mercurial repository called 'brand-new'
     # with a description 'Repo-description'
     rhodecode-api --instance-name=enterprise-1 create_repo repo_name:brand-new repo_type:hg description:Repo-description
     {
       "error": null,
       "id": 1110,
       "result": {
         "msg": "Created new repository `brand-new`",
         "success": true,
         "task": null
       }
     }
 
 A broken example, what not to do.
 
 .. code-block:: bash
 
     # A call missing the required arguments
     # and not specifying the instance
     rhodecode-api get_repo
 
     Calling method get_repo => http://127.0.0.1:5000
     Server response
     "Missing non optional `repoid` arg in JSON DATA"
 
 You can specify pure JSON using the ``--format`` parameter.
 
 .. code-block:: bash
 
     rhodecode-api --format=json get_repo repoid:rhodecode
 
 In such case only output that this function shows is pure JSON, we can use that
 and pipe output to some json formatter.
 
 If output is in pure JSON format, you can pipe output to a JSON formatter.
 
 .. code-block:: bash
 
     rhodecode-api --instance-name=enterprise-1 --format=json get_repo repoid:rhodecode | python -m json.tool
 
 API METHODS
 -----------
 
 Each method by default required following arguments.
 
 .. code-block:: bash
 
     id :      "<id_for_response>"
     auth_token : "<auth_token>"
     method :  "<method name>"
     args :    {}
 
 Use each **param** from docs and put it in args, Optional parameters
 are not required in args.
 
 .. code-block:: bash
 
     args: {"repoid": "rhodecode"}
 
 .. Note: From this point on things are generated by the script in
    `scripts/fabfile.py`. To change things below, update the docstrings in the
    ApiController.
 
 .. --- API DEFS MARKER ---
 .. toctree::
 
    methods/license-methods
    methods/deprecated-methods
    methods/gist-methods
    methods/pull-request-methods
    methods/repo-methods
    methods/repo-group-methods
    methods/server-methods
    methods/user-methods
    methods/user-group-methods

docs/api/methods/pull-request-methods.rst

@@ -1,376 +1,371 b''
 .. _pull-request-methods-ref:
 
 pull_request methods
 ====================
 
 close_pull_request 
 ------------------
 
 .. py:function:: close_pull_request(apiuser, repoid, pullrequestid, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Close the pull request specified by `pullrequestid`.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Repository name or repository ID to which the pull
        request belongs.
    :type repoid: str or int
    :param pullrequestid: ID of the pull request to be closed.
    :type pullrequestid: int
    :param userid: Close the pull request as this user.
    :type userid: Optional(str or int)
 
    Example output:
 
    .. code-block:: bash
 
-     "id": <id_given_in_input>,
-     "result":
-       {
+       "id": <id_given_in_input>,
+       "result": {
            "pull_request_id":  "<int>",
            "closed":           "<bool>"
        },
-     "error": null
+       "error": null
 
 
 comment_pull_request 
 --------------------
 
 .. py:function:: comment_pull_request(apiuser, repoid, pullrequestid, message=<Optional:None>, commit_id=<Optional:None>, status=<Optional:None>, comment_type=<Optional:u'note'>, resolves_comment_id=<Optional:None>, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Comment on the pull request specified with the `pullrequestid`,
    in the |repo| specified by the `repoid`, and optionally change the
    review status.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
    :param pullrequestid: The pull request ID.
    :type pullrequestid: int
    :param commit_id: Specify the commit_id for which to set a comment. If
        given commit_id is different than latest in the PR status
        change won't be performed.
    :type commit_id: str
    :param message: The text content of the comment.
    :type message: str
    :param status: (**Optional**) Set the approval status of the pull
        request. One of: 'not_reviewed', 'approved', 'rejected',
        'under_review'
    :type status: str
    :param comment_type: Comment type, one of: 'note', 'todo'
    :type comment_type: Optional(str), default: 'note'
    :param userid: Comment on the pull request as this user
    :type userid: Optional(str or int)
 
    Example output:
 
    .. code-block:: bash
 
-     id : <id_given_in_input>
-     result :
-       {
+       id : <id_given_in_input>
+       result : {
            "pull_request_id":  "<Integer>",
            "comment_id":       "<Integer>",
            "status": {"given": <given_status>,
                       "was_changed": <bool status_was_actually_changed> },
-       }
-     error :  null
+       },
+       error :  null
 
 
 create_pull_request 
 -------------------
 
 .. py:function:: create_pull_request(apiuser, source_repo, target_repo, source_ref, target_ref, title, description=<Optional:''>, reviewers=<Optional:None>)
 
    Creates a new pull request.
 
    Accepts refs in the following formats:
 
        * branch:<branch_name>:<sha>
        * branch:<branch_name>
        * bookmark:<bookmark_name>:<sha> (Mercurial only)
        * bookmark:<bookmark_name> (Mercurial only)
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param source_repo: Set the source repository name.
    :type source_repo: str
    :param target_repo: Set the target repository name.
    :type target_repo: str
    :param source_ref: Set the source ref name.
    :type source_ref: str
    :param target_ref: Set the target ref name.
    :type target_ref: str
    :param title: Set the pull request title.
    :type title: str
    :param description: Set the pull request description.
    :type description: Optional(str)
    :param reviewers: Set the new pull request reviewers list.
    :type reviewers: Optional(list)
        Accepts username strings or objects of the format:
-       {
-           'username': 'nick', 'reasons': ['original author']
-       }
+
+           {'username': 'nick', 'reasons': ['original author']}
 
 
 get_pull_request 
 ----------------
 
 .. py:function:: get_pull_request(apiuser, repoid, pullrequestid)
 
    Get a pull request based on the given ID.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Repository name or repository ID from where the pull
        request was opened.
    :type repoid: str or int
    :param pullrequestid: ID of the requested pull request.
    :type pullrequestid: int
 
    Example output:
 
    .. code-block:: bash
 
      "id": <id_given_in_input>,
      "result":
        {
            "pull_request_id":   "<pull_request_id>",
            "url":               "<url>",
            "title":             "<title>",
            "description":       "<description>",
            "status" :           "<status>",
            "created_on":        "<date_time_created>",
            "updated_on":        "<date_time_updated>",
            "commit_ids":        [
                                     ...
                                     "<commit_id>",
                                     "<commit_id>",
                                     ...
                                 ],
            "review_status":    "<review_status>",
            "mergeable":         {
                                     "status":  "<bool>",
                                     "message": "<message>",
                                 },
            "source":            {
                                     "clone_url":     "<clone_url>",
                                     "repository":    "<repository_name>",
                                     "reference":
                                     {
                                         "name":      "<name>",
                                         "type":      "<type>",
                                         "commit_id": "<commit_id>",
                                     }
                                 },
            "target":            {
                                     "clone_url":   "<clone_url>",
                                     "repository":    "<repository_name>",
                                     "reference":
                                     {
                                         "name":      "<name>",
                                         "type":      "<type>",
                                         "commit_id": "<commit_id>",
                                     }
                                 },
            "merge":             {
                                     "clone_url":   "<clone_url>",
                                     "reference":
                                     {
                                         "name":      "<name>",
                                         "type":      "<type>",
                                         "commit_id": "<commit_id>",
                                     }
                                 },
           "author":             <user_obj>,
           "reviewers":          [
                                     ...
                                     {
                                        "user":          "<user_obj>",
                                        "review_status": "<review_status>",
                                     }
                                     ...
                                 ]
        },
       "error": null
 
 
 get_pull_requests 
 -----------------
 
 .. py:function:: get_pull_requests(apiuser, repoid, status=<Optional:'new'>)
 
    Get all pull requests from the repository specified in `repoid`.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Repository name or repository ID.
    :type repoid: str or int
    :param status: Only return pull requests with the specified status.
        Valid options are.
        * ``new`` (default)
        * ``open``
        * ``closed``
    :type status: str
 
    Example output:
 
    .. code-block:: bash
 
      "id": <id_given_in_input>,
      "result":
        [
            ...
            {
                "pull_request_id":   "<pull_request_id>",
                "url":               "<url>",
                "title" :            "<title>",
                "description":       "<description>",
                "status":            "<status>",
                "created_on":        "<date_time_created>",
                "updated_on":        "<date_time_updated>",
                "commit_ids":        [
                                         ...
                                         "<commit_id>",
                                         "<commit_id>",
                                         ...
                                     ],
                "review_status":    "<review_status>",
                "mergeable":         {
                                        "status":      "<bool>",
                                        "message:      "<message>",
                                     },
                "source":            {
                                         "clone_url":     "<clone_url>",
                                         "reference":
                                         {
                                             "name":      "<name>",
                                             "type":      "<type>",
                                             "commit_id": "<commit_id>",
                                         }
                                     },
                "target":            {
                                         "clone_url":   "<clone_url>",
                                         "reference":
                                         {
                                             "name":      "<name>",
                                             "type":      "<type>",
                                             "commit_id": "<commit_id>",
                                         }
                                     },
                "merge":             {
                                         "clone_url":   "<clone_url>",
                                         "reference":
                                         {
                                             "name":      "<name>",
                                             "type":      "<type>",
                                             "commit_id": "<commit_id>",
                                         }
                                     },
               "author":             <user_obj>,
               "reviewers":          [
                                         ...
                                         {
                                            "user":          "<user_obj>",
                                            "review_status": "<review_status>",
                                         }
                                         ...
                                     ]
            }
            ...
        ],
      "error": null
 
 
 merge_pull_request 
 ------------------
 
 .. py:function:: merge_pull_request(apiuser, repoid, pullrequestid, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Merge the pull request specified by `pullrequestid` into its target
    repository.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The Repository name or repository ID of the
        target repository to which the |pr| is to be merged.
    :type repoid: str or int
    :param pullrequestid: ID of the pull request which shall be merged.
    :type pullrequestid: int
    :param userid: Merge the pull request as this user.
    :type userid: Optional(str or int)
 
    Example output:
 
    .. code-block:: bash
 
-     "id": <id_given_in_input>,
-     "result":
-       {
+       "id": <id_given_in_input>,
+       "result": {
            "executed":         "<bool>",
            "failure_reason":   "<int>",
            "merge_commit_id":  "<merge_commit_id>",
            "possible":         "<bool>",
            "merge_ref":        {
                                    "commit_id": "<commit_id>",
                                    "type":      "<type>",
                                    "name":      "<name>"
                                }
        },
-     "error": null
+       "error": null
 
 
 update_pull_request 
 -------------------
 
 .. py:function:: update_pull_request(apiuser, repoid, pullrequestid, title=<Optional:''>, description=<Optional:''>, reviewers=<Optional:None>, update_commits=<Optional:None>, close_pull_request=<Optional:None>)
 
    Updates a pull request.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
    :param pullrequestid: The pull request ID.
    :type pullrequestid: int
    :param title: Set the pull request title.
    :type title: str
    :param description: Update pull request description.
    :type description: Optional(str)
    :param reviewers: Update pull request reviewers list with new value.
    :type reviewers: Optional(list)
    :param update_commits: Trigger update of commits for this pull request
    :type: update_commits: Optional(bool)
    :param close_pull_request: Close this pull request with rejected state
    :type: close_pull_request: Optional(bool)
 
    Example output:
 
    .. code-block:: bash
 
-     id : <id_given_in_input>
-     result :
-       {
+       id : <id_given_in_input>
+       result : {
            "msg": "Updated pull request `63`",
            "pull_request": <pull_request_object>,
            "updated_reviewers": {
              "added": [
                "username"
              ],
              "removed": []
            },
            "updated_commits": {
              "added": [
                "<sha1_hash>"
              ],
              "common": [
                "<sha1_hash>",
                "<sha1_hash>",
              ],
              "removed": []
            }
        }
-     error :  null
+       error :  null
 
 

docs/api/methods/server-methods.rst

@@ -1,158 +1,210 b''
 .. _server-methods-ref:
 
 server methods
 ==============
 
 cleanup_sessions 
 ----------------
 
 .. py:function:: cleanup_sessions(apiuser, older_then=<Optional:60>)
 
    Triggers a session cleanup action.
 
    If the ``older_then`` option is set, only sessions that hasn't been
    accessed in the given number of days will be removed.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param older_then: Deletes session that hasn't been accessed
        in given number of days.
    :type older_then: Optional(int)
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result: {
        "backend": "<type of backend>",
        "sessions_removed": <number_of_removed_sessions>
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        'Error occurred during session cleanup'
      }
 
 
 get_ip 
 ------
 
 .. py:function:: get_ip(apiuser, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Displays the IP Address as seen from the |RCE| server.
 
    * This command displays the IP Address, as well as all the defined IP
      addresses for the specified user. If the ``userid`` is not set, the
      data returned is for the user calling the method.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from |authtoken|.
    :type apiuser: AuthUser
    :param userid: Sets the userid for which associated IP Address data
        is returned.
    :type userid: Optional(str or int)
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result : {
                     "server_ip_addr": "<ip_from_clien>",
                     "user_ips": [
                                    {
                                       "ip_addr": "<ip_with_mask>",
                                       "ip_range": ["<start_ip>", "<end_ip>"],
                                    },
                                    ...
                                 ]
        }
 
 
+get_method 
+----------
+
+.. py:function:: get_method(apiuser, pattern=<Optional:'*'>)
+
+   Returns list of all available API methods. By default match pattern
+   os "*" but any other pattern can be specified. eg *comment* will return
+   all methods with comment inside them. If just single method is matched
+   returned data will also include method specification
+
+   This command can only be run using an |authtoken| with admin rights to
+   the specified repository.
+
+   This command takes the following options:
+
+   :param apiuser: This is filled automatically from the |authtoken|.
+   :type apiuser: AuthUser
+   :param pattern: pattern to match method names against
+   :type older_then: Optional("*")
+
+   Example output:
+
+   .. code-block:: bash
+
+     id : <id_given_in_input>
+     "result": [
+       "changeset_comment",
+       "comment_pull_request",
+       "comment_commit"
+     ]
+     error :  null
+
+   .. code-block:: bash
+
+     id : <id_given_in_input>
+     "result": [
+       "comment_commit",
+       {
+         "apiuser": "<RequiredType>",
+         "comment_type": "<Optional:u'note'>",
+         "commit_id": "<RequiredType>",
+         "message": "<RequiredType>",
+         "repoid": "<RequiredType>",
+         "request": "<RequiredType>",
+         "resolves_comment_id": "<Optional:None>",
+         "status": "<Optional:None>",
+         "userid": "<Optional:<OptionalAttr:apiuser>>"
+       }
+     ]
+     error :  null
+
+
 get_server_info 
 ---------------
 
 .. py:function:: get_server_info(apiuser)
 
    Returns the |RCE| server information.
 
    This includes the running version of |RCE| and all installed
    packages. This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        'modules': [<module name>,...]
        'py_version': <python version>,
        'platform': <platform type>,
        'rhodecode_version': <rhodecode version>
      }
      error :  null
 
 
 rescan_repos 
 ------------
 
 .. py:function:: rescan_repos(apiuser, remove_obsolete=<Optional:False>)
 
    Triggers a rescan of the specified repositories.
 
    * If the ``remove_obsolete`` option is set, it also deletes repositories
      that are found in the database but not on the file system, so called
      "clean zombies".
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param remove_obsolete: Deletes repositories from the database that
        are not found on the filesystem.
    :type remove_obsolete: Optional(``True`` | ``False``)
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        'added': [<added repository name>,...]
        'removed': [<removed repository name>,...]
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        'Error occurred during rescan repositories action'
      }
 
 

docs/api/methods/user-methods.rst

@@ -1,297 +1,325 b''
 .. _user-methods-ref:
 
 user methods
 ============
 
 create_user 
 -----------
 
 .. py:function:: create_user(apiuser, username, email, password=<Optional:''>, firstname=<Optional:''>, lastname=<Optional:''>, active=<Optional:True>, admin=<Optional:False>, extern_name=<Optional:'rhodecode'>, extern_type=<Optional:'rhodecode'>, force_password_change=<Optional:False>, create_personal_repo_group=<Optional:None>)
 
    Creates a new user and returns the new user object.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param username: Set the new username.
    :type username: str or int
    :param email: Set the user email address.
    :type email: str
    :param password: Set the new user password.
    :type password: Optional(str)
    :param firstname: Set the new user firstname.
    :type firstname: Optional(str)
    :param lastname: Set the new user surname.
    :type lastname: Optional(str)
    :param active: Set the user as active.
    :type active: Optional(``True`` | ``False``)
    :param admin: Give the new user admin rights.
    :type admin: Optional(``True`` | ``False``)
    :param extern_name: Set the authentication plugin name.
        Using LDAP this is filled with LDAP UID.
    :type extern_name: Optional(str)
    :param extern_type: Set the new user authentication plugin.
    :type extern_type: Optional(str)
    :param force_password_change: Force the new user to change password
        on next login.
    :type force_password_change: Optional(``True`` | ``False``)
    :param create_personal_repo_group: Create personal repo group for this user
    :type create_personal_repo_group: Optional(``True`` | ``False``)
+
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
-                 "msg" : "created new user `<username>`",
-                 "user": <user_obj>
-               }
+           "msg" : "created new user `<username>`",
+           "user": <user_obj>
+       }
        error:  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "user `<username>` already exist"
        or
        "email `<email>` already exist"
        or
        "failed to create user `<username>`"
      }
 
 
 delete_user 
 -----------
 
 .. py:function:: delete_user(apiuser, userid)
 
    Deletes the specified user from the |RCE| user database.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    .. important::
 
       Ensure all open pull requests and open code review
       requests to this user are close.
 
       Also ensure all repositories, or repository groups owned by this
       user are reassigned before deletion.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param userid: Set the user to delete.
    :type userid: str or int
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
-                 "msg" : "deleted user ID:<userid> <username>",
-                 "user": null
-               }
+           "msg" : "deleted user ID:<userid> <username>",
+           "user": null
+       }
        error:  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "failed to delete user ID:<userid> <username>"
      }
 
 
 get_user 
 --------
 
 .. py:function:: get_user(apiuser, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Returns the information associated with a username or userid.
 
    * If the ``userid`` is not set, this command returns the information
      for the ``userid`` calling the method.
 
    .. note::
 
       Normal users may only run this command against their ``userid``. For
       full privileges you must run this command using an |authtoken| with
       admin rights.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param userid: Sets the userid for which data will be returned.
    :type userid: Optional(str or int)
 
    Example output:
 
    .. code-block:: bash
 
        {
          "error": null,
          "id": <id>,
          "result": {
            "active": true,
            "admin": false,
-           "api_key": "api-key",
            "api_keys": [ list of keys ],
+           "auth_tokens": [ list of tokens with details ],
            "email": "[e-mail]",
            "emails": [
              "[e-mail]"
            ],
            "extern_name": "rhodecode",
            "extern_type": "rhodecode",
            "firstname": "username",
            "ip_addresses": [],
            "language": null,
            "last_login": "Timestamp",
+           "last_activity": "Timestamp",
            "lastname": "surnae",
            "permissions": {
              "global": [
                "hg.inherit_default_perms.true",
                "usergroup.read",
                "hg.repogroup.create.false",
                "hg.create.none",
                "hg.password_reset.enabled",
                "hg.extern_activate.manual",
                "hg.create.write_on_repogroup.false",
                "hg.usergroup.create.false",
                "group.none",
                "repository.none",
                "hg.register.none",
                "hg.fork.repository"
              ],
              "repositories": { "username/example": "repository.write"},
              "repositories_groups": { "user-group/repo": "group.none" },
              "user_groups": { "user_group_name": "usergroup.read" }
            },
            "user_id": 32,
            "username": "username"
          }
        }
 
 
+get_user_audit_logs 
+-------------------
+
+.. py:function:: get_user_audit_logs(apiuser, userid=<Optional:<OptionalAttr:apiuser>>)
+
+   Fetches all action logs made by the specified user.
+
+   This command takes the following options:
+
+   :param apiuser: This is filled automatically from the |authtoken|.
+   :type apiuser: AuthUser
+   :param userid: Sets the userid whose list of locked |repos| will be
+       displayed.
+   :type userid: Optional(str or int)
+
+   Example output:
+
+   .. code-block:: bash
+
+       id : <id_given_in_input>
+       result : {
+           [action, action,...]
+       }
+       error :  null
+
+
 get_user_locks 
 --------------
 
 .. py:function:: get_user_locks(apiuser, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Displays all repositories locked by the specified user.
 
    * If this command is run by a non-admin user, it returns
      a list of |repos| locked by that user.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param userid: Sets the userid whose list of locked |repos| will be
        displayed.
    :type userid: Optional(str or int)
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result : {
            [repo_object, repo_object,...]
        }
        error :  null
 
 
 get_users 
 ---------
 
 .. py:function:: get_users(apiuser)
 
    Lists all users in the |RCE| user database.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
-           result: [<user_object>, ...]
+       result: [<user_object>, ...]
        error:  null
 
 
 update_user 
 -----------
 
 .. py:function:: update_user(apiuser, userid, username=<Optional:None>, email=<Optional:None>, password=<Optional:None>, firstname=<Optional:None>, lastname=<Optional:None>, active=<Optional:None>, admin=<Optional:None>, extern_type=<Optional:None>, extern_name=<Optional:None>)
 
    Updates the details for the specified user, if that user exists.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from |authtoken|.
    :type apiuser: AuthUser
    :param userid: Set the ``userid`` to update.
    :type userid: str or int
    :param username: Set the new username.
    :type username: str or int
    :param email: Set the new email.
    :type email: str
    :param password: Set the new password.
    :type password: Optional(str)
    :param firstname: Set the new first name.
    :type firstname: Optional(str)
    :param lastname: Set the new surname.
    :type lastname: Optional(str)
    :param active: Set the new user as active.
    :type active: Optional(``True`` | ``False``)
    :param admin: Give the user admin rights.
    :type admin: Optional(``True`` | ``False``)
    :param extern_name: Set the authentication plugin user name.
        Using LDAP this is filled with LDAP UID.
    :type extern_name: Optional(str)
    :param extern_type: Set the authentication plugin type.
    :type extern_type: Optional(str)
 
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
-                 "msg" : "updated user ID:<userid> <username>",
-                 "user": <user_object>,
-               }
+           "msg" : "updated user ID:<userid> <username>",
+           "user": <user_object>,
+       }
        error:  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "failed to update user `<username>`"
      }
 
 

docs/auth/ldap-active-directory.rst

@@ -1,22 +1,29 b''
 .. _ldap-act-dir-ref:
 
 Active Directory
 ----------------
 
 |RCM| can use Microsoft Active Directory for user authentication. This is
 done through an LDAP or LDAPS connection to Active Directory. Use the
 following example LDAP configuration setting to set your Active Directory
 authentication.
 
 .. code-block:: ini
     
     # Set the Base DN
     Base DN              = OU=SBSUsers,OU=Users,OU=MyBusiness,DC=v3sys,DC=local
     # Set the Active Directory SAM-Account-Name
     Login Attribute      = sAMAccountName
     # Set the Active Directory user name
     First Name Attribute = usernameame
     # Set the Active Directory user surname
     Last Name Attribute  = user_surname
     # Set the Active Directory user email
-    E-mail Attribute     = userEmail
  No newline at end of file
+    E-mail Attribute     = userEmail
+
+
+Below is example setup that can be used with Active Directory and ldap groups.
+
+.. image:: ../images/ldap-groups-example.png
+   :alt: LDAP/AD setup example
+   :scale: 50 %
  No newline at end of file

docs/auth/ldap-authentication.rst

@@ -1,107 +1,113 b''
 .. _ldap-gloss-ref:
 
 |LDAP| Glossary
 ---------------
 
 This topic aims to give you a concise overview of the different settings and
 requirements that enabling |LDAP| on |RCE| requires.
 
 Required settings
 ^^^^^^^^^^^^^^^^^
 
 The following LDAP attributes are required when enabling |LDAP| on |RCE|.
 
 * **Hostname** or **IP Address**: Use a comma separated list for failover
   support.
 * **First Name**
 * **Surname**
 * **Email**
 * **Port**: Port `389` for unencrypted LDAP or port `636` for SSL-encrypted
   LDAP (LDAPS).
 * **Base DN (Distinguished Name)**: The Distinguished Name (DN)
   is how searches for users will be performed, and these searches can be
   controlled by using an LDAP Filter or LDAP Search Scope. A DN is a sequence of
   relative distinguished names (RDN) connected by commas. For example,
       
 .. code-block:: vim
     
     DN: cn='Monty Python',ou='people',dc='example',dc='com'
             
 * **Connection security level**: The following are the valid types:
    
   * *No encryption*: This connection type uses a plain non-encrypted connection.
   * *LDAPS connection*: This connection type uses end-to-end SSL. To enable
     an LDAPS connection you must set the following requirements:
 
       * You must specify port `636`
       * Certificate checks are required.
       * To enable ``START_TLS`` on LDAP connection, set the path to the SSL
         certificate in the default LDAP configuration file. The default
         `ldap.conf` file is located in `/etc/openldap/ldap.conf`.
 
 .. code-block:: vim
     
    TLS_CACERT	/etc/ssl/certs/ca.crt
 
 * The LDAP username or account used to connect to |RCE|. This will be added
   to the LDAP filter for locating the user object.
 * For example, if an LDAP filter is specified as `LDAPFILTER`,
   the login attribute is specified as `uid`, and the user connects as
   `jsmith`, then the LDAP Filter will be like the following example.
       
 .. code-block:: vim
                       
    (&(LDAPFILTER)(uid=jsmith))
     
 * The LDAP search scope must be set. This limits how far LDAP will search for
   a matching object.
 
   * ``BASE`` Only allows searching of the Base DN.
   * ``ONELEVEL`` Searches all entries under the Base DN,
     but not the Base DN itself.
   * ``SUBTREE`` Searches all entries below the Base DN, but not Base DN itself.
       
 .. note::
             
    When using ``SUBTREE`` LDAP filtering it is useful to limit object location.
  
 Optional settings
 ^^^^^^^^^^^^^^^^^
 
 The following are optional when enabling LDAP on |RCM|
  
 * An LDAP account is only required if the LDAP server does not allow
   anonymous browsing of records.
 * An LDAP password is only required if the LDAP server does not allow
   anonymous browsing of records
 * Using an LDAP filter is optional. An LDAP filter defined by `RFC 2254`_. This
   is more useful that the LDAP Search Scope if set to `SUBTREE`. The filter
   is useful for limiting which LDAP objects are identified as representing
   Users for authentication. The filter is augmented by Login Attribute
   below. This can commonly be left blank.
 * Certificate Checks are only required if you need to use LDAPS.
   You can use the following levels of LDAP service with RhodeCode Enterprise:
 
    * **NEVER** : A serve certificate will never be requested or checked.
    * **ALLOW** : A server certificate is requested. Failure to provide a
      certificate or providing a bad certificate will not terminate the session.
    * **TRY** : A server certificate is requested. Failure to provide a
      certificate does not halt the session; providing a bad certificate
      halts the session.
    * **DEMAND** : A server certificate is requested and must be provided
      and authenticated for the session to proceed.
    * **HARD** : The same as DEMAND.
 
 .. note::
 
    Only **DEMAND** or **HARD** offer full SSL security while the other
    options are vulnerable to man-in-the-middle attacks.
 
    |RCE| uses ``OPENLDAP`` libraries. This allows **DEMAND** or
    **HARD** LDAPS connections to use self-signed certificates or
    certificates that do not have traceable certificates of authority.
    To enable this functionality install the SSL certificates in the
    following directory: `/etc/openldap/cacerts`
 
 
+Below is example setup that can be used with Active Directory and ldap groups.
+
+.. image:: ../images/ldap-groups-example.png
+   :alt: LDAP/AD setup example
+   :scale: 50 %
+
 .. _RFC 2254: http://www.rfc-base.org/rfc-2254.html
  No newline at end of file

docs/contributing/dev-setup.rst

@@ -1,195 +1,219 b''
 .. _dev-setup:
 
 ===================
  Development setup
 ===================
 
 
 RhodeCode Enterprise runs inside a Nix managed environment. This ensures build
 environment dependencies are correctly declared and installed during setup.
 It also enables atomic upgrades, rollbacks, and multiple instances of RhodeCode
 Enterprise running with isolation.
 
 To set up RhodeCode Enterprise inside the Nix environment, use the following steps:
 
 
 
 Setup Nix Package Manager
 -------------------------
 
 To install the Nix Package Manager, please run::
 
    $ curl https://nixos.org/nix/install | sh
 
 or go to https://nixos.org/nix/ and follow the installation instructions.
 Once this is correctly set up on your system, you should be able to use the
 following commands:
 
 * `nix-env`
 
 * `nix-shell`
 
 
 .. tip::
 
    Update your channels frequently by running ``nix-channel --update``.
 
 
 Switch nix to the latest STABLE channel
 ---------------------------------------
 
 run::
 
    nix-channel --add https://nixos.org/channels/nixos-16.03 nixpkgs
 
 Followed by::
 
    nix-channel --update
 
 
 Install required binaries
 -------------------------
 
 We need some handy tools first.
 
 run::
 
     nix-env -i nix-prefetch-hg
     nix-env -i nix-prefetch-git
 
 
 Clone the required repositories
 -------------------------------
 
 After Nix is set up, clone the RhodeCode Enterprise Community Edition and
 RhodeCode VCSServer repositories into the same directory.
 RhodeCode currently is using Mercurial Version Control System, please make sure
 you have it installed before continuing.
 
 To obtain the required sources, use the following commands:
 
     mkdir rhodecode-develop && cd rhodecode-develop
     hg clone https://code.rhodecode.com/rhodecode-enterprise-ce
     hg clone https://code.rhodecode.com/rhodecode-vcsserver
 
 .. note::
 
    If you cannot clone the repository, please contact us via [e-mail]
 
 
 Install some required libraries
 -------------------------------
 
 There are some required drivers that we need to install to test RhodeCode
 under different types of databases. For example in Ubuntu we need to install
 the following.
 
 required libraries::
 
     sudo apt-get install libapr1-dev libaprutil1-dev
     sudo apt-get install libsvn-dev
     sudo apt-get install mysql-server libmysqlclient-dev
     sudo apt-get install postgresql postgresql-contrib libpq-dev
     sudo apt-get install libcurl4-openssl-dev
 
 
 Enter the Development Shell
 ---------------------------
 
 The final step is to start the development shells. To do this, run the
 following command from inside the cloned repository::
 
    #first, the vcsserver
    cd ~/rhodecode-vcsserver
    nix-shell
 
    # then enterprise sources
    cd ~/rhodecode-enterprise-ce
    nix-shell
 
 .. note::
 
    On the first run, this will take a while to download and optionally compile
    a few things. The following runs will be faster. The development shell works
    fine on both MacOS and Linux platforms.
 
 
+Create config.nix for development
+---------------------------------
+
+In order to run proper tests and setup linking across projects, a config.nix
+file needs to be setup::
+
+    # create config
+    mkdir -p ~/.nixpkgs
+    touch ~/.nixpkgs/config.nix
+
+    # put the below content into the ~/.nixpkgs/config.nix file
+    # adjusts, the path to where you cloned your repositories.
+
+    {
+      rc = {
+       sources = {
+        rhodecode-vcsserver = "/home/dev/rhodecode-vcsserver";
+        rhodecode-enterprise-ce = "/home/dev/rhodecode-enterprise-ce";
+        rhodecode-enterprise-ee = "/home/dev/rhodecode-enterprise-ee";
+       };
+      };
+    }
+
+
 
 Creating a Development Configuration
 ------------------------------------
 
 To create a development environment for RhodeCode Enterprise,
 use the following steps:
 
 1. Create a copy of vcsserver config:
     `cp ~/rhodecode-vcsserver/configs/development.ini ~/rhodecode-vcsserver/configs/dev.ini`
 2. Create a copy of rhodocode config:
     `cp ~/rhodecode-enterprise-ce/configs/development.ini ~/rhodecode-enterprise-ce/configs/dev.ini`
 3. Adjust the configuration settings to your needs if needed.
 
 .. note::
 
   It is recommended to use the name `dev.ini` since it's included in .hgignore file.
 
 
 Setup the Development Database
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To create a development database, use the following example. This is a one
 time operation executed from the nix-shell of rhodecode-enterprise-ce sources ::
 
     paster setup-rhodecode dev.ini \
         --user=admin --password=secret \
         --email=[e-mail] \
         --repos=~/my_dev_repos
 
 
 Compile CSS and JavaScript
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To use the application's frontend and prepare it for production deployment,
 you will need to compile the CSS and JavaScript with Grunt.
 This is easily done from within the nix-shell using the following command::
 
     grunt
 
 When developing new features you will need to recompile following any
 changes made to the CSS or JavaScript files when developing the code::
 
     grunt watch
 
 This prepares the development (with comments/whitespace) versions of files.
 
 Start the Development Servers
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 From the rhodecode-vcsserver directory, start the development server in another
 nix-shell, using the following command::
 
       pserve configs/dev.ini
 
 In the adjacent nix-shell which you created for your development server, you may
 now start CE with the following command::
 
 
       pserve --reload configs/dev.ini
 
 .. note::
 
   `--reload` flag will automatically reload the server when source file changes.
 
 
 Run the Environment Tests
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Please make sure that the tests are passing to verify that your environment is
 set up correctly. RhodeCode uses py.test to run tests.
 While your instance is running, start a new nix-shell and simply run
 ``make test`` to run the basic test suite.
 
 
 Need Help?
 ^^^^^^^^^^
 
 Join us on Slack via https://rhodecode.com/join or post questions in our
 Community Portal at https://community.rhodecode.com

docs/release-notes/release-notes-4.5.0.rst

@@ -1,180 +1,180 b''
 |RCE| 4.5.0 |RNS|
 -----------------
 
 Release Date
 ^^^^^^^^^^^^
 
 - 2016-12-02
 
 
 New Features
 ^^^^^^^^^^^^
 
 - Diffs: re-implemented diff engine. Added: Syntax highlighting inside diffs,
   new side-by-side view with commenting and live chat. Enabled soft-wrapping of
   long lines and much improved rendering speed for large diffs.
 - File source view: new file display engine. File view now
   soft wraps long lines. Double click inside file view show occurrences of
   clicked item. Added pygments-markdown-lexer for highlighting markdown syntax.
 - Files annotation: Added new grouped annotations. Color all related commits
   by double clicking singe commit in annotation view.
 - Pull request reviewers (EE only): added new default reviewers functionality.
   Allows picking users or user groups defined as reviewers for new pull request.
   Picking reviewers can be based on branch name, changed file name patterns or
-  original author of changed source code. eg *.css -> design team.
+  original author of changed source code. eg \*.css -> design team.
   Master branch -> repo owner, fixes #1131.
 - Pull request reviewers: store and show reasons why given person is a reviewer.
   Manually adding reviewers after creating a PR will now be also indicated
   together with who added a given person to review.
 - Integrations: Webhooks integration now allows to use variables inside the
   call URL. Currently supported variables are ${repo_name}, ${repo_type},
   ${repo_id}, ${repo_url}, ${branch}, ${commit_id}, ${pull_request_id},
   ${pull_request_url}. Commits are now grouped by branches as well.
   Allows much easier integration with CI systems.
 - Integrations (EE only): allow wildcard * project key in Jira integration
   settings to allow referencing multiple projects per commit, fixes #4267.
 - Live notifications: RhodeCode sends live notification to online
   users on certain events and pages. Currently this works on: invite to chat,
   update pull request, commit/inline comment. Part of live code review system.
   Allows users to update the reviewed code while doing the review and never
   miss any updates or comment replies as they happen. Requires channelstream
   to be enabled.
 - Repository groups: added default personal repository groups. Personal groups
   are isolated playground for users allowing them to create projects or forks.
   Adds new setting to automatically create personal repo groups for newly
   created users. New groups are created from specified pattern, for example
   /u/{username}. Implements #4003.
 - Security: It's now possible to disable password reset functionality.
   This is useful for cases when users only use LDAP or similar types of
   authentication. Implements #3944
 - Pull requests: exposed shadow repositories to end users. Users are now given
   access to the shadow repository which represents state after merge performed.
   In this way users or especially CI servers can much easier perform code
   analysis of the final merged code.
 - Pull requests: My account > pull request page now uses datagrid.
   It's faster, filterable and sortable. Fixes #4297.
 - Pull requests: delete pull request action was moved from my account
   into pull request view itself. This is where everyone was looking for it.
 - Pull requests: improve showing failed merges with proper status in pull
   request page.
 - User groups: overhaul of edit user group page. Added new selector for
   adding new user group members.
 - Licensing (EE only): exposed unlock link to deactivate users that are over
   license limit, to unlock full functionality. This might happen when migrating
   from CE into EE, and your license supports less active users then allowed.
 - Global settings: add a new header/footer template to allow flash filtering.
   In case a license warning appears and admin wants to hide it for some time.
   The new template can be used to do this.
 - System info: added create snapshot button to easily generate system state
   report. Comes in handy for support and reporting. System state holds
   information such as free disk/memory, CPU load and some of RhodeCode settings.
 - System info: fetch and show vcs settings from vcsserver. Fixes #4276.
 - System info: use real memory usage based on new psutil api available.
 - System info: added info about temporary storage.
 - System info: expose inode limits and usage. Fixes #4282.
 - Ui: added new icon for merge commit.
 
 
 
 General
 ^^^^^^^
 
 - Notifications: move all notifications into polymer for consistency.
   Fixes #4201.
 - Live chat (EE): Improved UI for live-chat. Use Codemirror editor as
   input for text box.
 - Api: WARNING DEPRECATION, refactor repository group schemas. Fixes #4133.
   When using create_repo, create_repo_group, update_repo, update_repo_group
-  the *_name parameter now takes full path including sub repository groups.
+  the \*_name parameter now takes full path including sub repository groups.
   This is the only way to add resource under another repository group.
   Furthermore giving non-existing path will no longer create the missing
   structure. This change makes the api more consistent, it better validates
   the errors in the data sent to given api call.
 - Pull requests: disable subrepo handling on pull requests. It means users can
   now use more types of repositories with subrepos to create pull requests.
   Since handling is disabled, repositories behind authentication, or outside
   of network can be used.
 - VCSServer: fetch backend info from vcsserver including git/hg/svn versions
   and connection information.
 - Svn support: it's no longer required to put in repo root path to
   generate mod-dav-svn config. Fixes #4203.
 - Svn support: Add reload command option (svn.proxy.reload_cmd) to ini files.
   Apache can now be automatically reloaded when the mod_dav_svn config changes.
 - Svn support: Add a view to trigger the (re)generation of Apache mod_dav_svn
   configuration file. Users are able to generate such file manually by clicking
   that button.
 - Dependency: updated subversion library to 1.9.
 - Dependency: updated ipython to 5.1.0.
 - Dependency: updated psutil to 4.3.1.
 
 
 Security
 ^^^^^^^^
 
 - Hipchat: escape user entered data to avoid xss/formatting problems.
 - VCSServer: obfuscate credentials added into remote url during remote
   repository creation. Prevents leaking of those credentials inside
   RhodeCode logs.
 
 
 Performance
 ^^^^^^^^^^^
 
 - Diffs: new diff engine is much smarter when it comes to showing huge diffs.
   The rendering speed should be much improved in such cases, however showing
   full diff is still supported.
 - VCS backends: when using a repo object from database, re-use this information
   instead of trying to detect a backend. Reduces the traffic to vcsserver.
 - Pull requests: Add a column to hold the last merge revision. This will skip
   heavy recalculation of merge state if nothing changed inside a pull request.
 - File source view: don't load the file if it is over the size limit since it
   won't be displayed anyway. This increases speed of loading the page when a
   file is above cut-off limit defined.
 
 
 Fixes
 ^^^^^
 
 - Users admin: fixed search filter in user admin page.
 - Autocomplete: improve the lookup of users with non-ascii characters. In case
   of unicode email the previous method could generate wrong data, and
   make search not show up such users.
 - Svn: added request header downgrade for COPY command to work on
   https setup. Fixes #4307.
 - Svn: add handling of renamed files inside our generated changes metadata.
   Fixes #4258.
 - Pull requests: fixed problem with creating pull requests on empty repositories.
 - Events: use branch from previous commit for repo push event commits data so
   that per-branch grouping works. Fixes #4233.
 - Login: make sure recaptcha data is always validated. Fixes #4279.
 - Vcs: Use commit date as modification time when creating archives.
   Fixes problem with unstable hashes for archives. Fixes #4247.
 - Issue trackers: fixed bug where saving empty issue tracker via form was
   causing exception. Fixes #4278.
 - Styling: fixed gravatar size for pull request reviewers.
 - Ldap: fixed email extraction typo. An empty email from LDAP server will now
   not overwrite the stored one.
 - Integrations: use consistent formatting of users data in Slack integration.
 - Meta-tags: meta tags are not taken into account when truncating descriptions
   that are too long. Fixes #4305.
 
 
 Upgrade notes
 ^^^^^^^^^^^^^
 
 - Api: please adjust your scripts that uses any of create_repo,
   create_repo_group, update_repo, update_repo_group. There's an important change
   in how the repo_name/group_name parameters work. Please check the API docs
   for latest information.
 
 - Installation: starting from 4.5.0 installer now changes the default mode to http.
   If you were using the `self_managed_supervisor=True` flag inside
   `.rccontrol.ini` to manually switch to that mode. This is no longer required
   and we recommend removing that flag. Migration should already change that
   however in case of any troubles with VCSServer after upgrade
   please make sure `vcs.protocol=` is set to `http` and not `pyro4` inside
   rhodecode.ini
 
 - New setting about password recovery was introduced. Please make sure to
   adjust what ever default you want to have inside your instance. The default
   is that password recovery is enabled.

docs/release-notes/release-notes.rst

@@ -1,94 +1,95 b''
 .. _rhodecode-release-notes-ref:
 
 Release Notes
 =============
 
 |RCE| 4.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
+   release-notes-4.7.0.rst
    release-notes-4.6.1.rst
    release-notes-4.6.0.rst
    release-notes-4.5.2.rst
    release-notes-4.5.1.rst
    release-notes-4.5.0.rst
    release-notes-4.4.2.rst
    release-notes-4.4.1.rst
    release-notes-4.4.0.rst
    release-notes-4.3.1.rst
    release-notes-4.3.0.rst
    release-notes-4.2.1.rst
    release-notes-4.2.0.rst
    release-notes-4.1.2.rst
    release-notes-4.1.1.rst
    release-notes-4.1.0.rst
    release-notes-4.0.1.rst
    release-notes-4.0.0.rst
 
 |RCE| 3.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-3.8.4.rst
    release-notes-3.8.3.rst
    release-notes-3.8.2.rst
    release-notes-3.8.1.rst
    release-notes-3.8.0.rst
    release-notes-3.7.1.rst
    release-notes-3.7.0.rst
    release-notes-3.6.1.rst
    release-notes-3.6.0.rst
    release-notes-3.5.2.rst
    release-notes-3.5.1.rst
    release-notes-3.5.0.rst
    release-notes-3.4.1.rst
    release-notes-3.4.0.rst
    release-notes-3.3.4.rst
    release-notes-3.3.3.rst
    release-notes-3.3.2.rst
    release-notes-3.3.1.rst
    release-notes-3.3.0.rst
    release-notes-3.2.3.rst
    release-notes-3.2.2.rst
    release-notes-3.2.1.rst
    release-notes-3.2.0.rst
    release-notes-3.1.1.rst
    release-notes-3.1.0.rst
    release-notes-3.0.2.rst
    release-notes-3.0.1.rst
    release-notes-3.0.0.rst
 
 |RCE| 2.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-2.2.8.rst
    release-notes-2.2.7.rst
    release-notes-2.2.6.rst
    release-notes-2.2.5.rst
    release-notes-2.2.4.rst
    release-notes-2.2.3.rst
    release-notes-2.2.2.rst
    release-notes-2.2.1.rst
    release-notes-2.2.0.rst
    release-notes-2.1.0.rst
    release-notes-2.0.2.rst
    release-notes-2.0.1.rst
    release-notes-2.0.0.rst
 
 |RCE| 1.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-1.7.2.rst
    release-notes-1.7.1.rst
    release-notes-1.7.0.rst
    release-notes-1.6.0.rst

docs/tutorials/hg-large-ext.rst

@@ -1,39 +1,69 b''
 .. _hg-big-files:
 
 |hg| Large Files Extension
 ==========================
 
 Large files, such as image or zip files can cause a lot of bandwidth overhead
 during clone, push, and pull operations. To remove this inefficiency, |hg|
 has a large files extension which tracks their revisions by checksums. This
 means that the large files are only downloaded when they are needed as part
 of the current revision. This saves both disk space and bandwidth.
 
+
+Enabling HG Largefiles
+++++++++++++++++++++++
+
+Mercurial Largefiles extension is disabled by default within |RC| Server.
+
+To enable Mercurial Largefiles Globally:
+
+    - Go to :menuselection:`Admin --> Settings --> VCS`
+
+    - Scroll down into `Mercurial settings`
+
+    - Tick `Enable largefiles extension`
+
+    - Save your settings.
+
+Those settings apply globally to each repository that inherits from the defaults
+You can leave `largefiles extension` disabled globally, and only enable it per
+repository that would use the largefiles.
+
+
+.. note::
+
+    You might want to adjust the global storage location at that point, however
+    we recommend leaving the default one created.
+
+
+Installing and using the |hg| Largefiles
+++++++++++++++++++++++++++++++++++++++++
+
 To find out more, see the |hg| `Large Files Extensions Documentation`_.
 
 To configure the large files extension, you need to set up your
 :file:`~/.hgrc` file.
 
 1. Open your :file:`~/.hgrc` file.
 2. Add ``largefiles =`` to the ``[extensions]`` section.
 3. Configure the ``[largefiles]`` section with the patterns and file size you
    wish |hg| to handle as large. The ``minsize`` option is specified in
    megabytes.
 4. Save your changes.
 
 .. code-block:: ini
 
     [extensions]
     hgext.churn =
     largefiles =
     rebase =
     record =
     histedit =
 
     [largefiles]
     patterns = re:.*\.(png|bmp|jpg|zip|tar|tar.gz|rar)$
     minsize = 10
 
 For a complete :file:`~/.hgrc` file example, see :ref:`config-hgrc`.
 
 .. _Large Files Extensions Documentation: http://mercurial.selenic.com/wiki/LargefilesExtension

docs/tutorials/tutorials.rst

@@ -1,26 +1,27 b''
 .. _rc-tutorials:
 
 |RCE| Tutorials
 ===============
 
 The following tutorials are aimed at helping you to deploy |RCE| in various
 different ways. If you would like to see a certain scenario documented,
 then please send a request to [e-mail]
 
 .. toctree::
    :maxdepth: 1
 
    deploy-from-host
    hg-large-ext
+   git-lfs-ext
    multi-instance-setup
    scaling-best-practices
    squash-commits
    squash-commits-git
    rebase-commits
    rebase-commits-git
    api-examples
    merging
    branching-vs-bookmarking
    windows-to-linux
    dvcs-best-practices
    appenlight-setup

pkgs/python-packages-overrides.nix

@@ -1,261 +1,276 b''
 # Overrides for the generated python-packages.nix
 #
 # This function is intended to be used as an extension to the generated file
 # python-packages.nix. The main objective is to add needed dependencies of C
 # libraries and tweak the build instructions where needed.
 
 { pkgs, basePythonPackages }:
 
 let
   sed = "sed -i";
   localLicenses = {
     repoze = {
       fullName = "Repoze License";
       url =  http://www.repoze.org/LICENSE.txt;
     };
   };
 
 in
 
 self: super: {
 
   appenlight-client = super.appenlight-client.override (attrs: {
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   });
 
   future = super.future.override (attrs: {
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   });
 
   gnureadline = super.gnureadline.override (attrs: {
     buildInputs = attrs.buildInputs ++ [
       pkgs.ncurses
     ];
     patchPhase = ''
       substituteInPlace setup.py --replace "/bin/bash" "${pkgs.bash}/bin/bash"
     '';
   });
 
   gunicorn = super.gunicorn.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       # johbo: futures is needed as long as we are on Python 2, otherwise
       # gunicorn explodes if used with multiple threads per worker.
       self.futures
     ];
   });
 
+  nbconvert = super.nbconvert.override (attrs: {
+    propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
+      # marcink: plug in jupyter-client for notebook rendering
+      self.jupyter-client
+    ];
+  });
+
   ipython = super.ipython.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       self.gnureadline
     ];
   });
 
+  celery = super.celery.override (attrs: {
+    # The current version of kombu needs some patching to work with the
+    # other libs. Should be removed once we update celery and kombu.
+    patches = [
+      ./patch-celery-dateutil.diff
+    ];
+  });
+
   kombu = super.kombu.override (attrs: {
     # The current version of kombu needs some patching to work with the
     # other libs. Should be removed once we update celery and kombu.
     patches = [
       ./patch-kombu-py-2-7-11.diff
       ./patch-kombu-msgpack.diff
     ];
   });
 
   lxml = super.lxml.override (attrs: {
     # johbo: On 16.09 we need this to compile on darwin, otherwise compilation
     # fails on Darwin.
     hardeningDisable = if pkgs.stdenv.isDarwin then [ "format" ] else null;
     buildInputs = with self; [
       pkgs.libxml2
       pkgs.libxslt
     ];
   });
 
   MySQL-python = super.MySQL-python.override (attrs: {
     buildInputs = attrs.buildInputs ++ [
       pkgs.openssl
     ];
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.mysql.lib
       pkgs.zlib
     ];
   });
 
   psutil = super.psutil.override (attrs: {
     buildInputs = attrs.buildInputs ++
       pkgs.lib.optional pkgs.stdenv.isDarwin pkgs.darwin.IOKit;
   });
 
   psycopg2 = super.psycopg2.override (attrs: {
     buildInputs = attrs.buildInputs ++
       pkgs.lib.optional pkgs.stdenv.isDarwin pkgs.openssl;
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.postgresql
     ];
     meta = {
       license = pkgs.lib.licenses.lgpl3Plus;
     };
   });
 
   py-gfm = super.py-gfm.override  {
     name = "py-gfm-0.1.3.rhodecode-upstream1";
   };
 
   pycurl = super.pycurl.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.curl
       pkgs.openssl
     ];
     preConfigure = ''
       substituteInPlace setup.py --replace '--static-libs' '--libs'
       export PYCURL_SSL_LIBRARY=openssl
     '';
     meta = {
       # TODO: It is LGPL and MIT
       license = pkgs.lib.licenses.mit;
     };
   });
 
   Pylons = super.Pylons.override (attrs: {
     name = "Pylons-1.0.2.rhodecode-patch1";
   });
 
   pyramid = super.pyramid.override (attrs: {
     postFixup = ''
       wrapPythonPrograms
       # TODO: johbo: "wrapPython" adds this magic line which
       # confuses pserve.
       ${sed} '/import sys; sys.argv/d' $out/bin/.pserve-wrapped
     '';
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   pyramid-debugtoolbar = super.pyramid-debugtoolbar.override (attrs: {
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal localLicenses.repoze ];
     };
   });
 
   pysqlite = super.pysqlite.override (attrs: {
     propagatedBuildInputs = [
       pkgs.sqlite
     ];
     meta = {
       license = [ pkgs.lib.licenses.zlib pkgs.lib.licenses.libpng ];
     };
   });
 
   pytest-runner = super.pytest-runner.override (attrs: {
     propagatedBuildInputs = [
       self.setuptools-scm
     ];
   });
 
   python-ldap = super.python-ldap.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.cyrus_sasl
       pkgs.openldap
       pkgs.openssl
     ];
     # TODO: johbo: Remove the "or" once we drop 16.03 support.
     NIX_CFLAGS_COMPILE = "-I${pkgs.cyrus_sasl.dev or pkgs.cyrus_sasl}/include/sasl";
   });
 
   python-pam = super.python-pam.override (attrs:
     let
       includeLibPam = pkgs.stdenv.isLinux;
     in {
       # TODO: johbo: Move the option up into the default.nix, we should
       # include python-pam only on supported platforms.
       propagatedBuildInputs = attrs.propagatedBuildInputs ++
         pkgs.lib.optional includeLibPam [
           pkgs.pam
         ];
       # TODO: johbo: Check if this can be avoided, or transform into
       # a real patch
       patchPhase = pkgs.lib.optionals includeLibPam ''
         substituteInPlace pam.py \
           --replace 'find_library("pam")' '"${pkgs.pam}/lib/libpam.so.0"'
       '';
     });
 
   URLObject = super.URLObject.override (attrs: {
     meta = {
       license = {
         spdxId = "Unlicense";
         fullName = "The Unlicense";
         url = http://unlicense.org/;
       };
     };
   });
 
   amqplib = super.amqplib.override (attrs: {
     meta = {
       license = pkgs.lib.licenses.lgpl3;
     };
   });
 
   docutils = super.docutils.override (attrs: {
     meta = {
       license = pkgs.lib.licenses.bsd2;
     };
   });
 
   colander = super.colander.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   pyramid-beaker  = super.pyramid-beaker.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   pyramid-mako = super.pyramid-mako.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   repoze.lru = super.repoze.lru.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   recaptcha-client = super.recaptcha-client.override (attrs: {
     meta = {
       # TODO: It is MIT/X11
       license = pkgs.lib.licenses.mit;
     };
   });
 
   python-editor = super.python-editor.override (attrs: {
     meta = {
       license = pkgs.lib.licenses.asl20;
     };
   });
 
   translationstring = super.translationstring.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   venusian = super.venusian.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   # Avoid that setuptools is replaced, this leads to trouble
   # with buildPythonPackage.
   setuptools = basePythonPackages.setuptools;
 
 }

pkgs/python-packages.nix

@@ -1,1852 +1,1995 b''
 # Generated by pip2nix 0.4.0
 # See https://github.com/johbo/pip2nix
 
 {
   Babel = super.buildPythonPackage {
     name = "Babel-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytz];
     src = fetchurl {
       url = "https://pypi.python.org/packages/33/27/e3978243a03a76398c384c83f7ca879bc6e8f1511233a621fcada135606e/Babel-1.3.tar.gz";
       md5 = "5264ceb02717843cbc9ffce8e6e06bdb";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   Beaker = super.buildPythonPackage {
     name = "Beaker-1.7.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/97/8e/409d2e7c009b8aa803dc9e6f239f1db7c3cdf578249087a404e7c27a505d/Beaker-1.7.0.tar.gz";
       md5 = "386be3f7fe427358881eee4622b428b3";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   CProfileV = super.buildPythonPackage {
     name = "CProfileV-1.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [bottle];
     src = fetchurl {
       url = "https://pypi.python.org/packages/eb/df/983a0b6cfd3ac94abf023f5011cb04f33613ace196e33f53c86cf91850d5/CProfileV-1.0.6.tar.gz";
       md5 = "08c7c242b6e64237bc53c5d13537e03d";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Chameleon = super.buildPythonPackage {
     name = "Chameleon-2.24";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5a/9e/637379ffa13c5172b5c0e704833ffea6bf51cec7567f93fd6e903d53ed74/Chameleon-2.24.tar.gz";
       md5 = "1b01f1f6533a8a11d0d2f2366dec5342";
     };
     meta = {
       license = [ { fullName = "BSD-like (http://repoze.org/license.html)"; } ];
     };
   };
   FormEncode = super.buildPythonPackage {
     name = "FormEncode-1.2.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8e/59/0174271a6f004512e0201188593e6d319db139d14cb7490e488bbb078015/FormEncode-1.2.4.tar.gz";
       md5 = "6bc17fb9aed8aea198975e888e2077f4";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   Jinja2 = super.buildPythonPackage {
     name = "Jinja2-2.7.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b0/73/eab0bca302d6d6a0b5c402f47ad1760dc9cb2dd14bbc1873ad48db258e4d/Jinja2-2.7.3.tar.gz";
       md5 = "b9dffd2f3b43d673802fe857c8445b1a";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   Mako = super.buildPythonPackage {
     name = "Mako-1.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/56/4b/cb75836863a6382199aefb3d3809937e21fa4cb0db15a4f4ba0ecc2e7e8e/Mako-1.0.6.tar.gz";
       md5 = "a28e22a339080316b2acc352b9ee631c";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Markdown = super.buildPythonPackage {
     name = "Markdown-2.6.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/48/a4/fc6b002789c2239ac620ca963694c95b8f74e4747769cdf6021276939e74/Markdown-2.6.7.zip";
       md5 = "632710a7474bbb74a82084392251061f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   MarkupSafe = super.buildPythonPackage {
     name = "MarkupSafe-0.23";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c0/41/bae1254e0396c0cc8cf1751cb7d9afc90a602353695af5952530482c963f/MarkupSafe-0.23.tar.gz";
       md5 = "f5ab3deee4c37cd6a922fb81e730da6e";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   MySQL-python = super.buildPythonPackage {
     name = "MySQL-python-1.2.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a5/e9/51b544da85a36a68debe7a7091f068d802fc515a3a202652828c73453cad/MySQL-python-1.2.5.zip";
       md5 = "654f75b302db6ed8dc5a898c625e030c";
     };
     meta = {
       license = [ pkgs.lib.licenses.gpl1 ];
     };
   };
   Paste = super.buildPythonPackage {
     name = "Paste-2.0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/30/c3/5c2f7c7a02e4f58d4454353fa1c32c94f79fa4e36d07a67c0ac295ea369e/Paste-2.0.3.tar.gz";
       md5 = "1231e14eae62fa7ed76e9130b04bc61e";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   PasteDeploy = super.buildPythonPackage {
     name = "PasteDeploy-1.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0f/90/8e20cdae206c543ea10793cbf4136eb9a8b3f417e04e40a29d72d9922cbd/PasteDeploy-1.5.2.tar.gz";
       md5 = "352b7205c78c8de4987578d19431af3b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   PasteScript = super.buildPythonPackage {
     name = "PasteScript-1.7.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Paste PasteDeploy];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a5/05/fc60efa7c2f17a1dbaeccb2a903a1e90902d92b9d00eebabe3095829d806/PasteScript-1.7.5.tar.gz";
       md5 = "4c72d78dcb6bb993f30536842c16af4d";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Pygments = super.buildPythonPackage {
     name = "Pygments-2.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/71/2a/2e4e77803a8bd6408a2903340ac498cb0a2181811af7c9ec92cb70b0308a/Pygments-2.2.0.tar.gz";
       md5 = "13037baca42f16917cbd5ad2fab50844";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   Pylons = super.buildPythonPackage {
     name = "Pylons-1.0.2.dev20170205";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Routes WebHelpers Beaker Paste PasteDeploy PasteScript FormEncode simplejson decorator nose Mako WebError WebTest Tempita MarkupSafe WebOb];
     src = fetchurl {
       url = "https://code.rhodecode.com/upstream/pylons/archive/707354ee4261b9c10450404fc9852ccea4fd667d.tar.gz?md5=f26633726fa2cd3a340316ee6a5d218f";
       md5 = "f26633726fa2cd3a340316ee6a5d218f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
-  Pyro4 = super.buildPythonPackage {
-    name = "Pyro4-4.41";
-    buildInputs = with self; [];
-    doCheck = false;
-    propagatedBuildInputs = with self; [serpent];
-    src = fetchurl {
-      url = "https://pypi.python.org/packages/56/2b/89b566b4bf3e7f8ba790db2d1223852f8cb454c52cab7693dd41f608ca2a/Pyro4-4.41.tar.gz";
-      md5 = "ed69e9bfafa9c06c049a87cb0c4c2b6c";
-    };
-    meta = {
-      license = [ pkgs.lib.licenses.mit ];
-    };
-  };
   Routes = super.buildPythonPackage {
     name = "Routes-1.13";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [repoze.lru];
     src = fetchurl {
       url = "https://pypi.python.org/packages/88/d3/259c3b3cde8837eb9441ab5f574a660e8a4acea8f54a078441d4d2acac1c/Routes-1.13.tar.gz";
       md5 = "d527b0ab7dd9172b1275a41f97448783";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   SQLAlchemy = super.buildPythonPackage {
     name = "SQLAlchemy-0.9.9";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/28/f7/1bbfd0d8597e8c358d5e15a166a486ad82fc5579b4e67b6ef7c05b1d182b/SQLAlchemy-0.9.9.tar.gz";
       md5 = "8a10a9bd13ed3336ef7333ac2cc679ff";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Sphinx = super.buildPythonPackage {
     name = "Sphinx-1.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Pygments docutils Jinja2];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0a/50/34017e6efcd372893a416aba14b84a1a149fc7074537b0e9cb6ca7b7abe9/Sphinx-1.2.2.tar.gz";
       md5 = "3dc73ccaa8d0bfb2d62fb671b1f7e8a4";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   Tempita = super.buildPythonPackage {
     name = "Tempita-0.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/56/c8/8ed6eee83dbddf7b0fc64dd5d4454bc05e6ccaafff47991f73f2894d9ff4/Tempita-0.5.2.tar.gz";
       md5 = "4c2f17bb9d481821c41b6fbee904cea1";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   URLObject = super.buildPythonPackage {
     name = "URLObject-2.4.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/cb/b6/e25e58500f9caef85d664bec71ec67c116897bfebf8622c32cb75d1ca199/URLObject-2.4.0.tar.gz";
       md5 = "2ed819738a9f0a3051f31dc9924e3065";
     };
     meta = {
       license = [  ];
     };
   };
   WebError = super.buildPythonPackage {
     name = "WebError-0.10.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb Tempita Pygments Paste];
     src = fetchurl {
       url = "https://pypi.python.org/packages/35/76/e7e5c2ce7e9c7f31b54c1ff295a495886d1279a002557d74dd8957346a79/WebError-0.10.3.tar.gz";
       md5 = "84b9990b0baae6fd440b1e60cdd06f9a";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   WebHelpers = super.buildPythonPackage {
     name = "WebHelpers-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ee/68/4d07672821d514184357f1552f2dad923324f597e722de3b016ca4f7844f/WebHelpers-1.3.tar.gz";
       md5 = "32749ffadfc40fea51075a7def32588b";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   WebHelpers2 = super.buildPythonPackage {
     name = "WebHelpers2-2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ff/30/56342c6ea522439e3662427c8d7b5e5b390dff4ff2dc92d8afcb8ab68b75/WebHelpers2-2.0.tar.gz";
       md5 = "0f6b68d70c12ee0aed48c00b24da13d3";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   WebOb = super.buildPythonPackage {
     name = "WebOb-1.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/16/78/adfc0380b8a0d75b2d543fa7085ba98a573b1ae486d9def88d172b81b9fa/WebOb-1.3.1.tar.gz";
       md5 = "20918251c5726956ba8fef22d1556177";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   WebTest = super.buildPythonPackage {
     name = "WebTest-1.4.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb];
     src = fetchurl {
       url = "https://pypi.python.org/packages/51/3d/84fd0f628df10b30c7db87895f56d0158e5411206b721ca903cb51bfd948/WebTest-1.4.3.zip";
       md5 = "631ce728bed92c681a4020a36adbc353";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Whoosh = super.buildPythonPackage {
     name = "Whoosh-2.7.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/25/2b/6beed2107b148edc1321da0d489afc4617b9ed317ef7b72d4993cad9b684/Whoosh-2.7.4.tar.gz";
       md5 = "c2710105f20b3e29936bd2357383c325";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.bsd2 ];
     };
   };
   alembic = super.buildPythonPackage {
     name = "alembic-0.8.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [SQLAlchemy Mako python-editor];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ca/7e/299b4499b5c75e5a38c5845145ad24755bebfb8eec07a2e1c366b7181eeb/alembic-0.8.4.tar.gz";
       md5 = "5f95d8ee62b443f9b37eb5bee76c582d";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   amqplib = super.buildPythonPackage {
     name = "amqplib-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/75/b7/8c2429bf8d92354a0118614f9a4d15e53bc69ebedce534284111de5a0102/amqplib-1.0.2.tgz";
       md5 = "5c92f17fbedd99b2b4a836d4352d1e2f";
     };
     meta = {
       license = [ { fullName = "LGPL"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
     };
   };
   anyjson = super.buildPythonPackage {
     name = "anyjson-0.3.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz";
       md5 = "2ea28d6ec311aeeebaf993cb3008b27c";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   appenlight-client = super.buildPythonPackage {
     name = "appenlight-client-0.6.14";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb requests];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4d/e0/23fee3ebada8143f707e65c06bcb82992040ee64ea8355e044ed55ebf0c1/appenlight_client-0.6.14.tar.gz";
       md5 = "578c69b09f4356d898fff1199b98a95c";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal { fullName = "DFSG approved"; } ];
     };
   };
   authomatic = super.buildPythonPackage {
     name = "authomatic-0.1.0.post1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/08/1a/8a930461e604c2d5a7a871e1ac59fa82ccf994c32e807230c8d2fb07815a/Authomatic-0.1.0.post1.tar.gz";
       md5 = "be3f3ce08747d776aae6d6cc8dcb49a9";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   backport-ipaddress = super.buildPythonPackage {
     name = "backport-ipaddress-0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d3/30/54c6dab05a4dec44db25ff309f1fbb6b7a8bde3f2bade38bb9da67bbab8f/backport_ipaddress-0.1.tar.gz";
       md5 = "9c1f45f4361f71b124d7293a60006c05";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   backports.shutil-get-terminal-size = super.buildPythonPackage {
     name = "backports.shutil-get-terminal-size-1.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ec/9c/368086faa9c016efce5da3e0e13ba392c9db79e3ab740b763fe28620b18b/backports.shutil_get_terminal_size-1.0.0.tar.gz";
       md5 = "03267762480bd86b50580dc19dff3c66";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
+  bleach = super.buildPythonPackage {
+    name = "bleach-1.5.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [six html5lib];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/99/00/25a8fce4de102bf6e3cc76bc4ea60685b2fee33bde1b34830c70cacc26a7/bleach-1.5.0.tar.gz";
+      md5 = "b663300efdf421b3b727b19d7be9c7e7";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
+  };
   bottle = super.buildPythonPackage {
     name = "bottle-0.12.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/52/df/e4a408f3a7af396d186d4ecd3b389dd764f0f943b4fa8d257bfe7b49d343/bottle-0.12.8.tar.gz";
       md5 = "13132c0a8f607bf860810a6ee9064c5b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   bumpversion = super.buildPythonPackage {
     name = "bumpversion-0.5.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/14/41/8c9da3549f8e00c84f0432c3a8cf8ed6898374714676aab91501d48760db/bumpversion-0.5.3.tar.gz";
       md5 = "c66a3492eafcf5ad4b024be9fca29820";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   celery = super.buildPythonPackage {
     name = "celery-2.2.10";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [python-dateutil anyjson kombu pyparsing];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b1/64/860fd50e45844c83442e7953effcddeff66b2851d90b2d784f7201c111b8/celery-2.2.10.tar.gz";
       md5 = "898bc87e54f278055b561316ba73e222";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   channelstream = super.buildPythonPackage {
     name = "channelstream-0.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [gevent ws4py pyramid pyramid-jinja2 itsdangerous requests six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2b/31/29a8e085cf5bf97fa88e7b947adabfc581a18a3463adf77fb6dada34a65f/channelstream-0.5.2.tar.gz";
       md5 = "1c5eb2a8a405be6f1073da94da6d81d3";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   click = super.buildPythonPackage {
     name = "click-5.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b7/34/a496632c4fb6c1ee76efedf77bb8d28b29363d839953d95095b12defe791/click-5.1.tar.gz";
       md5 = "9c5323008cccfe232a8b161fc8196d41";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   colander = super.buildPythonPackage {
     name = "colander-1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [translationstring iso8601];
     src = fetchurl {
       url = "https://pypi.python.org/packages/14/23/c9ceba07a6a1dc0eefbb215fc0dc64aabc2b22ee756bc0f0c13278fa0887/colander-1.2.tar.gz";
       md5 = "83db21b07936a0726e588dae1914b9ed";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   configobj = super.buildPythonPackage {
     name = "configobj-5.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz";
       md5 = "e472a3a1c2a67bb0ec9b5d54c13a47d6";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
+  configparser = super.buildPythonPackage {
+    name = "configparser-3.5.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/7c/69/c2ce7e91c89dc073eb1aa74c0621c3eefbffe8216b3f9af9d3885265c01c/configparser-3.5.0.tar.gz";
+      md5 = "cfdd915a5b7a6c09917a64a573140538";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
   cov-core = super.buildPythonPackage {
     name = "cov-core-1.15.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [coverage];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4b/87/13e75a47b4ba1be06f29f6d807ca99638bedc6b57fa491cd3de891ca2923/cov-core-1.15.0.tar.gz";
       md5 = "f519d4cb4c4e52856afb14af52919fe6";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   coverage = super.buildPythonPackage {
     name = "coverage-3.7.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/09/4f/89b06c7fdc09687bca507dc411c342556ef9c5a3b26756137a4878ff19bf/coverage-3.7.1.tar.gz";
       md5 = "c47b36ceb17eaff3ecfab3bcd347d0df";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   cssselect = super.buildPythonPackage {
-    name = "cssselect-0.9.1";
+    name = "cssselect-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/aa/e5/9ee1460d485b94a6d55732eb7ad5b6c084caf73dd6f9cb0bb7d2a78fafe8/cssselect-0.9.1.tar.gz";
-      md5 = "c74f45966277dc7a0f768b9b0f3522ac";
+      url = "https://pypi.python.org/packages/77/ff/9c865275cd19290feba56344eba570e719efb7ca5b34d67ed12b22ebbb0d/cssselect-1.0.1.tar.gz";
+      md5 = "3fa03bf82a9f0b1223c0f1eb1369e139";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   decorator = super.buildPythonPackage {
-    name = "decorator-3.4.2";
+    name = "decorator-4.0.11";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/35/3a/42566eb7a2cbac774399871af04e11d7ae3fc2579e7dae85213b8d1d1c57/decorator-3.4.2.tar.gz";
-      md5 = "9e0536870d2b83ae27d58dbf22582f4d";
+      url = "https://pypi.python.org/packages/cc/ac/5a16f1fc0506ff72fcc8fd4e858e3a1c231f224ab79bb7c4c9b2094cc570/decorator-4.0.11.tar.gz";
+      md5 = "73644c8f0bd4983d1b6a34b49adec0ae";
     };
     meta = {
-      license = [ pkgs.lib.licenses.bsdOriginal ];
+      license = [ pkgs.lib.licenses.bsdOriginal { fullName = "new BSD License"; } ];
     };
   };
   deform = super.buildPythonPackage {
     name = "deform-2.0a2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Chameleon colander peppercorn translationstring zope.deprecation];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8d/b3/aab57e81da974a806dc9c5fa024a6404720f890a6dcf2e80885e3cb4609a/deform-2.0a2.tar.gz";
       md5 = "7a90d41f7fbc18002ce74f39bd90a5e4";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   docutils = super.buildPythonPackage {
     name = "docutils-0.12";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/37/38/ceda70135b9144d84884ae2fc5886c6baac4edea39550f28bcd144c1234d/docutils-0.12.tar.gz";
       md5 = "4622263b62c5c771c03502afa3157768";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.publicDomain pkgs.lib.licenses.gpl1 { fullName = "public domain, Python, 2-Clause BSD, GPL 3 (see COPYING.txt)"; } pkgs.lib.licenses.psfl ];
     };
   };
   dogpile.cache = super.buildPythonPackage {
     name = "dogpile.cache-0.6.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f6/a0/6f2142c58c6588d17c734265b103ae1cd0741e1681dd9483a63f22033375/dogpile.cache-0.6.1.tar.gz";
       md5 = "35d7fb30f22bbd0685763d894dd079a9";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   dogpile.core = super.buildPythonPackage {
     name = "dogpile.core-0.4.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0e/77/e72abc04c22aedf874301861e5c1e761231c288b5de369c18be8f4b5c9bb/dogpile.core-0.4.1.tar.gz";
       md5 = "01cb19f52bba3e95c9b560f39341f045";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ecdsa = super.buildPythonPackage {
     name = "ecdsa-0.11";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6c/3f/92fe5dcdcaa7bd117be21e5520c9a54375112b66ec000d209e9e9519fad1/ecdsa-0.11.tar.gz";
       md5 = "8ef586fe4dbb156697d756900cb41d7c";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   elasticsearch = super.buildPythonPackage {
     name = "elasticsearch-2.3.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [urllib3];
     src = fetchurl {
       url = "https://pypi.python.org/packages/10/35/5fd52c5f0b0ee405ed4b5195e8bce44c5e041787680dc7b94b8071cac600/elasticsearch-2.3.0.tar.gz";
       md5 = "2550f3b51629cf1ef9636608af92c340";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   elasticsearch-dsl = super.buildPythonPackage {
     name = "elasticsearch-dsl-2.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six python-dateutil elasticsearch];
     src = fetchurl {
       url = "https://pypi.python.org/packages/66/2f/52a086968788e58461641570f45c3207a52d46ebbe9b77dc22b6a8ffda66/elasticsearch-dsl-2.2.0.tar.gz";
       md5 = "fa6bd3c87ea3caa8f0f051bc37c53221";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
+  entrypoints = super.buildPythonPackage {
+    name = "entrypoints-0.2.2";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [configparser];
+    src = fetchurl {
+      url = "https://code.rhodecode.com/upstream/entrypoints/archive/96e6d645684e1af3d7df5b5272f3fe85a546b233.tar.gz?md5=7db37771aea9ac9fefe093e5d6987313";
+      md5 = "7db37771aea9ac9fefe093e5d6987313";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
   enum34 = super.buildPythonPackage {
     name = "enum34-1.1.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bf/3e/31d502c25302814a7c2f1d3959d2a3b3f78e509002ba91aea64993936876/enum34-1.1.6.tar.gz";
       md5 = "5f13a0841a61f7fc295c514490d120d0";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
+  functools32 = super.buildPythonPackage {
+    name = "functools32-3.2.3.post2";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/5e/1a/0aa2c8195a204a9f51284018562dea77e25511f02fe924fac202fc012172/functools32-3.2.3-2.zip";
+      md5 = "d55232eb132ec779e6893c902a0bc5ad";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.psfl ];
+    };
+  };
   future = super.buildPythonPackage {
     name = "future-0.14.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/83/80/8ef3a11a15f8eaafafa0937b20c1b3f73527e69ab6b3fa1cf94a5a96aabb/future-0.14.3.tar.gz";
       md5 = "e94079b0bd1fc054929e8769fc0f6083";
     };
     meta = {
       license = [ { fullName = "OSI Approved"; } pkgs.lib.licenses.mit ];
     };
   };
   futures = super.buildPythonPackage {
     name = "futures-3.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f8/e7/fc0fcbeb9193ba2d4de00b065e7fd5aecd0679e93ce95a07322b2b1434f4/futures-3.0.2.tar.gz";
       md5 = "42aaf1e4de48d6e871d77dc1f9d96d5a";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   gevent = super.buildPythonPackage {
     name = "gevent-1.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [greenlet];
     src = fetchurl {
       url = "https://pypi.python.org/packages/43/8f/cb3224a0e6ab663547f45c10d0651cfd52633fde4283bf68d627084df8cc/gevent-1.1.2.tar.gz";
       md5 = "bb32a2f852a4997138014d5007215c6e";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   gnureadline = super.buildPythonPackage {
     name = "gnureadline-6.3.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3a/ee/2c3f568b0a74974791ac590ec742ef6133e2fbd287a074ba72a53fa5e97c/gnureadline-6.3.3.tar.gz";
       md5 = "c4af83c9a3fbeac8f2da9b5a7c60e51c";
     };
     meta = {
       license = [ pkgs.lib.licenses.gpl1 ];
     };
   };
   gprof2dot = super.buildPythonPackage {
     name = "gprof2dot-2016.10.13";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a0/e0/73c71baed306f0402a00a94ffc7b2be94ad1296dfcb8b46912655b93154c/gprof2dot-2016.10.13.tar.gz";
       md5 = "0125401f15fd2afe1df686a76c64a4fd";
     };
     meta = {
       license = [ { fullName = "LGPL"; } ];
     };
   };
   greenlet = super.buildPythonPackage {
     name = "greenlet-0.4.10";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/67/62/ca2a95648666eaa2ffeb6a9b3964f21d419ae27f82f2e66b53da5b943fc4/greenlet-0.4.10.zip";
       md5 = "bed0c4b3b896702131f4d5c72f87c41d";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   gunicorn = super.buildPythonPackage {
     name = "gunicorn-19.6.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/84/ce/7ea5396efad1cef682bbc4068e72a0276341d9d9d0f501da609fab9fcb80/gunicorn-19.6.0.tar.gz";
       md5 = "338e5e8a83ea0f0625f768dba4597530";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
+  html5lib = super.buildPythonPackage {
+    name = "html5lib-0.9999999";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [six];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/ae/ae/bcb60402c60932b32dfaf19bb53870b29eda2cd17551ba5639219fb5ebf9/html5lib-0.9999999.tar.gz";
+      md5 = "ef43cb05e9e799f25d65d1135838a96f";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
   infrae.cache = super.buildPythonPackage {
     name = "infrae.cache-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Beaker repoze.lru];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bb/f0/e7d5e984cf6592fd2807dc7bc44a93f9d18e04e6a61f87fdfb2622422d74/infrae.cache-1.0.1.tar.gz";
       md5 = "b09076a766747e6ed2a755cc62088e32";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   invoke = super.buildPythonPackage {
     name = "invoke-0.13.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/47/bf/d07ef52fa1ac645468858bbac7cb95b246a972a045e821493d17d89c81be/invoke-0.13.0.tar.gz";
       md5 = "c0d1ed4bfb34eaab551662d8cfee6540";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipdb = super.buildPythonPackage {
     name = "ipdb-0.10.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [ipython setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/eb/0a/0a37dc19572580336ad3813792c0d18c8d7117c2d66fc63c501f13a7a8f8/ipdb-0.10.1.tar.gz";
       md5 = "4aeab65f633ddc98ebdb5eebf08dc713";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipython = super.buildPythonPackage {
     name = "ipython-5.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools decorator pickleshare simplegeneric traitlets prompt-toolkit Pygments pexpect backports.shutil-get-terminal-size pathlib2 pexpect];
     src = fetchurl {
       url = "https://pypi.python.org/packages/89/63/a9292f7cd9d0090a0f995e1167f3f17d5889dcbc9a175261719c513b9848/ipython-5.1.0.tar.gz";
       md5 = "47c8122420f65b58784cb4b9b4af35e3";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipython-genutils = super.buildPythonPackage {
     name = "ipython-genutils-0.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/71/b7/a64c71578521606edbbce15151358598f3dfb72a3431763edc2baf19e71f/ipython_genutils-0.1.0.tar.gz";
       md5 = "9a8afbe0978adbcbfcb3b35b2d015a56";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   iso8601 = super.buildPythonPackage {
     name = "iso8601-0.1.11";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c0/75/c9209ee4d1b5975eb8c2cba4428bde6b61bd55664a98290dd015cdb18e98/iso8601-0.1.11.tar.gz";
       md5 = "b06d11cd14a64096f907086044f0fe38";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   itsdangerous = super.buildPythonPackage {
     name = "itsdangerous-0.24";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz";
       md5 = "a3d55aa79369aef5345c036a8a26307f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
+  jsonschema = super.buildPythonPackage {
+    name = "jsonschema-2.6.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [functools32];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/58/b9/171dbb07e18c6346090a37f03c7e74410a1a56123f847efed59af260a298/jsonschema-2.6.0.tar.gz";
+      md5 = "50c6b69a373a8b55ff1e0ec6e78f13f4";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
+  jupyter-client = super.buildPythonPackage {
+    name = "jupyter-client-5.0.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [traitlets jupyter-core pyzmq python-dateutil];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/e5/6f/65412ed462202b90134b7e761b0b7e7f949e07a549c1755475333727b3d0/jupyter_client-5.0.0.tar.gz";
+      md5 = "1acd331b5c9fb4d79dae9939e79f2426";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
+  jupyter-core = super.buildPythonPackage {
+    name = "jupyter-core-4.3.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [traitlets];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/2f/39/5138f975100ce14d150938df48a83cd852a3fd8e24b1244f4113848e69e2/jupyter_core-4.3.0.tar.gz";
+      md5 = "18819511a809afdeed9a995a9c27bcfb";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
   kombu = super.buildPythonPackage {
     name = "kombu-1.5.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [anyjson amqplib];
     src = fetchurl {
       url = "https://pypi.python.org/packages/19/53/74bf2a624644b45f0850a638752514fc10a8e1cbd738f10804951a6df3f5/kombu-1.5.1.tar.gz";
       md5 = "50662f3c7e9395b3d0721fb75d100b63";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   lxml = super.buildPythonPackage {
-    name = "lxml-3.4.4";
+    name = "lxml-3.7.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz";
-      md5 = "a9a65972afc173ec7a39c585f4eea69c";
+      url = "https://pypi.python.org/packages/39/e8/a8e0b1fa65dd021d48fe21464f71783655f39a41f218293c1c590d54eb82/lxml-3.7.3.tar.gz";
+      md5 = "075692ce442e69bbd604d44e21c02753";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   meld3 = super.buildPythonPackage {
     name = "meld3-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/45/a0/317c6422b26c12fe0161e936fc35f36552069ba8e6f7ecbd99bbffe32a5f/meld3-1.0.2.tar.gz";
       md5 = "3ccc78cd79cffd63a751ad7684c02c91";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
+  mistune = super.buildPythonPackage {
+    name = "mistune-0.7.3";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/88/1e/be99791262b3a794332fda598a07c2749a433b9378586361ba9d8e824607/mistune-0.7.3.tar.gz";
+      md5 = "4eba50bd121b83716fa4be6a4049004b";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
   mock = super.buildPythonPackage {
     name = "mock-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/15/45/30273ee91feb60dabb8fbb2da7868520525f02cf910279b3047182feed80/mock-1.0.1.zip";
       md5 = "869f08d003c289a97c1a6610faf5e913";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   msgpack-python = super.buildPythonPackage {
     name = "msgpack-python-0.4.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/21/27/8a1d82041c7a2a51fcc73675875a5f9ea06c2663e02fcfeb708be1d081a0/msgpack-python-0.4.8.tar.gz";
       md5 = "dcd854fb41ee7584ebbf35e049e6be98";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
+  nbconvert = super.buildPythonPackage {
+    name = "nbconvert-5.1.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [mistune Jinja2 Pygments traitlets jupyter-core nbformat entrypoints bleach pandocfilters testpath];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/95/58/df1c91f1658ee5df19097f915a1e71c91fc824a708d82d2b2e35f8b80e9a/nbconvert-5.1.1.tar.gz";
+      md5 = "d0263fb03a44db2f94eea09a608ed813";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
+  nbformat = super.buildPythonPackage {
+    name = "nbformat-4.3.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [ipython-genutils traitlets jsonschema jupyter-core];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/f9/c5/89df4abf906f766727f976e170caa85b4f1c1d1feb1f45d716016e68e19f/nbformat-4.3.0.tar.gz";
+      md5 = "9a00d20425914cd5ba5f97769d9963ca";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
   nose = super.buildPythonPackage {
     name = "nose-1.3.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/70/c7/469e68148d17a0d3db5ed49150242fd70a74a8147b8f3f8b87776e028d99/nose-1.3.6.tar.gz";
       md5 = "0ca546d81ca8309080fc80cb389e7a16";
     };
     meta = {
       license = [ { fullName = "GNU Library or Lesser General Public License (LGPL)"; } { fullName = "GNU LGPL"; } ];
     };
   };
   objgraph = super.buildPythonPackage {
     name = "objgraph-2.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d7/33/ace750b59247496ed769b170586c5def7202683f3d98e737b75b767ff29e/objgraph-2.0.0.tar.gz";
       md5 = "25b0d5e5adc74aa63ead15699614159c";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   packaging = super.buildPythonPackage {
     name = "packaging-15.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/24/c4/185da1304f07047dc9e0c46c31db75c0351bd73458ac3efad7da3dbcfbe1/packaging-15.2.tar.gz";
       md5 = "c16093476f6ced42128bf610e5db3784";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
+  pandocfilters = super.buildPythonPackage {
+    name = "pandocfilters-1.4.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/e3/1f/21d1b7e8ca571e80b796c758d361fdf5554335ff138158654684bc5401d8/pandocfilters-1.4.1.tar.gz";
+      md5 = "7680d9f9ec07397dd17f380ee3818b9d";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
   paramiko = super.buildPythonPackage {
     name = "paramiko-1.15.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pycrypto ecdsa];
     src = fetchurl {
       url = "https://pypi.python.org/packages/04/2b/a22d2a560c1951abbbf95a0628e245945565f70dc082d9e784666887222c/paramiko-1.15.1.tar.gz";
       md5 = "48c274c3f9b1282932567b21f6acf3b5";
     };
     meta = {
       license = [ { fullName = "LGPL"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
     };
   };
   pathlib2 = super.buildPythonPackage {
     name = "pathlib2-2.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c9/27/8448b10d8440c08efeff0794adf7d0ed27adb98372c70c7b38f3947d4749/pathlib2-2.1.0.tar.gz";
       md5 = "38e4f58b4d69dfcb9edb49a54a8b28d2";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   peppercorn = super.buildPythonPackage {
     name = "peppercorn-0.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/45/ec/a62ec317d1324a01567c5221b420742f094f05ee48097e5157d32be3755c/peppercorn-0.5.tar.gz";
       md5 = "f08efbca5790019ab45d76b7244abd40";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pexpect = super.buildPythonPackage {
     name = "pexpect-4.2.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [ptyprocess];
     src = fetchurl {
       url = "https://pypi.python.org/packages/e8/13/d0b0599099d6cd23663043a2a0bb7c61e58c6ba359b2656e6fb000ef5b98/pexpect-4.2.1.tar.gz";
       md5 = "3694410001a99dff83f0b500a1ca1c95";
     };
     meta = {
       license = [ pkgs.lib.licenses.isc { fullName = "ISC License (ISCL)"; } ];
     };
   };
   pickleshare = super.buildPythonPackage {
     name = "pickleshare-0.7.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pathlib2];
     src = fetchurl {
       url = "https://pypi.python.org/packages/69/fe/dd137d84daa0fd13a709e448138e310d9ea93070620c9db5454e234af525/pickleshare-0.7.4.tar.gz";
       md5 = "6a9e5dd8dfc023031f6b7b3f824cab12";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   prompt-toolkit = super.buildPythonPackage {
-    name = "prompt-toolkit-1.0.9";
+    name = "prompt-toolkit-1.0.13";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six wcwidth];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/83/14/5ac258da6c530eca02852ee25c7a9ff3ca78287bb4c198d0d0055845d856/prompt_toolkit-1.0.9.tar.gz";
-      md5 = "a39f91a54308fb7446b1a421c11f227c";
+      url = "https://pypi.python.org/packages/23/be/4876b52d5cc159cbd4b0ff6e7aa419a26470849a43a8f647857a4a24467b/prompt_toolkit-1.0.13.tar.gz";
+      md5 = "427b496d2c147bd3819bc3a7f6e0d493";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   psutil = super.buildPythonPackage {
     name = "psutil-4.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/78/cc/f267a1371f229bf16db6a4e604428c3b032b823b83155bd33cef45e49a53/psutil-4.3.1.tar.gz";
       md5 = "199a366dba829c88bddaf5b41d19ddc0";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   psycopg2 = super.buildPythonPackage {
     name = "psycopg2-2.6.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/86/fd/cc8315be63a41fe000cce20482a917e874cdc1151e62cb0141f5e55f711e/psycopg2-2.6.1.tar.gz";
       md5 = "842b44f8c95517ed5b792081a2370da1";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 { fullName = "GNU Library or Lesser General Public License (LGPL)"; } { fullName = "LGPL with exceptions or ZPL"; } ];
     };
   };
   ptyprocess = super.buildPythonPackage {
     name = "ptyprocess-0.5.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/db/d7/b465161910f3d1cef593c5e002bff67e0384898f597f1a7fdc8db4c02bf6/ptyprocess-0.5.1.tar.gz";
       md5 = "94e537122914cc9ec9c1eadcd36e73a1";
     };
     meta = {
       license = [  ];
     };
   };
   py = super.buildPythonPackage {
     name = "py-1.4.31";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f4/9a/8dfda23f36600dd701c6722316ba8a3ab4b990261f83e7d3ffc6dfedf7ef/py-1.4.31.tar.gz";
       md5 = "5d2c63c56dc3f2115ec35c066ecd582b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   py-bcrypt = super.buildPythonPackage {
     name = "py-bcrypt-0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/68/b1/1c3068c5c4d2e35c48b38dcc865301ebfdf45f54507086ac65ced1fd3b3d/py-bcrypt-0.4.tar.gz";
       md5 = "dd8b367d6b716a2ea2e72392525f4e36";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   py-gfm = super.buildPythonPackage {
     name = "py-gfm-0.1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools Markdown];
     src = fetchurl {
       url = "https://code.rhodecode.com/upstream/py-gfm/archive/0d66a19bc16e3d49de273c0f797d4e4781e8c0f2.tar.gz?md5=0d0d5385bfb629eea636a80b9c2bfd16";
       md5 = "0d0d5385bfb629eea636a80b9c2bfd16";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   pycrypto = super.buildPythonPackage {
     name = "pycrypto-2.6.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/60/db/645aa9af249f059cc3a368b118de33889219e0362141e75d4eaf6f80f163/pycrypto-2.6.1.tar.gz";
       md5 = "55a61a054aa66812daf5161a0d5d7eda";
     };
     meta = {
       license = [ pkgs.lib.licenses.publicDomain ];
     };
   };
   pycurl = super.buildPythonPackage {
     name = "pycurl-7.19.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6c/48/13bad289ef6f4869b1d8fc11ae54de8cfb3cc4a2eb9f7419c506f763be46/pycurl-7.19.5.tar.gz";
       md5 = "47b4eac84118e2606658122104e62072";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit { fullName = "LGPL/MIT"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
     };
   };
   pyflakes = super.buildPythonPackage {
     name = "pyflakes-0.8.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/75/22/a90ec0252f4f87f3ffb6336504de71fe16a49d69c4538dae2f12b9360a38/pyflakes-0.8.1.tar.gz";
       md5 = "905fe91ad14b912807e8fdc2ac2e2c23";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pygments-markdown-lexer = super.buildPythonPackage {
     name = "pygments-markdown-lexer-0.1.0.dev39";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Pygments];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c3/12/674cdee66635d638cedb2c5d9c85ce507b7b2f91bdba29e482f1b1160ff6/pygments-markdown-lexer-0.1.0.dev39.zip";
       md5 = "6360fe0f6d1f896e35b7a0142ce6459c";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   pyparsing = super.buildPythonPackage {
     name = "pyparsing-1.5.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2e/26/e8fb5b4256a5f5036be7ce115ef8db8d06bc537becfbdc46c6af008314ee/pyparsing-1.5.7.zip";
       md5 = "b86854857a368d6ccb4d5b6e76d0637f";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pyramid = super.buildPythonPackage {
-    name = "pyramid-1.6.1";
+    name = "pyramid-1.7.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools WebOb repoze.lru zope.interface zope.deprecation venusian translationstring PasteDeploy];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/30/b3/fcc4a2a4800cbf21989e00454b5828cf1f7fe35c63e0810b350e56d4c475/pyramid-1.6.1.tar.gz";
-      md5 = "b18688ff3cc33efdbb098a35b45dd122";
+      url = "https://pypi.python.org/packages/33/91/55f5c661f8923902cd1f68d75f2b937c45e7682857356cf18f0be5493899/pyramid-1.7.4.tar.gz";
+      md5 = "6ef1dfdcff9136d04490410757c4c446";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pyramid-beaker = super.buildPythonPackage {
     name = "pyramid-beaker-0.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid Beaker];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d9/6e/b85426e00fd3d57f4545f74e1c3828552d8700f13ededeef9233f7bca8be/pyramid_beaker-0.8.tar.gz";
       md5 = "22f14be31b06549f80890e2c63a93834";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pyramid-debugtoolbar = super.buildPythonPackage {
-    name = "pyramid-debugtoolbar-2.4.2";
+    name = "pyramid-debugtoolbar-3.0.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid pyramid-mako repoze.lru Pygments];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/89/00/ed5426ee41ed747ba3ffd30e8230841a6878286ea67d480b1444d24f06a2/pyramid_debugtoolbar-2.4.2.tar.gz";
-      md5 = "073ea67086cc4bd5decc3a000853642d";
+      url = "https://pypi.python.org/packages/64/0e/df00bfb55605900e7a2f7e4a18dd83575a6651688e297d5a0aa4c208fd7d/pyramid_debugtoolbar-3.0.5.tar.gz";
+      md5 = "aebab8c3bfdc6f89e4d3adc1d126538e";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } pkgs.lib.licenses.bsdOriginal ];
     };
   };
   pyramid-jinja2 = super.buildPythonPackage {
     name = "pyramid-jinja2-2.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid zope.deprecation Jinja2 MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a1/80/595e26ffab7deba7208676b6936b7e5a721875710f982e59899013cae1ed/pyramid_jinja2-2.5.tar.gz";
       md5 = "07cb6547204ac5e6f0b22a954ccee928";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pyramid-mako = super.buildPythonPackage {
     name = "pyramid-mako-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid Mako];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f1/92/7e69bcf09676d286a71cb3bbb887b16595b96f9ba7adbdc239ffdd4b1eb9/pyramid_mako-1.0.2.tar.gz";
       md5 = "ee25343a97eb76bd90abdc2a774eb48a";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pysqlite = super.buildPythonPackage {
     name = "pysqlite-2.6.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5c/a6/1c429cd4c8069cf4bfbd0eb4d592b3f4042155a8202df83d7e9b93aa3dc2/pysqlite-2.6.3.tar.gz";
       md5 = "7ff1cedee74646b50117acff87aa1cfa";
     };
     meta = {
       license = [ { fullName = "zlib/libpng License"; } { fullName = "zlib/libpng license"; } ];
     };
   };
   pytest = super.buildPythonPackage {
     name = "pytest-3.0.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [py];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a8/87/b7ca49efe52d2b4169f2bfc49aa5e384173c4619ea8e635f123a0dac5b75/pytest-3.0.5.tar.gz";
       md5 = "cefd527b59332688bf5db4a10aa8a7cb";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-catchlog = super.buildPythonPackage {
     name = "pytest-catchlog-1.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [py pytest];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f2/2b/2faccdb1a978fab9dd0bf31cca9f6847fbe9184a0bdcc3011ac41dd44191/pytest-catchlog-1.2.2.zip";
       md5 = "09d890c54c7456c818102b7ff8c182c8";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-cov = super.buildPythonPackage {
     name = "pytest-cov-2.4.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytest coverage];
     src = fetchurl {
       url = "https://pypi.python.org/packages/00/c0/2bfd1fcdb9d407b8ac8185b1cb5ff458105c6b207a9a7f0e13032de9828f/pytest-cov-2.4.0.tar.gz";
       md5 = "2fda09677d232acc99ec1b3c5831e33f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.mit ];
     };
   };
   pytest-profiling = super.buildPythonPackage {
     name = "pytest-profiling-1.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six pytest gprof2dot];
     src = fetchurl {
       url = "https://pypi.python.org/packages/73/e8/804681323bac0bc45c520ec34185ba8469008942266d0074699b204835c1/pytest-profiling-1.2.2.tar.gz";
       md5 = "0a16d7dda2d23b91e9730fa4558cf728";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-runner = super.buildPythonPackage {
     name = "pytest-runner-2.9";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/11/d4/c335ddf94463e451109e3494e909765c3e5205787b772e3b25ee8601b86a/pytest-runner-2.9.tar.gz";
       md5 = "2212a2e34404b0960b2fdc2c469247b2";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-sugar = super.buildPythonPackage {
     name = "pytest-sugar-0.7.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytest termcolor];
     src = fetchurl {
       url = "https://pypi.python.org/packages/03/97/05d988b4fa870e7373e8ee4582408543b9ca2bd35c3c67b569369c6f9c49/pytest-sugar-0.7.1.tar.gz";
       md5 = "7400f7c11f3d572b2c2a3b60352d35fe";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   pytest-timeout = super.buildPythonPackage {
     name = "pytest-timeout-1.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytest];
     src = fetchurl {
       url = "https://pypi.python.org/packages/cc/b7/b2a61365ea6b6d2e8881360ae7ed8dad0327ad2df89f2f0be4a02304deb2/pytest-timeout-1.2.0.tar.gz";
       md5 = "83607d91aa163562c7ee835da57d061d";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit { fullName = "DFSG approved"; } ];
     };
   };
   python-dateutil = super.buildPythonPackage {
-    name = "python-dateutil-1.5";
+    name = "python-dateutil-2.1";
     buildInputs = with self; [];
     doCheck = false;
-    propagatedBuildInputs = with self; [];
+    propagatedBuildInputs = with self; [six];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/b4/7c/df59c89a753eb33c7c44e1dd42de0e9bc2ccdd5a4d576e0bfad97cc280cb/python-dateutil-1.5.tar.gz";
-      md5 = "0dcb1de5e5cad69490a3b6ab63f0cfa5";
+      url = "https://pypi.python.org/packages/65/52/9c18dac21f174ad31b65e22d24297864a954e6fe65876eba3f5773d2da43/python-dateutil-2.1.tar.gz";
+      md5 = "1534bb15cf311f07afaa3aacba1c028b";
     };
     meta = {
-      license = [ pkgs.lib.licenses.psfl ];
+      license = [ { fullName = "Simplified BSD"; } ];
     };
   };
   python-editor = super.buildPythonPackage {
     name = "python-editor-1.0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/65/1e/adf6e000ea5dc909aa420352d6ba37f16434c8a3c2fa030445411a1ed545/python-editor-1.0.3.tar.gz";
       md5 = "0aca5f2ef176ce68e98a5b7e31372835";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 { fullName = "Apache"; } ];
     };
   };
   python-ldap = super.buildPythonPackage {
     name = "python-ldap-2.4.19";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/42/81/1b64838c82e64f14d4e246ff00b52e650a35c012551b891ada2b85d40737/python-ldap-2.4.19.tar.gz";
       md5 = "b941bf31d09739492aa19ef679e94ae3";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   python-memcached = super.buildPythonPackage {
     name = "python-memcached-1.57";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/52/9d/eebc0dcbc5c7c66840ad207dfc1baa376dadb74912484bff73819cce01e6/python-memcached-1.57.tar.gz";
       md5 = "de21f64b42b2d961f3d4ad7beb5468a1";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   python-pam = super.buildPythonPackage {
     name = "python-pam-1.8.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/de/8c/f8f5d38b4f26893af267ea0b39023d4951705ab0413a39e0cf7cf4900505/python-pam-1.8.2.tar.gz";
       md5 = "db71b6b999246fb05d78ecfbe166629d";
     };
     meta = {
       license = [ { fullName = "License :: OSI Approved :: MIT License"; } pkgs.lib.licenses.mit ];
     };
   };
   pytz = super.buildPythonPackage {
     name = "pytz-2015.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/7e/1a/f43b5c92df7b156822030fed151327ea096bcf417e45acc23bd1df43472f/pytz-2015.4.zip";
       md5 = "233f2a2b370d03f9b5911700cc9ebf3c";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pyzmq = super.buildPythonPackage {
     name = "pyzmq-14.6.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8a/3b/5463d5a9d712cd8bbdac335daece0d69f6a6792da4e3dd89956c0db4e4e6/pyzmq-14.6.0.tar.gz";
       md5 = "395b5de95a931afa5b14c9349a5b8024";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal { fullName = "LGPL+BSD"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
     };
   };
   recaptcha-client = super.buildPythonPackage {
     name = "recaptcha-client-1.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0a/ea/5f2fbbfd894bdac1c68ef8d92019066cfcf9fbff5fe3d728d2b5c25c8db4/recaptcha-client-1.0.6.tar.gz";
       md5 = "74228180f7e1fb76c4d7089160b0d919";
     };
     meta = {
       license = [ { fullName = "MIT/X11"; } ];
     };
   };
   repoze.lru = super.buildPythonPackage {
     name = "repoze.lru-0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6e/1e/aa15cc90217e086dc8769872c8778b409812ff036bf021b15795638939e4/repoze.lru-0.6.tar.gz";
       md5 = "2c3b64b17a8e18b405f55d46173e14dd";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   requests = super.buildPythonPackage {
     name = "requests-2.9.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f9/6d/07c44fb1ebe04d069459a189e7dab9e4abfe9432adcd4477367c25332748/requests-2.9.1.tar.gz";
       md5 = "0b7f480d19012ec52bab78292efd976d";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   rhodecode-enterprise-ce = super.buildPythonPackage {
-    name = "rhodecode-enterprise-ce-4.6.1";
-    buildInputs = with self; [pytest py pytest-cov pytest-sugar pytest-runner pytest-catchlog pytest-profiling gprof2dot pytest-timeout mock WebTest cov-core coverage cssselect lxml configobj];
+    name = "rhodecode-enterprise-ce-4.7.0";
+    buildInputs = with self; [pytest py pytest-cov pytest-sugar pytest-runner pytest-catchlog pytest-profiling gprof2dot pytest-timeout mock WebTest cov-core coverage configobj];
     doCheck = true;
-    propagatedBuildInputs = with self; [Babel Beaker FormEncode Mako Markdown MarkupSafe MySQL-python Paste PasteDeploy PasteScript Pygments pygments-markdown-lexer Pylons Pyro4 Routes SQLAlchemy Tempita URLObject WebError WebHelpers WebHelpers2 WebOb WebTest Whoosh alembic amqplib anyjson appenlight-client authomatic backport-ipaddress celery channelstream colander decorator deform docutils gevent gunicorn infrae.cache ipython iso8601 kombu msgpack-python packaging psycopg2 py-gfm pycrypto pycurl pyparsing pyramid pyramid-debugtoolbar pyramid-mako pyramid-beaker pysqlite python-dateutil python-ldap python-memcached python-pam recaptcha-client repoze.lru requests simplejson subprocess32 waitress zope.cachedescriptors dogpile.cache dogpile.core psutil py-bcrypt];
+    propagatedBuildInputs = with self; [Babel Beaker FormEncode Mako Markdown MarkupSafe MySQL-python Paste PasteDeploy PasteScript Pygments pygments-markdown-lexer Pylons Routes SQLAlchemy Tempita URLObject WebError WebHelpers WebHelpers2 WebOb WebTest Whoosh alembic amqplib anyjson appenlight-client authomatic backport-ipaddress cssselect celery channelstream colander decorator deform docutils gevent gunicorn infrae.cache ipython iso8601 kombu lxml msgpack-python nbconvert packaging psycopg2 py-gfm pycrypto pycurl pyparsing pyramid pyramid-debugtoolbar pyramid-mako pyramid-beaker pysqlite python-dateutil python-ldap python-memcached python-pam recaptcha-client repoze.lru requests simplejson subprocess32 waitress zope.cachedescriptors dogpile.cache dogpile.core psutil py-bcrypt];
     src = ./.;
     meta = {
       license = [ { fullName = "Affero GNU General Public License v3 or later (AGPLv3+)"; } { fullName = "AGPLv3, and Commercial License"; } ];
     };
   };
   rhodecode-tools = super.buildPythonPackage {
     name = "rhodecode-tools-0.11.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [click future six Mako MarkupSafe requests elasticsearch elasticsearch-dsl urllib3 Whoosh];
     src = fetchurl {
       url = "https://code.rhodecode.com/rhodecode-tools-ce/archive/v0.11.0.tar.gz?md5=e5fd0a8363af08a0ced71b50ca9cce15";
       md5 = "e5fd0a8363af08a0ced71b50ca9cce15";
     };
     meta = {
       license = [ { fullName = "AGPLv3 and Proprietary"; } ];
     };
   };
-  serpent = super.buildPythonPackage {
-    name = "serpent-1.15";
-    buildInputs = with self; [];
-    doCheck = false;
-    propagatedBuildInputs = with self; [];
-    src = fetchurl {
-      url = "https://pypi.python.org/packages/7b/38/b2b27673a882ff2ea5871bb3e3e6b496ebbaafd1612e51990ffb158b9254/serpent-1.15.tar.gz";
-      md5 = "e27b1aad5c218e16442f52abb7c7053a";
-    };
-    meta = {
-      license = [ pkgs.lib.licenses.mit ];
-    };
-  };
   setproctitle = super.buildPythonPackage {
     name = "setproctitle-1.1.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/33/c3/ad367a4f4f1ca90468863ae727ac62f6edb558fc09a003d344a02cfc6ea6/setproctitle-1.1.8.tar.gz";
       md5 = "728f4c8c6031bbe56083a48594027edd";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   setuptools = super.buildPythonPackage {
     name = "setuptools-30.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/1e/43/002c8616db9a3e7be23c2556e39b90a32bb40ba0dc652de1999d5334d372/setuptools-30.1.0.tar.gz";
       md5 = "cac497f42e5096ac8df29e38d3f81c3e";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   setuptools-scm = super.buildPythonPackage {
     name = "setuptools-scm-1.15.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/80/b7/31b6ae5fcb188e37f7e31abe75f9be90490a5456a72860fa6e643f8a3cbc/setuptools_scm-1.15.0.tar.gz";
       md5 = "b6916c78ed6253d6602444fad4279c5b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   simplegeneric = super.buildPythonPackage {
     name = "simplegeneric-0.8.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip";
       md5 = "f9c1fab00fd981be588fc32759f474e3";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   simplejson = super.buildPythonPackage {
     name = "simplejson-3.7.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6d/89/7f13f099344eea9d6722779a1f165087cb559598107844b1ac5dbd831fb1/simplejson-3.7.2.tar.gz";
       md5 = "a5fc7d05d4cb38492285553def5d4b46";
     };
     meta = {
       license = [ { fullName = "Academic Free License (AFL)"; } pkgs.lib.licenses.mit ];
     };
   };
   six = super.buildPythonPackage {
     name = "six-1.9.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/16/64/1dc5e5976b17466fd7d712e59cbe9fb1e18bec153109e5ba3ed6c9102f1a/six-1.9.0.tar.gz";
       md5 = "476881ef4012262dfc8adc645ee786c4";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   subprocess32 = super.buildPythonPackage {
     name = "subprocess32-3.2.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/28/8d/33ccbff51053f59ae6c357310cac0e79246bbed1d345ecc6188b176d72c3/subprocess32-3.2.6.tar.gz";
       md5 = "754c5ab9f533e764f931136974b618f1";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   supervisor = super.buildPythonPackage {
     name = "supervisor-3.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [meld3];
     src = fetchurl {
       url = "https://pypi.python.org/packages/80/37/964c0d53cbd328796b1aeb7abea4c0f7b0e8c7197ea9b0b9967b7d004def/supervisor-3.3.1.tar.gz";
       md5 = "202f760f9bf4930ec06557bac73e5cf2";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   termcolor = super.buildPythonPackage {
     name = "termcolor-1.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz";
       md5 = "043e89644f8909d462fbbfa511c768df";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
+  testpath = super.buildPythonPackage {
+    name = "testpath-0.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/f9/c4/c0b22f35138bc26a6058c39cb61db1e8977e5e9550b12cd2cb02ef56fc51/testpath-0.1.tar.gz";
+      md5 = "401918bcd0b0e5b71a9b909835117bc6";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
   traitlets = super.buildPythonPackage {
-    name = "traitlets-4.3.1";
+    name = "traitlets-4.3.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [ipython-genutils six decorator enum34];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/b1/d6/5b5aa6d5c474691909b91493da1e8972e309c9f01ecfe4aeafd272eb3234/traitlets-4.3.1.tar.gz";
-      md5 = "dd0b1b6e5d31ce446d55a4b5e5083c98";
+      url = "https://pypi.python.org/packages/a5/98/7f5ef2fe9e9e071813aaf9cb91d1a732e0a68b6c44a32b38cb8e14c3f069/traitlets-4.3.2.tar.gz";
+      md5 = "3068663f2f38fd939a9eb3a500ccc154";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   transifex-client = super.buildPythonPackage {
     name = "transifex-client-0.10";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f3/4e/7b925192aee656fb3e04fa6381c8b3dc40198047c3b4a356f6cfd642c809/transifex-client-0.10.tar.gz";
       md5 = "5549538d84b8eede6b254cd81ae024fa";
     };
     meta = {
       license = [ pkgs.lib.licenses.gpl2 ];
     };
   };
   translationstring = super.buildPythonPackage {
     name = "translationstring-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5e/eb/bee578cc150b44c653b63f5ebe258b5d0d812ddac12497e5f80fcad5d0b4/translationstring-1.3.tar.gz";
       md5 = "a4b62e0f3c189c783a1685b3027f7c90";
     };
     meta = {
       license = [ { fullName = "BSD-like (http://repoze.org/license.html)"; } ];
     };
   };
   trollius = super.buildPythonPackage {
     name = "trollius-1.0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [futures];
     src = fetchurl {
       url = "https://pypi.python.org/packages/aa/e6/4141db437f55e6ee7a3fb69663239e3fde7841a811b4bef293145ad6c836/trollius-1.0.4.tar.gz";
       md5 = "3631a464d49d0cbfd30ab2918ef2b783";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   uWSGI = super.buildPythonPackage {
     name = "uWSGI-2.0.11.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/9b/78/918db0cfab0546afa580c1e565209c49aaf1476bbfe491314eadbe47c556/uwsgi-2.0.11.2.tar.gz";
       md5 = "1f02dcbee7f6f61de4b1fd68350cf16f";
     };
     meta = {
       license = [ pkgs.lib.licenses.gpl2 ];
     };
   };
   urllib3 = super.buildPythonPackage {
     name = "urllib3-1.16";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3b/f0/e763169124e3f5db0926bc3dbfcd580a105f9ca44cf5d8e6c7a803c9f6b5/urllib3-1.16.tar.gz";
       md5 = "fcaab1c5385c57deeb7053d3d7d81d59";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   venusian = super.buildPythonPackage {
     name = "venusian-1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/86/20/1948e0dfc4930ddde3da8c33612f6a5717c0b4bc28f591a5c5cf014dd390/venusian-1.0.tar.gz";
       md5 = "dccf2eafb7113759d60c86faf5538756";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   waitress = super.buildPythonPackage {
     name = "waitress-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/78/7d/84d11b96c3f60164dec3bef4a859a03aeae0231aa93f57fbe0d05fa4ff36/waitress-1.0.1.tar.gz";
       md5 = "dda92358a7569669086155923a46e57c";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   wcwidth = super.buildPythonPackage {
     name = "wcwidth-0.1.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/55/11/e4a2bb08bb450fdbd42cc709dd40de4ed2c472cf0ccb9e64af22279c5495/wcwidth-0.1.7.tar.gz";
       md5 = "b3b6a0a08f0c8a34d1de8cf44150a4ad";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   ws4py = super.buildPythonPackage {
     name = "ws4py-0.3.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b6/4f/34af703be86939629479e74d6e650e39f3bd73b3b09212c34e5125764cbc/ws4py-0.3.5.zip";
       md5 = "a261b75c20b980e55ce7451a3576a867";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   wsgiref = super.buildPythonPackage {
     name = "wsgiref-0.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/41/9e/309259ce8dff8c596e8c26df86dbc4e848b9249fd36797fd60be456f03fc/wsgiref-0.1.2.zip";
       md5 = "29b146e6ebd0f9fb119fe321f7bcf6cb";
     };
     meta = {
       license = [ { fullName = "PSF or ZPL"; } ];
     };
   };
   zope.cachedescriptors = super.buildPythonPackage {
     name = "zope.cachedescriptors-4.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/40/33/694b6644c37f28553f4b9f20b3c3a20fb709a22574dff20b5bdffb09ecd5/zope.cachedescriptors-4.0.0.tar.gz";
       md5 = "8d308de8c936792c8e758058fcb7d0f0";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   zope.deprecation = super.buildPythonPackage {
     name = "zope.deprecation-4.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c1/d3/3919492d5e57d8dd01b36f30b34fc8404a30577392b1eb817c303499ad20/zope.deprecation-4.1.2.tar.gz";
       md5 = "e9a663ded58f4f9f7881beb56cae2782";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   zope.event = super.buildPythonPackage {
     name = "zope.event-4.0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c1/29/91ba884d7d6d96691df592e9e9c2bfa57a47040ec1ff47eff18c85137152/zope.event-4.0.3.tar.gz";
       md5 = "9a3780916332b18b8b85f522bcc3e249";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   zope.interface = super.buildPythonPackage {
     name = "zope.interface-4.1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/9d/81/2509ca3c6f59080123c1a8a97125eb48414022618cec0e64eb1313727bfe/zope.interface-4.1.3.tar.gz";
       md5 = "9ae3d24c0c7415deb249dd1a132f0f79";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
 
 ### Test requirements
 
   
 }

pytest.ini

@@ -1,12 +1,11 b''
 [pytest]
 testpaths = ./rhodecode
 pylons_config = rhodecode/tests/rhodecode.ini
 vcsserver_protocol = http
-vcsserver_config_pyro4 = rhodecode/tests/vcsserver_pyro4.ini
 vcsserver_config_http = rhodecode/tests/vcsserver_http.ini
 norecursedirs = tests/scripts
 addopts = -k "not _BaseTest"
 markers =
   vcs_operations: Mark tests depending on a running RhodeCode instance.
   xfail_backends: Mark tests as xfail for given backends.
   skip_backends: Mark tests as skipped for given backends.

requirements.txt

@@ -1,131 +1,136 b''
 ## core
 setuptools==30.1.0
 setuptools-scm==1.15.0
 
 amqplib==1.0.2
 anyjson==0.3.3
 authomatic==0.1.0.post1
 Babel==1.3
 backport-ipaddress==0.1
 Beaker==1.7.0
 celery==2.2.10
 Chameleon==2.24
 channelstream==0.5.2
 click==5.1
 colander==1.2
 configobj==5.0.6
-decorator==3.4.2
+cssselect==1.0.1
+decorator==4.0.11
 deform==2.0a2
 docutils==0.12
 dogpile.cache==0.6.1
 dogpile.core==0.4.1
 ecdsa==0.11
 FormEncode==1.2.4
 future==0.14.3
 futures==3.0.2
 gnureadline==6.3.3
 infrae.cache==1.0.1
 iso8601==0.1.11
 itsdangerous==0.24
 Jinja2==2.7.3
 kombu==1.5.1
+lxml==3.7.3
 Mako==1.0.6
 Markdown==2.6.7
 MarkupSafe==0.23
 meld3==1.0.2
 msgpack-python==0.4.8
 MySQL-python==1.2.5
 nose==1.3.6
 objgraph==2.0.0
 packaging==15.2
 paramiko==1.15.1
 Paste==2.0.3
 PasteDeploy==1.5.2
 PasteScript==1.7.5
+pathlib2==2.1.0
 psutil==4.3.1
 psycopg2==2.6.1
 py-bcrypt==0.4
 pycrypto==2.6.1
 pycurl==7.19.5
 pyflakes==0.8.1
 pygments-markdown-lexer==0.1.0.dev39
 Pygments==2.2.0
 pyparsing==1.5.7
 pyramid-beaker==0.8
-pyramid-debugtoolbar==2.4.2
+pyramid-debugtoolbar==3.0.5
 pyramid-jinja2==2.5
 pyramid-mako==1.0.2
-pyramid==1.6.1
+pyramid==1.7.4
 pysqlite==2.6.3
-python-dateutil==1.5
+python-dateutil==2.1
 python-ldap==2.4.19
 python-memcached==1.57
 python-pam==1.8.2
 pytz==2015.4
 pyzmq==14.6.0
 recaptcha-client==1.0.6
 repoze.lru==0.6
 requests==2.9.1
 Routes==1.13
 setproctitle==1.1.8
 simplejson==3.7.2
 six==1.9.0
 Sphinx==1.2.2
 SQLAlchemy==0.9.9
 subprocess32==3.2.6
 supervisor==3.3.1
 Tempita==0.5.2
 translationstring==1.3
 trollius==1.0.4
 urllib3==1.16
 URLObject==2.4.0
 venusian==1.0
 WebError==0.10.3
 WebHelpers2==2.0
 WebHelpers==1.3
 WebOb==1.3.1
 Whoosh==2.7.4
 wsgiref==0.1.2
 zope.cachedescriptors==4.0.0
 zope.deprecation==4.1.2
 zope.event==4.0.3
 zope.interface==4.1.3
 
 ## customized/patched libs
 # our patched version of Pylons==1.0.2
 https://code.rhodecode.com/upstream/pylons/archive/707354ee4261b9c10450404fc9852ccea4fd667d.tar.gz?md5=f26633726fa2cd3a340316ee6a5d218f#egg=Pylons==1.0.2.rhodecode-patch-1
 # not released py-gfm==0.1.3
 https://code.rhodecode.com/upstream/py-gfm/archive/0d66a19bc16e3d49de273c0f797d4e4781e8c0f2.tar.gz?md5=0d0d5385bfb629eea636a80b9c2bfd16#egg=py-gfm==0.1.3.rhodecode-upstream1
 
+# IPYTHON RENDERING
+# entrypoints backport, pypi version doesn't support egg installs
+https://code.rhodecode.com/upstream/entrypoints/archive/96e6d645684e1af3d7df5b5272f3fe85a546b233.tar.gz?md5=7db37771aea9ac9fefe093e5d6987313#egg=entrypoints==0.2.2.rhodecode-upstream1
+nbconvert==5.1.1
+nbformat==4.3.0
+jupyter_client==5.0.0
 
 ## cli tools
 alembic==0.8.4
 invoke==0.13.0
 bumpversion==0.5.3
 transifex-client==0.10
 
 ## http servers
 gevent==1.1.2
 greenlet==0.4.10
 gunicorn==19.6.0
 waitress==1.0.1
 uWSGI==2.0.11.2
 
 ## debug
 ipdb==0.10.1
 ipython==5.1.0
 CProfileV==1.0.6
 bottle==0.12.8
 
 ## rhodecode-tools, special case
 https://code.rhodecode.com/rhodecode-tools-ce/archive/v0.11.0.tar.gz?md5=e5fd0a8363af08a0ced71b50ca9cce15#egg=rhodecode-tools==0.11.0
 
 ## appenlight
 appenlight-client==0.6.14
 
-# Pyro/Deprecated TODO(Marcink): remove in 4.7 release.
-Pyro4==4.41
-serpent==1.15
-
 ## test related requirements
 -r requirements_test.txt

requirements_test.txt

@@ -1,17 +1,15 b''
 # test related requirements
 pytest==3.0.5
 py==1.4.31
 pytest-cov==2.4.0
 pytest-sugar==0.7.1
 pytest-runner==2.9.0
 pytest-catchlog==1.2.2
 pytest-profiling==1.2.2
 gprof2dot==2016.10.13
 pytest-timeout==1.2.0
 
 mock==1.0.1
 WebTest==1.4.3
 cov-core==1.15.0
 coverage==3.7.1
-cssselect==0.9.1
-lxml==3.4.4

rhodecode/VERSION

@@ -1,1 +1,1 b''
-4.6.1
  No newline at end of file
+4.7.0
  No newline at end of file

rhodecode/__init__.py

@@ -1,63 +1,63 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 
 RhodeCode, a web based repository management software
 versioning implementation: http://www.python.org/dev/peps/pep-0386/
 """
 
 import os
 import sys
 import platform
 
 VERSION = tuple(open(os.path.join(
     os.path.dirname(__file__), 'VERSION')).read().split('.'))
 
 BACKENDS = {
     'hg': 'Mercurial repository',
     'git': 'Git repository',
     'svn': 'Subversion repository',
 }
 
 CELERY_ENABLED = False
 CELERY_EAGER = False
 
 # link to config for pylons
 CONFIG = {}
 
 # Populated with the settings dictionary from application init in
 # rhodecode.conf.environment.load_pyramid_environment
 PYRAMID_SETTINGS = {}
 
 # Linked module for extensions
 EXTENSIONS = {}
 
 __version__ = ('.'.join((str(each) for each in VERSION[:3])))
-__dbversion__ = 64  # defines current db version for migrations
+__dbversion__ = 71  # defines current db version for migrations
 __platform__ = platform.system()
 __license__ = 'AGPLv3, and Commercial License'
 __author__ = 'RhodeCode GmbH'
 __url__ = 'https://code.rhodecode.com'
 
 is_windows = __platform__ in ['Windows']
 is_unix = not is_windows
 is_test = False
 disable_error_handler = False

rhodecode/api/__init__.py

@@ -1,510 +1,536 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2011-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import inspect
 import itertools
 import logging
 import types
+import fnmatch
 
 import decorator
 import venusian
 from collections import OrderedDict
 
 from pyramid.exceptions import ConfigurationError
 from pyramid.renderers import render
 from pyramid.response import Response
 from pyramid.httpexceptions import HTTPNotFound
 
 from rhodecode.api.exc import (
     JSONRPCBaseError, JSONRPCError, JSONRPCForbidden, JSONRPCValidationError)
 from rhodecode.lib.auth import AuthUser
 from rhodecode.lib.base import get_ip_addr
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.utils2 import safe_str
 from rhodecode.lib.plugins.utils import get_plugin_settings
 from rhodecode.model.db import User, UserApiKeys
 
 log = logging.getLogger(__name__)
 
 DEFAULT_RENDERER = 'jsonrpc_renderer'
 DEFAULT_URL = '/_admin/apiv2'
 
 
+def find_methods(jsonrpc_methods, pattern):
+    matches = OrderedDict()
+    if not isinstance(pattern, (list, tuple)):
+        pattern = [pattern]
+
+    for single_pattern in pattern:
+        for method_name, method in jsonrpc_methods.items():
+            if fnmatch.fnmatch(method_name, single_pattern):
+                matches[method_name] = method
+    return matches
+
+
 class ExtJsonRenderer(object):
     """
     Custom renderer that mkaes use of our ext_json lib
 
     """
 
     def __init__(self, serializer=json.dumps, **kw):
         """ Any keyword arguments will be passed to the ``serializer``
         function."""
         self.serializer = serializer
         self.kw = kw
 
     def __call__(self, info):
         """ Returns a plain JSON-encoded string with content-type
         ``application/json``. The content-type may be overridden by
         setting ``request.response.content_type``."""
 
         def _render(value, system):
             request = system.get('request')
             if request is not None:
                 response = request.response
                 ct = response.content_type
                 if ct == response.default_content_type:
                     response.content_type = 'application/json'
 
             return self.serializer(value, **self.kw)
 
         return _render
 
 
 def jsonrpc_response(request, result):
     rpc_id = getattr(request, 'rpc_id', None)
     response = request.response
 
     # store content_type before render is called
     ct = response.content_type
 
     ret_value = ''
     if rpc_id:
         ret_value = {
             'id': rpc_id,
             'result': result,
             'error': None,
         }
 
         # fetch deprecation warnings, and store it inside results
         deprecation = getattr(request, 'rpc_deprecation', None)
         if deprecation:
             ret_value['DEPRECATION_WARNING'] = deprecation
 
     raw_body = render(DEFAULT_RENDERER, ret_value, request=request)
     response.body = safe_str(raw_body, response.charset)
 
     if ct == response.default_content_type:
         response.content_type = 'application/json'
 
     return response
 
 
 def jsonrpc_error(request, message, retid=None, code=None):
     """
     Generate a Response object with a JSON-RPC error body
 
     :param code:
     :param retid:
     :param message:
     """
     err_dict = {'id': retid, 'result': None, 'error': message}
     body = render(DEFAULT_RENDERER, err_dict, request=request).encode('utf-8')
     return Response(
         body=body,
         status=code,
         content_type='application/json'
     )
 
 
 def exception_view(exc, request):
     rpc_id = getattr(request, 'rpc_id', None)
 
     fault_message = 'undefined error'
     if isinstance(exc, JSONRPCError):
         fault_message = exc.message
         log.debug('json-rpc error rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, JSONRPCValidationError):
         colander_exc = exc.colander_exception
         # TODO(marcink): think maybe of nicer way to serialize errors ?
         fault_message = colander_exc.asdict()
         log.debug('json-rpc error rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, JSONRPCForbidden):
         fault_message = 'Access was denied to this resource.'
         log.warning('json-rpc forbidden call rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, HTTPNotFound):
         method = request.rpc_method
         log.debug('json-rpc method `%s` not found in list of '
                   'api calls: %s, rpc_id:%s',
                   method, request.registry.jsonrpc_methods.keys(), rpc_id)
-        fault_message = "No such method: {}".format(method)
+
+        similar = 'none'
+        try:
+            similar_paterns = ['*{}*'.format(x) for x in method.split('_')]
+            similar_found = find_methods(
+                request.registry.jsonrpc_methods, similar_paterns)
+            similar = ', '.join(similar_found.keys()) or similar
+        except Exception:
+            # make the whole above block safe
+            pass
+
+        fault_message = "No such method: {}. Similar methods: {}".format(
+            method, similar)
 
     return jsonrpc_error(request, fault_message, rpc_id)
 
 
 def request_view(request):
     """
     Main request handling method. It handles all logic to call a specific
     exposed method
     """
 
     # check if we can find this session using api_key, get_by_auth_token
     # search not expired tokens only
 
     try:
         api_user = User.get_by_auth_token(request.rpc_api_key)
 
         if api_user is None:
             return jsonrpc_error(
                 request, retid=request.rpc_id, message='Invalid API KEY')
 
         if not api_user.active:
             return jsonrpc_error(
                 request, retid=request.rpc_id,
                 message='Request from this user not allowed')
 
         # check if we are allowed to use this IP
         auth_u = AuthUser(
             api_user.user_id, request.rpc_api_key, ip_addr=request.rpc_ip_addr)
         if not auth_u.ip_allowed:
             return jsonrpc_error(
                 request, retid=request.rpc_id,
                 message='Request from IP:%s not allowed' % (
                     request.rpc_ip_addr,))
         else:
             log.info('Access for IP:%s allowed' % (request.rpc_ip_addr,))
 
         # register our auth-user
         request.rpc_user = auth_u
 
         # now check if token is valid for API
-        role = UserApiKeys.ROLE_API
-        extra_auth_tokens = [
-            x.api_key for x in User.extra_valid_auth_tokens(api_user, role=role)]
-        active_tokens = [api_user.api_key] + extra_auth_tokens
+        auth_token = request.rpc_api_key
+        token_match = api_user.authenticate_by_token(
+            auth_token, roles=[UserApiKeys.ROLE_API])
+        invalid_token = not token_match
 
-        log.debug('Checking if API key has proper role')
-        if request.rpc_api_key not in active_tokens:
+        log.debug('Checking if API KEY is valid with proper role')
+        if invalid_token:
             return jsonrpc_error(
                 request, retid=request.rpc_id,
-                message='API KEY has bad role for an API call')
+                message='API KEY invalid or, has bad role for an API call')
 
-    except Exception as e:
+    except Exception:
         log.exception('Error on API AUTH')
         return jsonrpc_error(
             request, retid=request.rpc_id, message='Invalid API KEY')
 
     method = request.rpc_method
     func = request.registry.jsonrpc_methods[method]
 
     # now that we have a method, add request._req_params to
     # self.kargs and dispatch control to WGIController
     argspec = inspect.getargspec(func)
     arglist = argspec[0]
     defaults = map(type, argspec[3] or [])
     default_empty = types.NotImplementedType
 
     # kw arguments required by this method
     func_kwargs = dict(itertools.izip_longest(
         reversed(arglist), reversed(defaults), fillvalue=default_empty))
 
     # This attribute will need to be first param of a method that uses
     # api_key, which is translated to instance of user at that name
     user_var = 'apiuser'
     request_var = 'request'
 
     for arg in [user_var, request_var]:
         if arg not in arglist:
             return jsonrpc_error(
                 request,
                 retid=request.rpc_id,
                 message='This method [%s] does not support '
                         'required parameter `%s`' % (func.__name__, arg))
 
     # get our arglist and check if we provided them as args
     for arg, default in func_kwargs.items():
         if arg in [user_var, request_var]:
             # user_var and request_var are pre-hardcoded parameters and we
             # don't need to do any translation
             continue
 
         # skip the required param check if it's default value is
         # NotImplementedType (default_empty)
         if default == default_empty and arg not in request.rpc_params:
             return jsonrpc_error(
                 request,
                 retid=request.rpc_id,
                 message=('Missing non optional `%s` arg in JSON DATA' % arg)
             )
 
     # sanitize extra passed arguments
     for k in request.rpc_params.keys()[:]:
         if k not in func_kwargs:
             del request.rpc_params[k]
 
     call_params = request.rpc_params
     call_params.update({
         'request': request,
         'apiuser': auth_u
     })
     try:
         ret_value = func(**call_params)
         return jsonrpc_response(request, ret_value)
     except JSONRPCBaseError:
         raise
     except Exception:
         log.exception('Unhandled exception occurred on api call: %s', func)
         return jsonrpc_error(request, retid=request.rpc_id,
                              message='Internal server error')
 
 
 def setup_request(request):
     """
     Parse a JSON-RPC request body. It's used inside the predicates method
     to validate and bootstrap requests for usage in rpc calls.
 
     We need to raise JSONRPCError here if we want to return some errors back to
     user.
     """
 
     log.debug('Executing setup request: %r', request)
     request.rpc_ip_addr = get_ip_addr(request.environ)
     # TODO(marcink): deprecate GET at some point
     if request.method not in ['POST', 'GET']:
         log.debug('unsupported request method "%s"', request.method)
         raise JSONRPCError(
             'unsupported request method "%s". Please use POST' % request.method)
 
     if 'CONTENT_LENGTH' not in request.environ:
         log.debug("No Content-Length")
         raise JSONRPCError("Empty body, No Content-Length in request")
 
     else:
         length = request.environ['CONTENT_LENGTH']
         log.debug('Content-Length: %s', length)
 
         if length == 0:
             log.debug("Content-Length is 0")
             raise JSONRPCError("Content-Length is 0")
 
     raw_body = request.body
     try:
         json_body = json.loads(raw_body)
     except ValueError as e:
         # catch JSON errors Here
         raise JSONRPCError("JSON parse error ERR:%s RAW:%r" % (e, raw_body))
 
     request.rpc_id = json_body.get('id')
     request.rpc_method = json_body.get('method')
 
     # check required base parameters
     try:
         api_key = json_body.get('api_key')
         if not api_key:
             api_key = json_body.get('auth_token')
 
         if not api_key:
             raise KeyError('api_key or auth_token')
 
         # TODO(marcink): support passing in token in request header
 
         request.rpc_api_key = api_key
         request.rpc_id = json_body['id']
         request.rpc_method = json_body['method']
         request.rpc_params = json_body['args'] \
             if isinstance(json_body['args'], dict) else {}
 
         log.debug(
             'method: %s, params: %s' % (request.rpc_method, request.rpc_params))
     except KeyError as e:
         raise JSONRPCError('Incorrect JSON data. Missing %s' % e)
 
     log.debug('setup complete, now handling method:%s rpcid:%s',
               request.rpc_method, request.rpc_id, )
 
 
 class RoutePredicate(object):
     def __init__(self, val, config):
         self.val = val
 
     def text(self):
         return 'jsonrpc route = %s' % self.val
 
     phash = text
 
     def __call__(self, info, request):
         if self.val:
             # potentially setup and bootstrap our call
             setup_request(request)
 
             # Always return True so that even if it isn't a valid RPC it
             # will fall through to the underlaying handlers like notfound_view
             return True
 
 
 class NotFoundPredicate(object):
     def __init__(self, val, config):
         self.val = val
+        self.methods = config.registry.jsonrpc_methods
 
     def text(self):
-        return 'jsonrpc method not found = %s' % self.val
+        return 'jsonrpc method not found = {}.'.format(self.val)
 
     phash = text
 
     def __call__(self, info, request):
         return hasattr(request, 'rpc_method')
 
 
 class MethodPredicate(object):
     def __init__(self, val, config):
         self.method = val
 
     def text(self):
         return 'jsonrpc method = %s' % self.method
 
     phash = text
 
     def __call__(self, context, request):
         # we need to explicitly return False here, so pyramid doesn't try to
         # execute our view directly. We need our main handler to execute things
         return getattr(request, 'rpc_method') == self.method
 
 
 def add_jsonrpc_method(config, view, **kwargs):
     # pop the method name
     method = kwargs.pop('method', None)
 
     if method is None:
         raise ConfigurationError(
             'Cannot register a JSON-RPC method without specifying the '
             '"method"')
 
     # we define custom predicate, to enable to detect conflicting methods,
     # those predicates are kind of "translation" from the decorator variables
     # to internal predicates names
 
     kwargs['jsonrpc_method'] = method
 
     # register our view into global view store for validation
     config.registry.jsonrpc_methods[method] = view
 
     # we're using our main request_view handler, here, so each method
     # has a unified handler for itself
     config.add_view(request_view, route_name='apiv2', **kwargs)
 
 
 class jsonrpc_method(object):
     """
     decorator that works similar to @add_view_config decorator,
     but tailored for our JSON RPC
     """
 
     venusian = venusian  # for testing injection
 
     def __init__(self, method=None, **kwargs):
         self.method = method
         self.kwargs = kwargs
 
     def __call__(self, wrapped):
         kwargs = self.kwargs.copy()
         kwargs['method'] = self.method or wrapped.__name__
         depth = kwargs.pop('_depth', 0)
 
         def callback(context, name, ob):
             config = context.config.with_package(info.module)
             config.add_jsonrpc_method(view=ob, **kwargs)
 
         info = venusian.attach(wrapped, callback, category='pyramid',
                                depth=depth + 1)
         if info.scope == 'class':
             # ensure that attr is set if decorating a class method
             kwargs.setdefault('attr', wrapped.__name__)
 
         kwargs['_info'] = info.codeinfo  # fbo action_method
         return wrapped
 
 
 class jsonrpc_deprecated_method(object):
     """
     Marks method as deprecated, adds log.warning, and inject special key to
     the request variable to mark method as deprecated.
     Also injects special docstring that extract_docs will catch to mark
     method as deprecated.
 
     :param use_method: specify which method should be used instead of
         the decorated one
 
     Use like::
 
         @jsonrpc_method()
         @jsonrpc_deprecated_method(use_method='new_func', deprecated_at_version='3.0.0')
         def old_func(request, apiuser, arg1, arg2):
             ...
     """
 
     def __init__(self, use_method, deprecated_at_version):
         self.use_method = use_method
         self.deprecated_at_version = deprecated_at_version
         self.deprecated_msg = ''
 
     def __call__(self, func):
         self.deprecated_msg = 'Please use method `{method}` instead.'.format(
             method=self.use_method)
 
         docstring = """\n
         .. deprecated:: {version}
 
            {deprecation_message}
 
         {original_docstring}
         """
         func.__doc__ = docstring.format(
             version=self.deprecated_at_version,
             deprecation_message=self.deprecated_msg,
             original_docstring=func.__doc__)
         return decorator.decorator(self.__wrapper, func)
 
     def __wrapper(self, func, *fargs, **fkwargs):
         log.warning('DEPRECATED API CALL on function %s, please '
                     'use `%s` instead', func, self.use_method)
         # alter function docstring to mark as deprecated, this is picked up
         # via fabric file that generates API DOC.
         result = func(*fargs, **fkwargs)
 
         request = fargs[0]
         request.rpc_deprecation = 'DEPRECATED METHOD ' + self.deprecated_msg
         return result
 
 
 def includeme(config):
     plugin_module = 'rhodecode.api'
     plugin_settings = get_plugin_settings(
         plugin_module, config.registry.settings)
 
     if not hasattr(config.registry, 'jsonrpc_methods'):
         config.registry.jsonrpc_methods = OrderedDict()
 
     # match filter by given method only
     config.add_view_predicate('jsonrpc_method', MethodPredicate)
 
     config.add_renderer(DEFAULT_RENDERER, ExtJsonRenderer(
         serializer=json.dumps, indent=4))
     config.add_directive('add_jsonrpc_method', add_jsonrpc_method)
 
     config.add_route_predicate(
         'jsonrpc_call', RoutePredicate)
 
     config.add_route(
         'apiv2', plugin_settings.get('url', DEFAULT_URL), jsonrpc_call=True)
 
     config.scan(plugin_module, ignore='rhodecode.api.tests')
     # register some exception handling view
     config.add_view(exception_view, context=JSONRPCBaseError)
     config.add_view_predicate('jsonrpc_method_not_found', NotFoundPredicate)
     config.add_notfound_view(exception_view, jsonrpc_method_not_found=True)

rhodecode/api/tests/conftest.py

@@ -1,42 +1,52 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import pytest
 
 from rhodecode.model.meta import Session
 from rhodecode.model.user import UserModel
+from rhodecode.model.auth_token import AuthTokenModel
 from rhodecode.tests import TEST_USER_ADMIN_LOGIN
 
 
 @pytest.fixture(scope="class")
 def testuser_api(request, pylonsapp):
     cls = request.cls
+
+    # ADMIN USER
     cls.usr = UserModel().get_by_username(TEST_USER_ADMIN_LOGIN)
     cls.apikey = cls.usr.api_key
+
+    # REGULAR USER
     cls.test_user = UserModel().create_or_update(
         username='test-api',
         password='test',
         email='[e-mail]',
         firstname='first',
         lastname='last'
     )
+    # create TOKEN for user, if he doesn't have one
+    if not cls.test_user.api_key:
+        AuthTokenModel().create(
+            user=cls.test_user, description='TEST_USER_TOKEN')
+
     Session().commit()
     cls.TEST_USER_LOGIN = cls.test_user.username
     cls.apikey_regular = cls.test_user.api_key

rhodecode/api/tests/test_api.py

@@ -1,125 +1,131 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import pytest
 
 from rhodecode.api.utils import Optional, OAttr
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_error, assert_ok)
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestApi(object):
     maxDiff = None
 
     def test_Optional_object(self):
 
         option1 = Optional(None)
         assert '<Optional:%s>' % (None,) == repr(option1)
         assert option1() is None
 
         assert 1 == Optional.extract(Optional(1))
         assert 'example' == Optional.extract('example')
 
     def test_Optional_OAttr(self):
         option1 = Optional(OAttr('apiuser'))
         assert 'apiuser' == Optional.extract(option1)
 
     def test_OAttr_object(self):
         oattr1 = OAttr('apiuser')
         assert '<OptionalAttr:apiuser>' == repr(oattr1)
         assert oattr1() == oattr1
 
     def test_api_wrong_key(self):
         id_, params = build_data('trololo', 'get_user')
         response = api_call(self.app, params)
 
         expected = 'Invalid API KEY'
         assert_error(id_, expected, given=response.body)
 
     def test_api_missing_non_optional_param(self):
         id_, params = build_data(self.apikey, 'get_repo')
         response = api_call(self.app, params)
 
         expected = 'Missing non optional `repoid` arg in JSON DATA'
         assert_error(id_, expected, given=response.body)
 
     def test_api_missing_non_optional_param_args_null(self):
         id_, params = build_data(self.apikey, 'get_repo')
         params = params.replace('"args": {}', '"args": null')
         response = api_call(self.app, params)
 
         expected = 'Missing non optional `repoid` arg in JSON DATA'
         assert_error(id_, expected, given=response.body)
 
     def test_api_missing_non_optional_param_args_bad(self):
         id_, params = build_data(self.apikey, 'get_repo')
         params = params.replace('"args": {}', '"args": 1')
         response = api_call(self.app, params)
 
         expected = 'Missing non optional `repoid` arg in JSON DATA'
         assert_error(id_, expected, given=response.body)
 
     def test_api_non_existing_method(self, request):
         id_, params = build_data(self.apikey, 'not_existing', args='xx')
         response = api_call(self.app, params)
-        expected = 'No such method: not_existing'
+        expected = 'No such method: not_existing. Similar methods: none'
+        assert_error(id_, expected, given=response.body)
+
+    def test_api_non_existing_method_have_similar(self, request):
+        id_, params = build_data(self.apikey, 'comment', args='xx')
+        response = api_call(self.app, params)
+        expected = 'No such method: comment. Similar methods: changeset_comment, comment_pull_request, comment_commit'
         assert_error(id_, expected, given=response.body)
 
     def test_api_disabled_user(self, request):
 
         def set_active(active):
             from rhodecode.model.db import Session, User
             user = User.get_by_auth_token(self.apikey)
             user.active = active
             Session().add(user)
             Session().commit()
 
         request.addfinalizer(lambda: set_active(True))
 
         set_active(False)
         id_, params = build_data(self.apikey, 'test', args='xx')
         response = api_call(self.app, params)
         expected = 'Request from this user not allowed'
         assert_error(id_, expected, given=response.body)
 
     def test_api_args_is_null(self):
         __, params = build_data(self.apikey, 'get_users', )
         params = params.replace('"args": {}', '"args": null')
         response = api_call(self.app, params)
         assert response.status == '200 OK'
 
     def test_api_args_is_bad(self):
         __, params = build_data(self.apikey, 'get_users', )
         params = params.replace('"args": {}', '"args": 1')
         response = api_call(self.app, params)
         assert response.status == '200 OK'
 
     def test_api_args_different_args(self):
         import string
         expected = {
             'ascii_letters': string.ascii_letters,
             'ws': string.whitespace,
             'printables': string.printable
         }
         id_, params = build_data(self.apikey, 'test', args=expected)
         response = api_call(self.app, params)
         assert response.status == '200 OK'
         assert_ok(id_, expected, response.body)

rhodecode/api/tests/test_close_pull_request.py

@@ -1,111 +1,112 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import pytest
 
 from rhodecode.model.db import UserLog
 from rhodecode.model.pull_request import PullRequestModel
 from rhodecode.tests import TEST_USER_ADMIN_LOGIN
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_error, assert_ok)
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestClosePullRequest(object):
+
     @pytest.mark.backends("git", "hg")
     def test_api_close_pull_request(self, pr_util):
         pull_request = pr_util.create_pull_request()
         pull_request_id = pull_request.pull_request_id
         author = pull_request.user_id
         repo = pull_request.target_repo.repo_id
         id_, params = build_data(
             self.apikey, 'close_pull_request',
             repoid=pull_request.target_repo.repo_name,
             pullrequestid=pull_request.pull_request_id)
         response = api_call(self.app, params)
         expected = {
             'pull_request_id': pull_request_id,
             'closed': True,
         }
         assert_ok(id_, expected, response.body)
         action = 'user_closed_pull_request:%d' % pull_request_id
         journal = UserLog.query()\
             .filter(UserLog.user_id == author)\
             .filter(UserLog.repository_id == repo)\
             .filter(UserLog.action == action)\
             .all()
         assert len(journal) == 1
 
     @pytest.mark.backends("git", "hg")
     def test_api_close_pull_request_already_closed_error(self, pr_util):
         pull_request = pr_util.create_pull_request()
         pull_request_id = pull_request.pull_request_id
         pull_request_repo = pull_request.target_repo.repo_name
         PullRequestModel().close_pull_request(
             pull_request, pull_request.author)
         id_, params = build_data(
             self.apikey, 'close_pull_request',
             repoid=pull_request_repo, pullrequestid=pull_request_id)
         response = api_call(self.app, params)
 
         expected = 'pull request `%s` is already closed' % pull_request_id
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.backends("git", "hg")
     def test_api_close_pull_request_repo_error(self):
         id_, params = build_data(
             self.apikey, 'close_pull_request',
             repoid=666, pullrequestid=1)
         response = api_call(self.app, params)
 
         expected = 'repository `666` does not exist'
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.backends("git", "hg")
     def test_api_close_pull_request_non_admin_with_userid_error(self,
                                                                 pr_util):
         pull_request = pr_util.create_pull_request()
         id_, params = build_data(
             self.apikey_regular, 'close_pull_request',
             repoid=pull_request.target_repo.repo_name,
             pullrequestid=pull_request.pull_request_id,
             userid=TEST_USER_ADMIN_LOGIN)
         response = api_call(self.app, params)
 
         expected = 'userid is not the same as your user'
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.backends("git", "hg")
     def test_api_close_pull_request_no_perms_to_close(
             self, user_util, pr_util):
         user = user_util.create_user()
         pull_request = pr_util.create_pull_request()
 
         id_, params = build_data(
             user.api_key, 'close_pull_request',
             repoid=pull_request.target_repo.repo_name,
             pullrequestid=pull_request.pull_request_id,)
         response = api_call(self.app, params)
 
         expected = ('pull request `%s` close failed, '
                     'no permission to close.') % pull_request.pull_request_id
 
         response_json = response.json['error']
         assert response_json == expected

rhodecode/api/tests/test_delete_repo.py

@@ -1,73 +1,72 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import mock
 import pytest
 
 from rhodecode.model.repo import RepoModel
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_error, assert_ok, crash)
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestApiDeleteRepo(object):
     def test_api_delete_repo(self, backend):
         repo = backend.create_repo()
 
         id_, params = build_data(
             self.apikey, 'delete_repo', repoid=repo.repo_name, )
         response = api_call(self.app, params)
 
         expected = {
             'msg': 'Deleted repository `%s`' % (repo.repo_name,),
             'success': True
         }
         assert_ok(id_, expected, given=response.body)
 
     def test_api_delete_repo_by_non_admin(self, backend, user_regular):
         repo = backend.create_repo(cur_user=user_regular.username)
         id_, params = build_data(
             user_regular.api_key, 'delete_repo', repoid=repo.repo_name, )
         response = api_call(self.app, params)
 
         expected = {
             'msg': 'Deleted repository `%s`' % (repo.repo_name,),
             'success': True
         }
         assert_ok(id_, expected, given=response.body)
 
-    def test_api_delete_repo_by_non_admin_no_permission(
-            self, backend, user_regular):
+    def test_api_delete_repo_by_non_admin_no_permission(self, backend):
         repo = backend.create_repo()
         id_, params = build_data(
-            user_regular.api_key, 'delete_repo', repoid=repo.repo_name, )
+            self.apikey_regular, 'delete_repo', repoid=repo.repo_name, )
         response = api_call(self.app, params)
         expected = 'repository `%s` does not exist' % (repo.repo_name)
         assert_error(id_, expected, given=response.body)
 
     def test_api_delete_repo_exception_occurred(self, backend):
         repo = backend.create_repo()
         id_, params = build_data(
             self.apikey, 'delete_repo', repoid=repo.repo_name, )
         with mock.patch.object(RepoModel, 'delete', crash):
             response = api_call(self.app, params)
         expected = 'failed to delete repository `%s`' % (
             repo.repo_name,)
         assert_error(id_, expected, given=response.body)

rhodecode/api/tests/test_update_user_group.py

@@ -1,108 +1,110 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import mock
 import pytest
 
 from rhodecode.model.user import UserModel
 from rhodecode.model.user_group import UserGroupModel
-from rhodecode.tests import TEST_USER_REGULAR_LOGIN
+from rhodecode.tests import TEST_USER_ADMIN_EMAIL
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_error, assert_ok, crash, jsonify)
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestUpdateUserGroup(object):
     @pytest.mark.parametrize("changing_attr, updates", [
         ('group_name', {'group_name': 'new_group_name'}),
         ('group_name', {'group_name': 'test_group_for_update'}),
-        ('owner', {'owner': TEST_USER_REGULAR_LOGIN}),
+        # ('owner', {'owner': TEST_USER_REGULAR_LOGIN}),
+        ('owner_email', {'owner_email': TEST_USER_ADMIN_EMAIL}),
         ('active', {'active': False}),
         ('active', {'active': True})
     ])
     def test_api_update_user_group(self, changing_attr, updates, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         expected_api_data = user_group.get_api_data()
         expected_api_data.update(updates)
 
         id_, params = build_data(
             self.apikey, 'update_user_group', usergroupid=group_name,
             **updates)
         response = api_call(self.app, params)
 
         expected = {
             'msg': 'updated user group ID:%s %s' % (
                 user_group.users_group_id, user_group.users_group_name),
             'user_group': jsonify(expected_api_data)
         }
         assert_ok(id_, expected, given=response.body)
 
     @pytest.mark.parametrize("changing_attr, updates", [
         # TODO: mikhail: decide if we need to test against the commented params
         # ('group_name', {'group_name': 'new_group_name'}),
         # ('group_name', {'group_name': 'test_group_for_update'}),
-        ('owner', {'owner': TEST_USER_REGULAR_LOGIN}),
+        # ('owner', {'owner': TEST_USER_REGULAR_LOGIN}),
+        ('owner_email', {'owner_email': TEST_USER_ADMIN_EMAIL}),
         ('active', {'active': False}),
         ('active', {'active': True})
     ])
     def test_api_update_user_group_regular_user(
             self, changing_attr, updates, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         expected_api_data = user_group.get_api_data()
         expected_api_data.update(updates)
 
 
         # grant permission to this user
         user = UserModel().get_by_username(self.TEST_USER_LOGIN)
 
         user_util.grant_user_permission_to_user_group(
             user_group, user, 'usergroup.admin')
         id_, params = build_data(
             self.apikey_regular, 'update_user_group',
             usergroupid=group_name, **updates)
         response = api_call(self.app, params)
         expected = {
             'msg': 'updated user group ID:%s %s' % (
                 user_group.users_group_id, user_group.users_group_name),
             'user_group': jsonify(expected_api_data)
         }
         assert_ok(id_, expected, given=response.body)
 
     def test_api_update_user_group_regular_user_no_permission(self, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         id_, params = build_data(
             self.apikey_regular, 'update_user_group', usergroupid=group_name)
         response = api_call(self.app, params)
 
         expected = 'user group `%s` does not exist' % (group_name)
         assert_error(id_, expected, given=response.body)
 
     @mock.patch.object(UserGroupModel, 'update', crash)
     def test_api_update_user_group_exception_occurred(self, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         id_, params = build_data(
             self.apikey, 'update_user_group', usergroupid=group_name)
         response = api_call(self.app, params)
         expected = 'failed to update user group `%s`' % (group_name,)
         assert_error(id_, expected, given=response.body)

rhodecode/api/tests/test_utils.py

@@ -1,268 +1,294 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import pytest
 from mock import Mock, patch
 
 from rhodecode.api import utils
 from rhodecode.api import JSONRPCError
 from rhodecode.lib.vcs.exceptions import RepositoryError
 
 
 class TestGetCommitOrError(object):
     def setup(self):
         self.commit_hash = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa10'
 
     @pytest.mark.parametrize("ref", ['ref', '12345', 'a:b:c:d', 'branch:name'])
     def test_ref_cannot_be_parsed(self, ref):
         repo = Mock()
         with pytest.raises(JSONRPCError) as excinfo:
             utils.get_commit_or_error(ref, repo)
         expected_message = (
             'Ref `{ref}` given in a wrong format. Please check the API'
             ' documentation for more details'.format(ref=ref)
         )
         assert excinfo.value.message == expected_message
 
     def test_success_with_hash_specified(self):
         repo = Mock()
         ref_type = 'branch'
         ref = '{}:master:{}'.format(ref_type, self.commit_hash)
 
         with patch('rhodecode.api.utils.get_commit_from_ref_name') as get_commit:
             result = utils.get_commit_or_error(ref, repo)
             get_commit.assert_called_once_with(
                 repo, self.commit_hash)
             assert result == get_commit()
 
     def test_raises_an_error_when_commit_not_found(self):
         repo = Mock()
         ref = 'branch:master:{}'.format(self.commit_hash)
 
         with patch('rhodecode.api.utils.get_commit_from_ref_name') as get_commit:
             get_commit.side_effect = RepositoryError('Commit not found')
             with pytest.raises(JSONRPCError) as excinfo:
                 utils.get_commit_or_error(ref, repo)
             expected_message = 'Ref `{}` does not exist'.format(ref)
             assert excinfo.value.message == expected_message
 
 
 class TestResolveRefOrError(object):
     def setup(self):
         self.commit_hash = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa10'
 
     def test_success_with_no_hash_specified(self):
         repo = Mock()
         ref_type = 'branch'
         ref_name = 'master'
         ref = '{}:{}'.format(ref_type, ref_name)
 
         with patch('rhodecode.api.utils._get_ref_hash') \
                 as _get_ref_hash:
             _get_ref_hash.return_value = self.commit_hash
             result = utils.resolve_ref_or_error(ref, repo)
             _get_ref_hash.assert_called_once_with(repo, ref_type, ref_name)
             assert result == '{}:{}'.format(ref, self.commit_hash)
 
     def test_non_supported_refs(self):
         repo = Mock()
         ref = 'ancestor:ref'
         with pytest.raises(JSONRPCError) as excinfo:
             utils.resolve_ref_or_error(ref, repo)
         expected_message = 'The specified ancestor `ref` does not exist'
         assert excinfo.value.message == expected_message
 
     def test_branch_is_not_found(self):
         repo = Mock()
         ref = 'branch:non-existing-one'
         with patch('rhodecode.api.utils._get_ref_hash')\
                 as _get_ref_hash:
             _get_ref_hash.side_effect = KeyError()
             with pytest.raises(JSONRPCError) as excinfo:
                 utils.resolve_ref_or_error(ref, repo)
         expected_message = (
             'The specified branch `non-existing-one` does not exist')
         assert excinfo.value.message == expected_message
 
     def test_bookmark_is_not_found(self):
         repo = Mock()
         ref = 'bookmark:non-existing-one'
         with patch('rhodecode.api.utils._get_ref_hash')\
                 as _get_ref_hash:
             _get_ref_hash.side_effect = KeyError()
             with pytest.raises(JSONRPCError) as excinfo:
                 utils.resolve_ref_or_error(ref, repo)
         expected_message = (
             'The specified bookmark `non-existing-one` does not exist')
         assert excinfo.value.message == expected_message
 
     @pytest.mark.parametrize("ref", ['ref', '12345', 'a:b:c:d'])
     def test_ref_cannot_be_parsed(self, ref):
         repo = Mock()
         with pytest.raises(JSONRPCError) as excinfo:
             utils.resolve_ref_or_error(ref, repo)
         expected_message = (
             'Ref `{ref}` given in a wrong format. Please check the API'
             ' documentation for more details'.format(ref=ref)
         )
         assert excinfo.value.message == expected_message
 
 
 class TestGetRefHash(object):
     def setup(self):
         self.commit_hash = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa10'
         self.bookmark_name = 'test-bookmark'
 
     @pytest.mark.parametrize("alias, branch_name", [
         ("git", "master"),
         ("hg", "default")
     ])
     def test_returns_hash_by_branch_name(self, alias, branch_name):
         with patch('rhodecode.model.db.Repository') as repo:
             repo.scm_instance().alias = alias
             repo.scm_instance().branches = {branch_name: self.commit_hash}
             result_hash = utils._get_ref_hash(repo, 'branch', branch_name)
             assert result_hash == self.commit_hash
 
     @pytest.mark.parametrize("alias, branch_name", [
         ("git", "master"),
         ("hg", "default")
     ])
     def test_raises_error_when_branch_is_not_found(self, alias, branch_name):
         with patch('rhodecode.model.db.Repository') as repo:
             repo.scm_instance().alias = alias
             repo.scm_instance().branches = {}
             with pytest.raises(KeyError):
                 utils._get_ref_hash(repo, 'branch', branch_name)
 
     def test_returns_hash_when_bookmark_is_specified_for_hg(self):
         with patch('rhodecode.model.db.Repository') as repo:
             repo.scm_instance().alias = 'hg'
             repo.scm_instance().bookmarks = {
                 self.bookmark_name: self.commit_hash}
             result_hash = utils._get_ref_hash(
                 repo, 'bookmark', self.bookmark_name)
             assert result_hash == self.commit_hash
 
     def test_raises_error_when_bookmark_is_not_found_in_hg_repo(self):
         with patch('rhodecode.model.db.Repository') as repo:
             repo.scm_instance().alias = 'hg'
             repo.scm_instance().bookmarks = {}
             with pytest.raises(KeyError):
                 utils._get_ref_hash(repo, 'bookmark', self.bookmark_name)
 
     def test_raises_error_when_bookmark_is_specified_for_git(self):
         with patch('rhodecode.model.db.Repository') as repo:
             repo.scm_instance().alias = 'git'
             repo.scm_instance().bookmarks = {
                 self.bookmark_name: self.commit_hash}
             with pytest.raises(ValueError):
                 utils._get_ref_hash(repo, 'bookmark', self.bookmark_name)
 
 
 class TestUserByNameOrError(object):
     def test_user_found_by_id(self):
         fake_user = Mock(id=123)
+
+        patcher = patch('rhodecode.model.user.UserModel.get_user')
+        with patcher as get_user:
+            get_user.return_value = fake_user
+
+            patcher = patch('rhodecode.model.user.UserModel.get_by_username')
+            with patcher as get_by_username:
+                result = utils.get_user_or_error(123)
+                assert result == fake_user
+
+    def test_user_not_found_by_id_as_str(self):
+        fake_user = Mock(id=123)
+
         patcher = patch('rhodecode.model.user.UserModel.get_user')
         with patcher as get_user:
             get_user.return_value = fake_user
-            result = utils.get_user_or_error('123')
-            assert result == fake_user
+            patcher = patch('rhodecode.model.user.UserModel.get_by_username')
+            with patcher as get_by_username:
+                get_by_username.return_value = None
+
+                with pytest.raises(JSONRPCError):
+                    utils.get_user_or_error('123')
 
     def test_user_found_by_name(self):
         fake_user = Mock(id=123)
-        patcher = patch('rhodecode.model.user.UserModel.get_by_username')
-        with patcher as get_by_username:
-            get_by_username.return_value = fake_user
-            result = utils.get_user_or_error('test')
-            assert result == fake_user
+
+        patcher = patch('rhodecode.model.user.UserModel.get_user')
+        with patcher as get_user:
+            get_user.return_value = None
+
+            patcher = patch('rhodecode.model.user.UserModel.get_by_username')
+            with patcher as get_by_username:
+                get_by_username.return_value = fake_user
+
+                result = utils.get_user_or_error('test')
+                assert result == fake_user
 
     def test_user_not_found_by_id(self):
         patcher = patch('rhodecode.model.user.UserModel.get_user')
         with patcher as get_user:
             get_user.return_value = None
-            with pytest.raises(JSONRPCError) as excinfo:
-                utils.get_user_or_error('123')
+            patcher = patch('rhodecode.model.user.UserModel.get_by_username')
+            with patcher as get_by_username:
+                get_by_username.return_value = None
 
-        expected_message = 'user `123` does not exist'
-        assert excinfo.value.message == expected_message
+                with pytest.raises(JSONRPCError) as excinfo:
+                    utils.get_user_or_error(123)
+
+                expected_message = 'user `123` does not exist'
+                assert excinfo.value.message == expected_message
 
     def test_user_not_found_by_name(self):
         patcher = patch('rhodecode.model.user.UserModel.get_by_username')
         with patcher as get_by_username:
             get_by_username.return_value = None
             with pytest.raises(JSONRPCError) as excinfo:
                 utils.get_user_or_error('test')
 
         expected_message = 'user `test` does not exist'
         assert excinfo.value.message == expected_message
 
 
-class TestGetCommitDict:
-
+class TestGetCommitDict(object):
     @pytest.mark.parametrize('filename, expected', [
         (b'sp\xc3\xa4cial', u'sp\xe4cial'),
         (b'sp\xa4cial', u'sp\ufffdcial'),
     ])
     def test_decodes_filenames_to_unicode(self, filename, expected):
         result = utils._get_commit_dict(filename=filename, op='A')
         assert result['filename'] == expected
 
 
 class TestRepoAccess(object):
     def setup_method(self, method):
 
         self.admin_perm_patch = patch(
             'rhodecode.api.utils.HasPermissionAnyApi')
         self.repo_perm_patch = patch(
             'rhodecode.api.utils.HasRepoPermissionAnyApi')
 
     def test_has_superadmin_permission_checks_for_admin(self):
         admin_mock = Mock()
         with self.admin_perm_patch as amock:
             amock.return_value = admin_mock
             assert utils.has_superadmin_permission('fake_user')
             amock.assert_called_once_with('hg.admin')
 
         admin_mock.assert_called_once_with(user='fake_user')
 
     def test_has_repo_permissions_checks_for_repo_access(self):
         repo_mock = Mock()
         fake_repo = Mock()
         with self.repo_perm_patch as rmock:
             rmock.return_value = repo_mock
             assert utils.validate_repo_permissions(
                 'fake_user', 'fake_repo_id', fake_repo,
                 ['perm1', 'perm2'])
             rmock.assert_called_once_with(*['perm1', 'perm2'])
 
         repo_mock.assert_called_once_with(
             user='fake_user', repo_name=fake_repo.repo_name)
 
     def test_has_repo_permissions_raises_not_found(self):
         repo_mock = Mock(return_value=False)
         fake_repo = Mock()
         with self.repo_perm_patch as rmock:
             rmock.return_value = repo_mock
             with pytest.raises(JSONRPCError) as excinfo:
                 utils.validate_repo_permissions(
                     'fake_user', 'fake_repo_id', fake_repo, 'perms')
                 assert 'fake_repo_id' in excinfo

rhodecode/api/utils.py

@@ -1,412 +1,442 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2014-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 JSON RPC utils
 """
 
 import collections
 import logging
 
 from rhodecode.api.exc import JSONRPCError
 from rhodecode.lib.auth import (
     HasPermissionAnyApi, HasRepoPermissionAnyApi, HasRepoGroupPermissionAnyApi)
 from rhodecode.lib.utils import safe_unicode
 from rhodecode.lib.vcs.exceptions import RepositoryError
 from rhodecode.controllers.utils import get_commit_from_ref_name
 from rhodecode.lib.utils2 import str2bool
 
 log = logging.getLogger(__name__)
 
 
 class OAttr(object):
     """
     Special Option that defines other attribute, and can default to them
 
     Example::
 
         def test(apiuser, userid=Optional(OAttr('apiuser')):
             user = Optional.extract(userid, evaluate_locals=local())
             #if we pass in userid, we get it, else it will default to apiuser
             #attribute
     """
 
     def __init__(self, attr_name):
         self.attr_name = attr_name
 
     def __repr__(self):
         return '<OptionalAttr:%s>' % self.attr_name
 
     def __call__(self):
         return self
 
 
 class Optional(object):
     """
     Defines an optional parameter::
 
         param = param.getval() if isinstance(param, Optional) else param
         param = param() if isinstance(param, Optional) else param
 
     is equivalent of::
 
         param = Optional.extract(param)
 
     """
 
     def __init__(self, type_):
         self.type_ = type_
 
     def __repr__(self):
         return '<Optional:%s>' % self.type_.__repr__()
 
     def __call__(self):
         return self.getval()
 
     def getval(self, evaluate_locals=None):
         """
         returns value from this Optional instance
         """
         if isinstance(self.type_, OAttr):
             param_name = self.type_.attr_name
             if evaluate_locals:
                 return evaluate_locals[param_name]
             # use params name
             return param_name
         return self.type_
 
     @classmethod
     def extract(cls, val, evaluate_locals=None, binary=None):
         """
         Extracts value from Optional() instance
 
         :param val:
         :return: original value if it's not Optional instance else
             value of instance
         """
         if isinstance(val, cls):
             val = val.getval(evaluate_locals)
 
         if binary:
             val = str2bool(val)
 
         return val
 
 
 def parse_args(cli_args, key_prefix=''):
     from rhodecode.lib.utils2 import (escape_split)
     kwargs = collections.defaultdict(dict)
     for el in escape_split(cli_args, ','):
         kv = escape_split(el, '=', 1)
         if len(kv) == 2:
             k, v = kv
             kwargs[key_prefix + k] = v
     return kwargs
 
 
 def get_origin(obj):
     """
     Get origin of permission from object.
 
     :param obj:
     """
     origin = 'permission'
 
     if getattr(obj, 'owner_row', '') and getattr(obj, 'admin_row', ''):
         # admin and owner case, maybe we should use dual string ?
         origin = 'owner'
     elif getattr(obj, 'owner_row', ''):
         origin = 'owner'
     elif getattr(obj, 'admin_row', ''):
         origin = 'super-admin'
     return origin
 
 
 def store_update(updates, attr, name):
     """
     Stores param in updates dict if it's not instance of Optional
     allows easy updates of passed in params
     """
     if not isinstance(attr, Optional):
         updates[name] = attr
 
 
 def has_superadmin_permission(apiuser):
     """
     Return True if apiuser is admin or return False
 
     :param apiuser:
     """
     if HasPermissionAnyApi('hg.admin')(user=apiuser):
         return True
     return False
 
 
 def validate_repo_permissions(apiuser, repoid, repo, perms):
     """
     Raise JsonRPCError if apiuser is not authorized or return True
 
     :param apiuser:
     :param repoid:
     :param repo:
     :param perms:
     """
     if not HasRepoPermissionAnyApi(*perms)(
             user=apiuser, repo_name=repo.repo_name):
         raise JSONRPCError(
             'repository `%s` does not exist' % repoid)
 
     return True
 
 
 def validate_repo_group_permissions(apiuser, repogroupid, repo_group, perms):
     """
     Raise JsonRPCError if apiuser is not authorized or return True
 
     :param apiuser:
     :param repogroupid: just the id of repository group
     :param repo_group: instance of repo_group
     :param perms:
     """
     if not HasRepoGroupPermissionAnyApi(*perms)(
             user=apiuser, group_name=repo_group.group_name):
         raise JSONRPCError(
             'repository group `%s` does not exist' % repogroupid)
 
     return True
 
 
 def validate_set_owner_permissions(apiuser, owner):
     if isinstance(owner, Optional):
         owner = get_user_or_error(apiuser.user_id)
     else:
         if has_superadmin_permission(apiuser):
             owner = get_user_or_error(owner)
         else:
             # forbid setting owner for non-admins
             raise JSONRPCError(
                 'Only RhodeCode super-admin can specify `owner` param')
     return owner
 
 
 def get_user_or_error(userid):
     """
     Get user by id or name or return JsonRPCError if not found
 
     :param userid:
     """
     from rhodecode.model.user import UserModel
+    user_model = UserModel()
 
-    user_model = UserModel()
-    try:
-        user = user_model.get_user(int(userid))
-    except ValueError:
+    if isinstance(userid, (int, long)):
+        try:
+            user = user_model.get_user(userid)
+        except ValueError:
+            user = None
+    else:
         user = user_model.get_by_username(userid)
 
     if user is None:
-        raise JSONRPCError("user `%s` does not exist" % (userid,))
+        raise JSONRPCError(
+            'user `%s` does not exist' % (userid,))
     return user
 
 
 def get_repo_or_error(repoid):
     """
     Get repo by id or name or return JsonRPCError if not found
 
     :param repoid:
     """
     from rhodecode.model.repo import RepoModel
+    repo_model = RepoModel()
 
-    repo = RepoModel().get_repo(repoid)
+    if isinstance(repoid, (int, long)):
+        try:
+            repo = repo_model.get_repo(repoid)
+        except ValueError:
+            repo = None
+    else:
+        repo = repo_model.get_by_repo_name(repoid)
+
     if repo is None:
-        raise JSONRPCError('repository `%s` does not exist' % (repoid,))
+        raise JSONRPCError(
+            'repository `%s` does not exist' % (repoid,))
     return repo
 
 
 def get_repo_group_or_error(repogroupid):
     """
     Get repo group by id or name or return JsonRPCError if not found
 
     :param repogroupid:
     """
     from rhodecode.model.repo_group import RepoGroupModel
+    repo_group_model = RepoGroupModel()
 
-    repo_group = RepoGroupModel()._get_repo_group(repogroupid)
+    if isinstance(repogroupid, (int, long)):
+        try:
+            repo_group = repo_group_model._get_repo_group(repogroupid)
+        except ValueError:
+            repo_group = None
+    else:
+        repo_group = repo_group_model.get_by_group_name(repogroupid)
+
     if repo_group is None:
         raise JSONRPCError(
             'repository group `%s` does not exist' % (repogroupid,))
     return repo_group
 
 
 def get_user_group_or_error(usergroupid):
     """
     Get user group by id or name or return JsonRPCError if not found
 
     :param usergroupid:
     """
     from rhodecode.model.user_group import UserGroupModel
+    user_group_model = UserGroupModel()
 
-    user_group = UserGroupModel().get_group(usergroupid)
+    if isinstance(usergroupid, (int, long)):
+        try:
+            user_group = user_group_model.get_group(usergroupid)
+        except ValueError:
+            user_group = None
+    else:
+        user_group = user_group_model.get_by_name(usergroupid)
+
     if user_group is None:
-        raise JSONRPCError('user group `%s` does not exist' % (usergroupid,))
+        raise JSONRPCError(
+            'user group `%s` does not exist' % (usergroupid,))
     return user_group
 
 
 def get_perm_or_error(permid, prefix=None):
     """
     Get permission by id or name or return JsonRPCError if not found
 
     :param permid:
     """
     from rhodecode.model.permission import PermissionModel
 
     perm = PermissionModel.cls.get_by_key(permid)
     if perm is None:
         raise JSONRPCError('permission `%s` does not exist' % (permid,))
     if prefix:
         if not perm.permission_name.startswith(prefix):
             raise JSONRPCError('permission `%s` is invalid, '
                                'should start with %s' % (permid, prefix))
     return perm
 
 
 def get_gist_or_error(gistid):
     """
     Get gist by id or gist_access_id or return JsonRPCError if not found
 
     :param gistid:
     """
     from rhodecode.model.gist import GistModel
 
     gist = GistModel.cls.get_by_access_id(gistid)
     if gist is None:
         raise JSONRPCError('gist `%s` does not exist' % (gistid,))
     return gist
 
 
 def get_pull_request_or_error(pullrequestid):
     """
     Get pull request by id or return JsonRPCError if not found
 
     :param pullrequestid:
     """
     from rhodecode.model.pull_request import PullRequestModel
 
     try:
         pull_request = PullRequestModel().get(int(pullrequestid))
     except ValueError:
         raise JSONRPCError('pullrequestid must be an integer')
     if not pull_request:
         raise JSONRPCError('pull request `%s` does not exist' % (
             pullrequestid,))
     return pull_request
 
 
 def build_commit_data(commit, detail_level):
     parsed_diff = []
     if detail_level == 'extended':
         for f in commit.added:
             parsed_diff.append(_get_commit_dict(filename=f.path, op='A'))
         for f in commit.changed:
             parsed_diff.append(_get_commit_dict(filename=f.path, op='M'))
         for f in commit.removed:
             parsed_diff.append(_get_commit_dict(filename=f.path, op='D'))
 
     elif detail_level == 'full':
         from rhodecode.lib.diffs import DiffProcessor
         diff_processor = DiffProcessor(commit.diff())
         for dp in diff_processor.prepare():
             del dp['stats']['ops']
             _stats = dp['stats']
             parsed_diff.append(_get_commit_dict(
                 filename=dp['filename'], op=dp['operation'],
                 new_revision=dp['new_revision'],
                 old_revision=dp['old_revision'],
                 raw_diff=dp['raw_diff'], stats=_stats))
 
     return parsed_diff
 
 
 def get_commit_or_error(ref, repo):
     try:
         ref_type, _, ref_hash = ref.split(':')
     except ValueError:
         raise JSONRPCError(
             'Ref `{ref}` given in a wrong format. Please check the API'
             ' documentation for more details'.format(ref=ref))
     try:
         # TODO: dan: refactor this to use repo.scm_instance().get_commit()
         # once get_commit supports ref_types
         return get_commit_from_ref_name(repo, ref_hash)
     except RepositoryError:
         raise JSONRPCError('Ref `{ref}` does not exist'.format(ref=ref))
 
 
 def resolve_ref_or_error(ref, repo):
     def _parse_ref(type_, name, hash_=None):
         return type_, name, hash_
 
     try:
         ref_type, ref_name, ref_hash = _parse_ref(*ref.split(':'))
     except TypeError:
         raise JSONRPCError(
             'Ref `{ref}` given in a wrong format. Please check the API'
             ' documentation for more details'.format(ref=ref))
 
     try:
         ref_hash = ref_hash or _get_ref_hash(repo, ref_type, ref_name)
     except (KeyError, ValueError):
         raise JSONRPCError(
             'The specified {type} `{name}` does not exist'.format(
                 type=ref_type, name=ref_name))
 
     return ':'.join([ref_type, ref_name, ref_hash])
 
 
 def _get_commit_dict(
         filename, op, new_revision=None, old_revision=None,
         raw_diff=None, stats=None):
     if stats is None:
         stats = {
             "added": None,
             "binary": None,
             "deleted": None
         }
     return {
         "filename": safe_unicode(filename),
         "op": op,
 
         # extra details
         "new_revision": new_revision,
         "old_revision": old_revision,
 
         "raw_diff": raw_diff,
         "stats": stats
     }
 
 
 # TODO: mikhail: Think about moving this function to some library
 def _get_ref_hash(repo, type_, name):
     vcs_repo = repo.scm_instance()
     if type_ == 'branch' and vcs_repo.alias in ('hg', 'git'):
         return vcs_repo.branches[name]
     elif type_ == 'bookmark' and vcs_repo.alias == 'hg':
         return vcs_repo.bookmarks[name]
     else:
         raise ValueError()

rhodecode/api/views/pull_request_api.py

@@ -1,731 +1,725 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2011-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import logging
 
 from rhodecode.api import jsonrpc_method, JSONRPCError
 from rhodecode.api.utils import (
     has_superadmin_permission, Optional, OAttr, get_repo_or_error,
     get_pull_request_or_error, get_commit_or_error, get_user_or_error,
     validate_repo_permissions, resolve_ref_or_error)
 from rhodecode.lib.auth import (HasRepoPermissionAnyApi)
 from rhodecode.lib.base import vcs_operation_context
 from rhodecode.lib.utils2 import str2bool
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import CommentsModel
 from rhodecode.model.db import Session, ChangesetStatus, ChangesetComment
 from rhodecode.model.pull_request import PullRequestModel, MergeCheck
 from rhodecode.model.settings import SettingsModel
 
 log = logging.getLogger(__name__)
 
 
 @jsonrpc_method()
 def get_pull_request(request, apiuser, repoid, pullrequestid):
     """
     Get a pull request based on the given ID.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Repository name or repository ID from where the pull
         request was opened.
     :type repoid: str or int
     :param pullrequestid: ID of the requested pull request.
     :type pullrequestid: int
 
     Example output:
 
     .. code-block:: bash
 
       "id": <id_given_in_input>,
       "result":
         {
             "pull_request_id":   "<pull_request_id>",
             "url":               "<url>",
             "title":             "<title>",
             "description":       "<description>",
             "status" :           "<status>",
             "created_on":        "<date_time_created>",
             "updated_on":        "<date_time_updated>",
             "commit_ids":        [
                                      ...
                                      "<commit_id>",
                                      "<commit_id>",
                                      ...
                                  ],
             "review_status":    "<review_status>",
             "mergeable":         {
                                      "status":  "<bool>",
                                      "message": "<message>",
                                  },
             "source":            {
                                      "clone_url":     "<clone_url>",
                                      "repository":    "<repository_name>",
                                      "reference":
                                      {
                                          "name":      "<name>",
                                          "type":      "<type>",
                                          "commit_id": "<commit_id>",
                                      }
                                  },
             "target":            {
                                      "clone_url":   "<clone_url>",
                                      "repository":    "<repository_name>",
                                      "reference":
                                      {
                                          "name":      "<name>",
                                          "type":      "<type>",
                                          "commit_id": "<commit_id>",
                                      }
                                  },
             "merge":             {
                                      "clone_url":   "<clone_url>",
                                      "reference":
                                      {
                                          "name":      "<name>",
                                          "type":      "<type>",
                                          "commit_id": "<commit_id>",
                                      }
                                  },
            "author":             <user_obj>,
            "reviewers":          [
                                      ...
                                      {
                                         "user":          "<user_obj>",
                                         "review_status": "<review_status>",
                                      }
                                      ...
                                  ]
         },
        "error": null
     """
     get_repo_or_error(repoid)
     pull_request = get_pull_request_or_error(pullrequestid)
     if not PullRequestModel().check_user_read(
             pull_request, apiuser, api=True):
         raise JSONRPCError('repository `%s` does not exist' % (repoid,))
     data = pull_request.get_api_data()
     return data
 
 
 @jsonrpc_method()
 def get_pull_requests(request, apiuser, repoid, status=Optional('new')):
     """
     Get all pull requests from the repository specified in `repoid`.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Repository name or repository ID.
     :type repoid: str or int
     :param status: Only return pull requests with the specified status.
         Valid options are.
         * ``new`` (default)
         * ``open``
         * ``closed``
     :type status: str
 
     Example output:
 
     .. code-block:: bash
 
       "id": <id_given_in_input>,
       "result":
         [
             ...
             {
                 "pull_request_id":   "<pull_request_id>",
                 "url":               "<url>",
                 "title" :            "<title>",
                 "description":       "<description>",
                 "status":            "<status>",
                 "created_on":        "<date_time_created>",
                 "updated_on":        "<date_time_updated>",
                 "commit_ids":        [
                                          ...
                                          "<commit_id>",
                                          "<commit_id>",
                                          ...
                                      ],
                 "review_status":    "<review_status>",
                 "mergeable":         {
                                         "status":      "<bool>",
                                         "message:      "<message>",
                                      },
                 "source":            {
                                          "clone_url":     "<clone_url>",
                                          "reference":
                                          {
                                              "name":      "<name>",
                                              "type":      "<type>",
                                              "commit_id": "<commit_id>",
                                          }
                                      },
                 "target":            {
                                          "clone_url":   "<clone_url>",
                                          "reference":
                                          {
                                              "name":      "<name>",
                                              "type":      "<type>",
                                              "commit_id": "<commit_id>",
                                          }
                                      },
                 "merge":             {
                                          "clone_url":   "<clone_url>",
                                          "reference":
                                          {
                                              "name":      "<name>",
                                              "type":      "<type>",
                                              "commit_id": "<commit_id>",
                                          }
                                      },
                "author":             <user_obj>,
                "reviewers":          [
                                          ...
                                          {
                                             "user":          "<user_obj>",
                                             "review_status": "<review_status>",
                                          }
                                          ...
                                      ]
             }
             ...
         ],
       "error": null
 
     """
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     status = Optional.extract(status)
     pull_requests = PullRequestModel().get_all(repo, statuses=[status])
     data = [pr.get_api_data() for pr in pull_requests]
     return data
 
 
 @jsonrpc_method()
 def merge_pull_request(request, apiuser, repoid, pullrequestid,
                        userid=Optional(OAttr('apiuser'))):
     """
     Merge the pull request specified by `pullrequestid` into its target
     repository.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The Repository name or repository ID of the
         target repository to which the |pr| is to be merged.
     :type repoid: str or int
     :param pullrequestid: ID of the pull request which shall be merged.
     :type pullrequestid: int
     :param userid: Merge the pull request as this user.
     :type userid: Optional(str or int)
 
     Example output:
 
     .. code-block:: bash
 
-      "id": <id_given_in_input>,
-      "result":
-        {
+        "id": <id_given_in_input>,
+        "result": {
             "executed":         "<bool>",
             "failure_reason":   "<int>",
             "merge_commit_id":  "<merge_commit_id>",
             "possible":         "<bool>",
             "merge_ref":        {
                                     "commit_id": "<commit_id>",
                                     "type":      "<type>",
                                     "name":      "<name>"
                                 }
         },
-      "error": null
-
+        "error": null
     """
     repo = get_repo_or_error(repoid)
     if not isinstance(userid, Optional):
         if (has_superadmin_permission(apiuser) or
                 HasRepoPermissionAnyApi('repository.admin')(
                     user=apiuser, repo_name=repo.repo_name)):
             apiuser = get_user_or_error(userid)
         else:
             raise JSONRPCError('userid is not the same as your user')
 
     pull_request = get_pull_request_or_error(pullrequestid)
 
     check = MergeCheck.validate(pull_request, user=apiuser)
     merge_possible = not check.failed
 
     if not merge_possible:
         reasons = ','.join([msg for _e, msg in check.errors])
         raise JSONRPCError(
             'merge not possible for following reasons: {}'.format(reasons))
 
     target_repo = pull_request.target_repo
     extras = vcs_operation_context(
         request.environ, repo_name=target_repo.repo_name,
         username=apiuser.username, action='push',
         scm=target_repo.repo_type)
     merge_response = PullRequestModel().merge(
         pull_request, apiuser, extras=extras)
     if merge_response.executed:
         PullRequestModel().close_pull_request(
             pull_request.pull_request_id, apiuser)
 
         Session().commit()
 
     # In previous versions the merge response directly contained the merge
     # commit id. It is now contained in the merge reference object. To be
     # backwards compatible we have to extract it again.
     merge_response = merge_response._asdict()
     merge_response['merge_commit_id'] = merge_response['merge_ref'].commit_id
 
     return merge_response
 
 
 @jsonrpc_method()
 def close_pull_request(request, apiuser, repoid, pullrequestid,
                        userid=Optional(OAttr('apiuser'))):
     """
     Close the pull request specified by `pullrequestid`.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Repository name or repository ID to which the pull
         request belongs.
     :type repoid: str or int
     :param pullrequestid: ID of the pull request to be closed.
     :type pullrequestid: int
     :param userid: Close the pull request as this user.
     :type userid: Optional(str or int)
 
     Example output:
 
     .. code-block:: bash
 
-      "id": <id_given_in_input>,
-      "result":
-        {
+        "id": <id_given_in_input>,
+        "result": {
             "pull_request_id":  "<int>",
             "closed":           "<bool>"
         },
-      "error": null
+        "error": null
 
     """
     repo = get_repo_or_error(repoid)
     if not isinstance(userid, Optional):
         if (has_superadmin_permission(apiuser) or
                 HasRepoPermissionAnyApi('repository.admin')(
                     user=apiuser, repo_name=repo.repo_name)):
             apiuser = get_user_or_error(userid)
         else:
             raise JSONRPCError('userid is not the same as your user')
 
     pull_request = get_pull_request_or_error(pullrequestid)
     if not PullRequestModel().check_user_update(
             pull_request, apiuser, api=True):
         raise JSONRPCError(
             'pull request `%s` close failed, no permission to close.' % (
                 pullrequestid,))
     if pull_request.is_closed():
         raise JSONRPCError(
             'pull request `%s` is already closed' % (pullrequestid,))
 
     PullRequestModel().close_pull_request(
         pull_request.pull_request_id, apiuser)
     Session().commit()
     data = {
         'pull_request_id': pull_request.pull_request_id,
         'closed': True,
     }
     return data
 
 
 @jsonrpc_method()
 def comment_pull_request(
         request, apiuser, repoid, pullrequestid, message=Optional(None),
         commit_id=Optional(None), status=Optional(None),
         comment_type=Optional(ChangesetComment.COMMENT_TYPE_NOTE),
         resolves_comment_id=Optional(None),
         userid=Optional(OAttr('apiuser'))):
     """
     Comment on the pull request specified with the `pullrequestid`,
     in the |repo| specified by the `repoid`, and optionally change the
     review status.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
     :param pullrequestid: The pull request ID.
     :type pullrequestid: int
     :param commit_id: Specify the commit_id for which to set a comment. If
         given commit_id is different than latest in the PR status
         change won't be performed.
     :type commit_id: str
     :param message: The text content of the comment.
     :type message: str
     :param status: (**Optional**) Set the approval status of the pull
         request. One of: 'not_reviewed', 'approved', 'rejected',
         'under_review'
     :type status: str
     :param comment_type: Comment type, one of: 'note', 'todo'
     :type comment_type: Optional(str), default: 'note'
     :param userid: Comment on the pull request as this user
     :type userid: Optional(str or int)
 
     Example output:
 
     .. code-block:: bash
 
-      id : <id_given_in_input>
-      result :
-        {
+        id : <id_given_in_input>
+        result : {
             "pull_request_id":  "<Integer>",
             "comment_id":       "<Integer>",
             "status": {"given": <given_status>,
                        "was_changed": <bool status_was_actually_changed> },
-        }
-      error :  null
+        },
+        error :  null
     """
     repo = get_repo_or_error(repoid)
     if not isinstance(userid, Optional):
         if (has_superadmin_permission(apiuser) or
                 HasRepoPermissionAnyApi('repository.admin')(
                     user=apiuser, repo_name=repo.repo_name)):
             apiuser = get_user_or_error(userid)
         else:
             raise JSONRPCError('userid is not the same as your user')
 
     pull_request = get_pull_request_or_error(pullrequestid)
     if not PullRequestModel().check_user_read(
             pull_request, apiuser, api=True):
         raise JSONRPCError('repository `%s` does not exist' % (repoid,))
     message = Optional.extract(message)
     status = Optional.extract(status)
     commit_id = Optional.extract(commit_id)
     comment_type = Optional.extract(comment_type)
     resolves_comment_id = Optional.extract(resolves_comment_id)
 
     if not message and not status:
         raise JSONRPCError(
             'Both message and status parameters are missing. '
             'At least one is required.')
 
     if (status not in (st[0] for st in ChangesetStatus.STATUSES) and
             status is not None):
         raise JSONRPCError('Unknown comment status: `%s`' % status)
 
     if commit_id and commit_id not in pull_request.revisions:
         raise JSONRPCError(
             'Invalid commit_id `%s` for this pull request.' % commit_id)
 
     allowed_to_change_status = PullRequestModel().check_user_change_status(
         pull_request, apiuser)
 
     # if commit_id is passed re-validated if user is allowed to change status
     # based on latest commit_id from the PR
     if commit_id:
         commit_idx = pull_request.revisions.index(commit_id)
         if commit_idx != 0:
             allowed_to_change_status = False
 
     if resolves_comment_id:
         comment = ChangesetComment.get(resolves_comment_id)
         if not comment:
             raise JSONRPCError(
                 'Invalid resolves_comment_id `%s` for this pull request.'
                 % resolves_comment_id)
         if comment.comment_type != ChangesetComment.COMMENT_TYPE_TODO:
             raise JSONRPCError(
                 'Comment `%s` is wrong type for setting status to resolved.'
                 % resolves_comment_id)
 
     text = message
     status_label = ChangesetStatus.get_status_lbl(status)
     if status and allowed_to_change_status:
         st_message = ('Status change %(transition_icon)s %(status)s'
                       % {'transition_icon': '>', 'status': status_label})
         text = message or st_message
 
     rc_config = SettingsModel().get_all_settings()
     renderer = rc_config.get('rhodecode_markup_renderer', 'rst')
 
     status_change = status and allowed_to_change_status
     comment = CommentsModel().create(
         text=text,
         repo=pull_request.target_repo.repo_id,
         user=apiuser.user_id,
         pull_request=pull_request.pull_request_id,
         f_path=None,
         line_no=None,
         status_change=(status_label if status_change else None),
         status_change_type=(status if status_change else None),
         closing_pr=False,
         renderer=renderer,
         comment_type=comment_type,
         resolves_comment_id=resolves_comment_id
     )
 
     if allowed_to_change_status and status:
         ChangesetStatusModel().set_status(
             pull_request.target_repo.repo_id,
             status,
             apiuser.user_id,
             comment,
             pull_request=pull_request.pull_request_id
         )
         Session().flush()
 
     Session().commit()
     data = {
         'pull_request_id': pull_request.pull_request_id,
         'comment_id': comment.comment_id if comment else None,
         'status': {'given': status, 'was_changed': status_change},
     }
     return data
 
 
 @jsonrpc_method()
 def create_pull_request(
         request, apiuser, source_repo, target_repo, source_ref, target_ref,
         title, description=Optional(''), reviewers=Optional(None)):
     """
     Creates a new pull request.
 
     Accepts refs in the following formats:
 
         * branch:<branch_name>:<sha>
         * branch:<branch_name>
         * bookmark:<bookmark_name>:<sha> (Mercurial only)
         * bookmark:<bookmark_name> (Mercurial only)
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param source_repo: Set the source repository name.
     :type source_repo: str
     :param target_repo: Set the target repository name.
     :type target_repo: str
     :param source_ref: Set the source ref name.
     :type source_ref: str
     :param target_ref: Set the target ref name.
     :type target_ref: str
     :param title: Set the pull request title.
     :type title: str
     :param description: Set the pull request description.
     :type description: Optional(str)
     :param reviewers: Set the new pull request reviewers list.
     :type reviewers: Optional(list)
         Accepts username strings or objects of the format:
-        {
-            'username': 'nick', 'reasons': ['original author']
-        }
+
+            {'username': 'nick', 'reasons': ['original author']}
     """
 
     source = get_repo_or_error(source_repo)
     target = get_repo_or_error(target_repo)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, source_repo, source, _perms)
 
     full_source_ref = resolve_ref_or_error(source_ref, source)
     full_target_ref = resolve_ref_or_error(target_ref, target)
     source_commit = get_commit_or_error(full_source_ref, source)
     target_commit = get_commit_or_error(full_target_ref, target)
     source_scm = source.scm_instance()
     target_scm = target.scm_instance()
 
     commit_ranges = target_scm.compare(
         target_commit.raw_id, source_commit.raw_id, source_scm,
         merge=True, pre_load=[])
 
     ancestor = target_scm.get_common_ancestor(
         target_commit.raw_id, source_commit.raw_id, source_scm)
 
     if not commit_ranges:
         raise JSONRPCError('no commits found')
 
     if not ancestor:
         raise JSONRPCError('no common ancestor found')
 
     reviewer_objects = Optional.extract(reviewers) or []
     if not isinstance(reviewer_objects, list):
         raise JSONRPCError('reviewers should be specified as a list')
 
     reviewers_reasons = []
     for reviewer_object in reviewer_objects:
         reviewer_reasons = []
         if isinstance(reviewer_object, (basestring, int)):
             reviewer_username = reviewer_object
         else:
             reviewer_username = reviewer_object['username']
             reviewer_reasons = reviewer_object.get('reasons', [])
 
         user = get_user_or_error(reviewer_username)
         reviewers_reasons.append((user.user_id, reviewer_reasons))
 
     pull_request_model = PullRequestModel()
     pull_request = pull_request_model.create(
         created_by=apiuser.user_id,
         source_repo=source_repo,
         source_ref=full_source_ref,
         target_repo=target_repo,
         target_ref=full_target_ref,
         revisions=reversed(
             [commit.raw_id for commit in reversed(commit_ranges)]),
         reviewers=reviewers_reasons,
         title=title,
         description=Optional.extract(description)
     )
 
     Session().commit()
     data = {
         'msg': 'Created new pull request `{}`'.format(title),
         'pull_request_id': pull_request.pull_request_id,
     }
     return data
 
 
 @jsonrpc_method()
 def update_pull_request(
         request, apiuser, repoid, pullrequestid, title=Optional(''),
         description=Optional(''), reviewers=Optional(None),
         update_commits=Optional(None), close_pull_request=Optional(None)):
     """
     Updates a pull request.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
     :param pullrequestid: The pull request ID.
     :type pullrequestid: int
     :param title: Set the pull request title.
     :type title: str
     :param description: Update pull request description.
     :type description: Optional(str)
     :param reviewers: Update pull request reviewers list with new value.
     :type reviewers: Optional(list)
     :param update_commits: Trigger update of commits for this pull request
     :type: update_commits: Optional(bool)
     :param close_pull_request: Close this pull request with rejected state
     :type: close_pull_request: Optional(bool)
 
     Example output:
 
     .. code-block:: bash
 
-      id : <id_given_in_input>
-      result :
-        {
+        id : <id_given_in_input>
+        result : {
             "msg": "Updated pull request `63`",
             "pull_request": <pull_request_object>,
             "updated_reviewers": {
               "added": [
                 "username"
               ],
               "removed": []
             },
             "updated_commits": {
               "added": [
                 "<sha1_hash>"
               ],
               "common": [
                 "<sha1_hash>",
                 "<sha1_hash>",
               ],
               "removed": []
             }
         }
-      error :  null
+        error :  null
     """
 
     repo = get_repo_or_error(repoid)
     pull_request = get_pull_request_or_error(pullrequestid)
     if not PullRequestModel().check_user_update(
             pull_request, apiuser, api=True):
         raise JSONRPCError(
             'pull request `%s` update failed, no permission to update.' % (
                 pullrequestid,))
     if pull_request.is_closed():
         raise JSONRPCError(
             'pull request `%s` update failed, pull request is closed' % (
                 pullrequestid,))
 
     reviewer_objects = Optional.extract(reviewers) or []
     if not isinstance(reviewer_objects, list):
         raise JSONRPCError('reviewers should be specified as a list')
 
     reviewers_reasons = []
     reviewer_ids = set()
     for reviewer_object in reviewer_objects:
         reviewer_reasons = []
         if isinstance(reviewer_object, (int, basestring)):
             reviewer_username = reviewer_object
         else:
             reviewer_username = reviewer_object['username']
             reviewer_reasons = reviewer_object.get('reasons', [])
 
         user = get_user_or_error(reviewer_username)
         reviewer_ids.add(user.user_id)
         reviewers_reasons.append((user.user_id, reviewer_reasons))
 
     title = Optional.extract(title)
     description = Optional.extract(description)
     if title or description:
         PullRequestModel().edit(
             pull_request, title or pull_request.title,
             description or pull_request.description)
         Session().commit()
 
     commit_changes = {"added": [], "common": [], "removed": []}
     if str2bool(Optional.extract(update_commits)):
         if PullRequestModel().has_valid_update_type(pull_request):
             update_response = PullRequestModel().update_commits(
                 pull_request)
             commit_changes = update_response.changes or commit_changes
         Session().commit()
 
     reviewers_changes = {"added": [], "removed": []}
     if reviewer_ids:
         added_reviewers, removed_reviewers = \
             PullRequestModel().update_reviewers(pull_request, reviewers_reasons)
 
         reviewers_changes['added'] = sorted(
             [get_user_or_error(n).username for n in added_reviewers])
         reviewers_changes['removed'] = sorted(
             [get_user_or_error(n).username for n in removed_reviewers])
         Session().commit()
 
     if str2bool(Optional.extract(close_pull_request)):
         PullRequestModel().close_pull_request_with_comment(
             pull_request, apiuser, repo)
         Session().commit()
 
     data = {
         'msg': 'Updated pull request `{}`'.format(
             pull_request.pull_request_id),
         'pull_request': pull_request.get_api_data(),
         'updated_commits': commit_changes,
         'updated_reviewers': reviewers_changes
     }
 
     return data

rhodecode/api/views/server_api.py

@@ -1,245 +1,321 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2011-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
-
+import inspect
 import logging
+import itertools
 
-from rhodecode.api import jsonrpc_method, JSONRPCError, JSONRPCForbidden
+from rhodecode.api import (
+    jsonrpc_method, JSONRPCError, JSONRPCForbidden, find_methods)
 
 from rhodecode.api.utils import (
     Optional, OAttr, has_superadmin_permission, get_user_or_error)
 from rhodecode.lib.utils import repo2db_mapper
 from rhodecode.lib import system_info
 from rhodecode.lib import user_sessions
 from rhodecode.lib.utils2 import safe_int
 from rhodecode.model.db import UserIpMap
 from rhodecode.model.scm import ScmModel
 
 log = logging.getLogger(__name__)
 
 
 @jsonrpc_method()
 def get_server_info(request, apiuser):
     """
     Returns the |RCE| server information.
 
     This includes the running version of |RCE| and all installed
     packages. This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         'modules': [<module name>,...]
         'py_version': <python version>,
         'platform': <platform type>,
         'rhodecode_version': <rhodecode version>
       }
       error :  null
     """
 
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     server_info = ScmModel().get_server_info(request.environ)
     # rhodecode-index requires those
 
     server_info['index_storage'] = server_info['search']['value']['location']
     server_info['storage'] = server_info['storage']['value']['path']
 
     return server_info
 
 
 @jsonrpc_method()
 def get_ip(request, apiuser, userid=Optional(OAttr('apiuser'))):
     """
     Displays the IP Address as seen from the |RCE| server.
 
     * This command displays the IP Address, as well as all the defined IP
       addresses for the specified user. If the ``userid`` is not set, the
       data returned is for the user calling the method.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from |authtoken|.
     :type apiuser: AuthUser
     :param userid: Sets the userid for which associated IP Address data
         is returned.
     :type userid: Optional(str or int)
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result : {
                      "server_ip_addr": "<ip_from_clien>",
                      "user_ips": [
                                     {
                                        "ip_addr": "<ip_with_mask>",
                                        "ip_range": ["<start_ip>", "<end_ip>"],
                                     },
                                     ...
                                  ]
         }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     userid = Optional.extract(userid, evaluate_locals=locals())
     userid = getattr(userid, 'user_id', userid)
 
     user = get_user_or_error(userid)
     ips = UserIpMap.query().filter(UserIpMap.user == user).all()
     return {
         'server_ip_addr': request.rpc_ip_addr,
         'user_ips': ips
     }
 
 
 @jsonrpc_method()
 def rescan_repos(request, apiuser, remove_obsolete=Optional(False)):
     """
     Triggers a rescan of the specified repositories.
 
     * If the ``remove_obsolete`` option is set, it also deletes repositories
       that are found in the database but not on the file system, so called
       "clean zombies".
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param remove_obsolete: Deletes repositories from the database that
         are not found on the filesystem.
     :type remove_obsolete: Optional(``True`` | ``False``)
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         'added': [<added repository name>,...]
         'removed': [<removed repository name>,...]
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         'Error occurred during rescan repositories action'
       }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     try:
         rm_obsolete = Optional.extract(remove_obsolete)
         added, removed = repo2db_mapper(ScmModel().repo_scan(),
                                         remove_obsolete=rm_obsolete)
         return {'added': added, 'removed': removed}
     except Exception:
         log.exception('Failed to run repo rescann')
         raise JSONRPCError(
             'Error occurred during rescan repositories action'
         )
 
 
 @jsonrpc_method()
 def cleanup_sessions(request, apiuser, older_then=Optional(60)):
     """
     Triggers a session cleanup action.
 
     If the ``older_then`` option is set, only sessions that hasn't been
     accessed in the given number of days will be removed.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param older_then: Deletes session that hasn't been accessed
         in given number of days.
     :type older_then: Optional(int)
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result: {
         "backend": "<type of backend>",
         "sessions_removed": <number_of_removed_sessions>
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         'Error occurred during session cleanup'
       }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     older_then = safe_int(Optional.extract(older_then)) or 60
     older_than_seconds = 60 * 60 * 24 * older_then
 
     config = system_info.rhodecode_config().get_value()['value']['config']
     session_model = user_sessions.get_session_handler(
         config.get('beaker.session.type', 'memory'))(config)
 
     backend = session_model.SESSION_TYPE
     try:
         cleaned = session_model.clean_sessions(
             older_than_seconds=older_than_seconds)
         return {'sessions_removed': cleaned, 'backend': backend}
     except user_sessions.CleanupCommand as msg:
         return {'cleanup_command': msg.message, 'backend': backend}
     except Exception as e:
         log.exception('Failed session cleanup')
         raise JSONRPCError(
             'Error occurred during session cleanup'
         )
+
+
+@jsonrpc_method()
+def get_method(request, apiuser, pattern=Optional('*')):
+    """
+    Returns list of all available API methods. By default match pattern
+    os "*" but any other pattern can be specified. eg *comment* will return
+    all methods with comment inside them. If just single method is matched
+    returned data will also include method specification
+
+    This command can only be run using an |authtoken| with admin rights to
+    the specified repository.
+
+    This command takes the following options:
+
+    :param apiuser: This is filled automatically from the |authtoken|.
+    :type apiuser: AuthUser
+    :param pattern: pattern to match method names against
+    :type older_then: Optional("*")
+
+    Example output:
+
+    .. code-block:: bash
+
+      id : <id_given_in_input>
+      "result": [
+        "changeset_comment",
+        "comment_pull_request",
+        "comment_commit"
+      ]
+      error :  null
+
+    .. code-block:: bash
+
+      id : <id_given_in_input>
+      "result": [
+        "comment_commit",
+        {
+          "apiuser": "<RequiredType>",
+          "comment_type": "<Optional:u'note'>",
+          "commit_id": "<RequiredType>",
+          "message": "<RequiredType>",
+          "repoid": "<RequiredType>",
+          "request": "<RequiredType>",
+          "resolves_comment_id": "<Optional:None>",
+          "status": "<Optional:None>",
+          "userid": "<Optional:<OptionalAttr:apiuser>>"
+        }
+      ]
+      error :  null
+    """
+    if not has_superadmin_permission(apiuser):
+        raise JSONRPCForbidden()
+
+    pattern = Optional.extract(pattern)
+
+    matches = find_methods(request.registry.jsonrpc_methods, pattern)
+
+    args_desc = []
+    if len(matches) == 1:
+        func = matches[matches.keys()[0]]
+
+        argspec = inspect.getargspec(func)
+        arglist = argspec[0]
+        defaults = map(repr, argspec[3] or [])
+
+        default_empty = '<RequiredType>'
+
+        # kw arguments required by this method
+        func_kwargs = dict(itertools.izip_longest(
+            reversed(arglist), reversed(defaults), fillvalue=default_empty))
+        args_desc.append(func_kwargs)
+
+    return matches.keys() + args_desc

rhodecode/api/views/user_api.py

@@ -1,472 +1,515 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2011-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 
 from rhodecode.api import jsonrpc_method, JSONRPCError, JSONRPCForbidden
 from rhodecode.api.utils import (
     Optional, OAttr, has_superadmin_permission, get_user_or_error, store_update)
 from rhodecode.lib.auth import AuthUser, PasswordGenerator
 from rhodecode.lib.exceptions import DefaultUserException
 from rhodecode.lib.utils2 import safe_int, str2bool
 from rhodecode.model.db import Session, User, Repository
 from rhodecode.model.user import UserModel
 
-
 log = logging.getLogger(__name__)
 
 
 @jsonrpc_method()
 def get_user(request, apiuser, userid=Optional(OAttr('apiuser'))):
     """
     Returns the information associated with a username or userid.
 
     * If the ``userid`` is not set, this command returns the information
       for the ``userid`` calling the method.
 
     .. note::
 
        Normal users may only run this command against their ``userid``. For
        full privileges you must run this command using an |authtoken| with
        admin rights.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param userid: Sets the userid for which data will be returned.
     :type userid: Optional(str or int)
 
     Example output:
 
     .. code-block:: bash
 
         {
           "error": null,
           "id": <id>,
           "result": {
             "active": true,
             "admin": false,
-            "api_key": "api-key",
             "api_keys": [ list of keys ],
+            "auth_tokens": [ list of tokens with details ],
             "email": "[e-mail]",
             "emails": [
               "[e-mail]"
             ],
             "extern_name": "rhodecode",
             "extern_type": "rhodecode",
             "firstname": "username",
             "ip_addresses": [],
             "language": null,
             "last_login": "Timestamp",
+            "last_activity": "Timestamp",
             "lastname": "surnae",
             "permissions": {
               "global": [
                 "hg.inherit_default_perms.true",
                 "usergroup.read",
                 "hg.repogroup.create.false",
                 "hg.create.none",
                 "hg.password_reset.enabled",
                 "hg.extern_activate.manual",
                 "hg.create.write_on_repogroup.false",
                 "hg.usergroup.create.false",
                 "group.none",
                 "repository.none",
                 "hg.register.none",
                 "hg.fork.repository"
               ],
               "repositories": { "username/example": "repository.write"},
               "repositories_groups": { "user-group/repo": "group.none" },
               "user_groups": { "user_group_name": "usergroup.read" }
             },
             "user_id": 32,
             "username": "username"
           }
         }
     """
 
     if not has_superadmin_permission(apiuser):
         # make sure normal user does not pass someone else userid,
         # he is not allowed to do that
         if not isinstance(userid, Optional) and userid != apiuser.user_id:
             raise JSONRPCError('userid is not the same as your user')
 
     userid = Optional.extract(userid, evaluate_locals=locals())
     userid = getattr(userid, 'user_id', userid)
 
     user = get_user_or_error(userid)
     data = user.get_api_data(include_secrets=True)
     data['permissions'] = AuthUser(user_id=user.user_id).permissions
     return data
 
 
 @jsonrpc_method()
 def get_users(request, apiuser):
     """
     Lists all users in the |RCE| user database.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
-            result: [<user_object>, ...]
+        result: [<user_object>, ...]
         error:  null
     """
 
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     result = []
     users_list = User.query().order_by(User.username) \
         .filter(User.username != User.DEFAULT_USER) \
         .all()
     for user in users_list:
         result.append(user.get_api_data(include_secrets=True))
     return result
 
 
 @jsonrpc_method()
 def create_user(request, apiuser, username, email, password=Optional(''),
                 firstname=Optional(''), lastname=Optional(''),
                 active=Optional(True), admin=Optional(False),
                 extern_name=Optional('rhodecode'),
                 extern_type=Optional('rhodecode'),
                 force_password_change=Optional(False),
                 create_personal_repo_group=Optional(None)):
     """
     Creates a new user and returns the new user object.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param username: Set the new username.
     :type username: str or int
     :param email: Set the user email address.
     :type email: str
     :param password: Set the new user password.
     :type password: Optional(str)
     :param firstname: Set the new user firstname.
     :type firstname: Optional(str)
     :param lastname: Set the new user surname.
     :type lastname: Optional(str)
     :param active: Set the user as active.
     :type active: Optional(``True`` | ``False``)
     :param admin: Give the new user admin rights.
     :type admin: Optional(``True`` | ``False``)
     :param extern_name: Set the authentication plugin name.
         Using LDAP this is filled with LDAP UID.
     :type extern_name: Optional(str)
     :param extern_type: Set the new user authentication plugin.
     :type extern_type: Optional(str)
     :param force_password_change: Force the new user to change password
         on next login.
     :type force_password_change: Optional(``True`` | ``False``)
     :param create_personal_repo_group: Create personal repo group for this user
     :type create_personal_repo_group: Optional(``True`` | ``False``)
+
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
-                  "msg" : "created new user `<username>`",
-                  "user": <user_obj>
-                }
+            "msg" : "created new user `<username>`",
+            "user": <user_obj>
+        }
         error:  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "user `<username>` already exist"
         or
         "email `<email>` already exist"
         or
         "failed to create user `<username>`"
       }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     if UserModel().get_by_username(username):
         raise JSONRPCError("user `%s` already exist" % (username,))
 
     if UserModel().get_by_email(email, case_insensitive=True):
         raise JSONRPCError("email `%s` already exist" % (email,))
 
     # generate random password if we actually given the
     # extern_name and it's not rhodecode
     if (not isinstance(extern_name, Optional) and
                 Optional.extract(extern_name) != 'rhodecode'):
         # generate temporary password if user is external
         password = PasswordGenerator().gen_password(length=16)
     create_repo_group = Optional.extract(create_personal_repo_group)
     if isinstance(create_repo_group, basestring):
         create_repo_group = str2bool(create_repo_group)
 
     try:
         user = UserModel().create_or_update(
             username=Optional.extract(username),
             password=Optional.extract(password),
             email=Optional.extract(email),
             firstname=Optional.extract(firstname),
             lastname=Optional.extract(lastname),
             active=Optional.extract(active),
             admin=Optional.extract(admin),
             extern_type=Optional.extract(extern_type),
             extern_name=Optional.extract(extern_name),
             force_password_change=Optional.extract(force_password_change),
             create_repo_group=create_repo_group
         )
         Session().commit()
         return {
             'msg': 'created new user `%s`' % username,
             'user': user.get_api_data(include_secrets=True)
         }
     except Exception:
         log.exception('Error occurred during creation of user')
         raise JSONRPCError('failed to create user `%s`' % (username,))
 
 
 @jsonrpc_method()
 def update_user(request, apiuser, userid, username=Optional(None),
                 email=Optional(None), password=Optional(None),
                 firstname=Optional(None), lastname=Optional(None),
                 active=Optional(None), admin=Optional(None),
                 extern_type=Optional(None), extern_name=Optional(None), ):
     """
     Updates the details for the specified user, if that user exists.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from |authtoken|.
     :type apiuser: AuthUser
     :param userid: Set the ``userid`` to update.
     :type userid: str or int
     :param username: Set the new username.
     :type username: str or int
     :param email: Set the new email.
     :type email: str
     :param password: Set the new password.
     :type password: Optional(str)
     :param firstname: Set the new first name.
     :type firstname: Optional(str)
     :param lastname: Set the new surname.
     :type lastname: Optional(str)
     :param active: Set the new user as active.
     :type active: Optional(``True`` | ``False``)
     :param admin: Give the user admin rights.
     :type admin: Optional(``True`` | ``False``)
     :param extern_name: Set the authentication plugin user name.
         Using LDAP this is filled with LDAP UID.
     :type extern_name: Optional(str)
     :param extern_type: Set the authentication plugin type.
     :type extern_type: Optional(str)
 
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
-                  "msg" : "updated user ID:<userid> <username>",
-                  "user": <user_object>,
-                }
+            "msg" : "updated user ID:<userid> <username>",
+            "user": <user_object>,
+        }
         error:  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "failed to update user `<username>`"
       }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     user = get_user_or_error(userid)
 
     # only non optional arguments will be stored in updates
     updates = {}
 
     try:
 
         store_update(updates, username, 'username')
         store_update(updates, password, 'password')
         store_update(updates, email, 'email')
         store_update(updates, firstname, 'name')
         store_update(updates, lastname, 'lastname')
         store_update(updates, active, 'active')
         store_update(updates, admin, 'admin')
         store_update(updates, extern_name, 'extern_name')
         store_update(updates, extern_type, 'extern_type')
 
         user = UserModel().update_user(user, **updates)
         Session().commit()
         return {
             'msg': 'updated user ID:%s %s' % (user.user_id, user.username),
             'user': user.get_api_data(include_secrets=True)
         }
     except DefaultUserException:
         log.exception("Default user edit exception")
         raise JSONRPCError('editing default user is forbidden')
     except Exception:
         log.exception("Error occurred during update of user")
         raise JSONRPCError('failed to update user `%s`' % (userid,))
 
 
 @jsonrpc_method()
 def delete_user(request, apiuser, userid):
     """
     Deletes the specified user from the |RCE| user database.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     .. important::
 
        Ensure all open pull requests and open code review
        requests to this user are close.
 
        Also ensure all repositories, or repository groups owned by this
        user are reassigned before deletion.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param userid: Set the user to delete.
     :type userid: str or int
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
-                  "msg" : "deleted user ID:<userid> <username>",
-                  "user": null
-                }
+            "msg" : "deleted user ID:<userid> <username>",
+            "user": null
+        }
         error:  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "failed to delete user ID:<userid> <username>"
       }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     user = get_user_or_error(userid)
 
     try:
         UserModel().delete(userid)
         Session().commit()
         return {
             'msg': 'deleted user ID:%s %s' % (user.user_id, user.username),
             'user': None
         }
     except Exception:
         log.exception("Error occurred during deleting of user")
         raise JSONRPCError(
             'failed to delete user ID:%s %s' % (user.user_id, user.username))
 
 
 @jsonrpc_method()
 def get_user_locks(request, apiuser, userid=Optional(OAttr('apiuser'))):
     """
     Displays all repositories locked by the specified user.
 
     * If this command is run by a non-admin user, it returns
       a list of |repos| locked by that user.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param userid: Sets the userid whose list of locked |repos| will be
         displayed.
     :type userid: Optional(str or int)
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result : {
             [repo_object, repo_object,...]
         }
         error :  null
     """
 
     include_secrets = False
     if not has_superadmin_permission(apiuser):
         # make sure normal user does not pass someone else userid,
         # he is not allowed to do that
         if not isinstance(userid, Optional) and userid != apiuser.user_id:
             raise JSONRPCError('userid is not the same as your user')
     else:
         include_secrets = True
 
     userid = Optional.extract(userid, evaluate_locals=locals())
     userid = getattr(userid, 'user_id', userid)
     user = get_user_or_error(userid)
 
     ret = []
 
     # show all locks
     for r in Repository.getAll():
         _user_id, _time, _reason = r.locked
         if _user_id and _time:
             _api_data = r.get_api_data(include_secrets=include_secrets)
             # if we use user filter just show the locks for this user
             if safe_int(_user_id) == user.user_id:
                 ret.append(_api_data)
 
     return ret
+
+
+@jsonrpc_method()
+def get_user_audit_logs(request, apiuser, userid=Optional(OAttr('apiuser'))):
+    """
+    Fetches all action logs made by the specified user.
+
+    This command takes the following options:
+
+    :param apiuser: This is filled automatically from the |authtoken|.
+    :type apiuser: AuthUser
+    :param userid: Sets the userid whose list of locked |repos| will be
+        displayed.
+    :type userid: Optional(str or int)
+
+    Example output:
+
+    .. code-block:: bash
+
+        id : <id_given_in_input>
+        result : {
+            [action, action,...]
+        }
+        error :  null
+    """
+
+    if not has_superadmin_permission(apiuser):
+        # make sure normal user does not pass someone else userid,
+        # he is not allowed to do that
+        if not isinstance(userid, Optional) and userid != apiuser.user_id:
+            raise JSONRPCError('userid is not the same as your user')
+
+    userid = Optional.extract(userid, evaluate_locals=locals())
+    userid = getattr(userid, 'user_id', userid)
+    user = get_user_or_error(userid)
+
+    ret = []
+
+    # show all user actions
+    for entry in UserModel().get_user_log(user, filter_term=None):
+        ret.append(entry)
+    return ret

rhodecode/apps/admin/__init__.py

@@ -1,57 +1,99 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
-from rhodecode.admin.navigation import NavigationRegistry
+from rhodecode.apps.admin.navigation import NavigationRegistry
 from rhodecode.config.routing import ADMIN_PREFIX
 from rhodecode.lib.utils2 import str2bool
 
 
+def admin_routes(config):
+    """
+    Admin prefixed routes
+    """
+
+    config.add_route(
+        name='admin_settings_open_source',
+        pattern='/settings/open_source')
+    config.add_route(
+        name='admin_settings_vcs_svn_generate_cfg',
+        pattern='/settings/vcs/svn_generate_cfg')
+
+    config.add_route(
+        name='admin_settings_system',
+        pattern='/settings/system')
+    config.add_route(
+        name='admin_settings_system_update',
+        pattern='/settings/system/updates')
+
+    config.add_route(
+        name='admin_settings_sessions',
+        pattern='/settings/sessions')
+    config.add_route(
+        name='admin_settings_sessions_cleanup',
+        pattern='/settings/sessions/cleanup')
+
+    # users admin
+    config.add_route(
+        name='users',
+        pattern='/users')
+
+    config.add_route(
+        name='users_data',
+        pattern='/users_data')
+
+    # user auth tokens
+    config.add_route(
+        name='edit_user_auth_tokens',
+        pattern='/users/{user_id:\d+}/edit/auth_tokens')
+    config.add_route(
+        name='edit_user_auth_tokens_add',
+        pattern='/users/{user_id:\d+}/edit/auth_tokens/new')
+    config.add_route(
+        name='edit_user_auth_tokens_delete',
+        pattern='/users/{user_id:\d+}/edit/auth_tokens/delete')
+
+    # user groups management
+    config.add_route(
+        name='edit_user_groups_management',
+        pattern='/users/{user_id:\d+}/edit/groups_management')
+
+    config.add_route(
+        name='edit_user_groups_management_updates',
+        pattern='/users/{user_id:\d+}/edit/edit_user_groups_management/updates')
+
+    # user audit logs
+    config.add_route(
+        name='edit_user_audit_logs',
+        pattern='/users/{user_id:\d+}/edit/audit')
+
+
 def includeme(config):
     settings = config.get_settings()
 
     # Create admin navigation registry and add it to the pyramid registry.
     labs_active = str2bool(settings.get('labs_settings_active', False))
     navigation_registry = NavigationRegistry(labs_active=labs_active)
     config.registry.registerUtility(navigation_registry)
 
-    config.add_route(
-        name='admin_settings_open_source',
-        pattern=ADMIN_PREFIX + '/settings/open_source')
-    config.add_route(
-        name='admin_settings_vcs_svn_generate_cfg',
-        pattern=ADMIN_PREFIX + '/settings/vcs/svn_generate_cfg')
-
-    config.add_route(
-        name='admin_settings_system',
-        pattern=ADMIN_PREFIX + '/settings/system')
-    config.add_route(
-        name='admin_settings_system_update',
-        pattern=ADMIN_PREFIX + '/settings/system/updates')
-
-    config.add_route(
-        name='admin_settings_sessions',
-        pattern=ADMIN_PREFIX + '/settings/sessions')
-    config.add_route(
-        name='admin_settings_sessions_cleanup',
-        pattern=ADMIN_PREFIX + '/settings/sessions/cleanup')
+    config.include(admin_routes, route_prefix=ADMIN_PREFIX)
 
     # Scan module for configuration decorators.
     config.scan()

rhodecode/apps/admin/navigation.py

@@ -1,140 +1,140 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import logging
 import collections
 
 from pylons import url
 from zope.interface import implementer
 
-from rhodecode.admin.interfaces import IAdminNavigationRegistry
+from rhodecode.apps.admin.interfaces import IAdminNavigationRegistry
 from rhodecode.lib.utils import get_registry
 from rhodecode.translation import _
 
 
 log = logging.getLogger(__name__)
 
 NavListEntry = collections.namedtuple('NavListEntry', ['key', 'name', 'url'])
 
 
 class NavEntry(object):
     """
     Represents an entry in the admin navigation.
 
     :param key: Unique identifier used to store reference in an OrderedDict.
     :param name: Display name, usually a translation string.
     :param view_name: Name of the view, used generate the URL.
     :param pyramid: Indicator to use pyramid for URL generation. This should
         be removed as soon as we are fully migrated to pyramid.
     """
 
     def __init__(self, key, name, view_name, pyramid=False):
         self.key = key
         self.name = name
         self.view_name = view_name
         self.pyramid = pyramid
 
     def generate_url(self, request):
         if self.pyramid:
             if hasattr(request, 'route_path'):
                 return request.route_path(self.view_name)
             else:
                 # TODO: johbo: Remove this after migrating to pyramid.
                 # We need the pyramid request here to generate URLs to pyramid
                 # views from within pylons views.
                 from pyramid.threadlocal import get_current_request
                 pyramid_request = get_current_request()
                 return pyramid_request.route_path(self.view_name)
         else:
             return url(self.view_name)
 
     def get_localized_name(self, request):
         if hasattr(request, 'translate'):
             return request.translate(self.name)
         else:
             # TODO(marcink): Remove this after migrating to pyramid
             from pyramid.threadlocal import get_current_request
             pyramid_request = get_current_request()
             return pyramid_request.translate(self.name)
 
 
 @implementer(IAdminNavigationRegistry)
 class NavigationRegistry(object):
 
     _base_entries = [
         NavEntry('global', _('Global'), 'admin_settings_global'),
         NavEntry('vcs', _('VCS'), 'admin_settings_vcs'),
         NavEntry('visual', _('Visual'), 'admin_settings_visual'),
         NavEntry('mapping', _('Remap and Rescan'), 'admin_settings_mapping'),
         NavEntry('issuetracker', _('Issue Tracker'),
                  'admin_settings_issuetracker'),
         NavEntry('email', _('Email'), 'admin_settings_email'),
         NavEntry('hooks', _('Hooks'), 'admin_settings_hooks'),
         NavEntry('search', _('Full Text Search'), 'admin_settings_search'),
 
         NavEntry('integrations', _('Integrations'),
                  'global_integrations_home', pyramid=True),
         NavEntry('system', _('System Info'),
                  'admin_settings_system', pyramid=True),
         NavEntry('sessions', _('User Sessions'),
                  'admin_settings_sessions', pyramid=True),
         NavEntry('open_source', _('Open Source Licenses'),
                  'admin_settings_open_source', pyramid=True),
 
         # TODO: marcink: we disable supervisor now until the supervisor stats
         # page is fixed in the nix configuration
         # NavEntry('supervisor', _('Supervisor'), 'admin_settings_supervisor'),
     ]
 
     _labs_entry = NavEntry('labs', _('Labs'), 'admin_settings_labs')
 
     def __init__(self, labs_active=False):
         self._registered_entries = collections.OrderedDict([
             (item.key, item) for item in self.__class__._base_entries
         ])
 
         if labs_active:
             self.add_entry(self._labs_entry)
 
     def add_entry(self, entry):
         self._registered_entries[entry.key] = entry
 
     def get_navlist(self, request):
         navlist = [NavListEntry(i.key, i.get_localized_name(request),
                                 i.generate_url(request))
                    for i in self._registered_entries.values()]
         return navlist
 
 
 def navigation_registry(request):
     """
     Helper that returns the admin navigation registry.
     """
     pyramid_registry = get_registry(request)
     nav_registry = pyramid_registry.queryUtility(IAdminNavigationRegistry)
     return nav_registry
 
 
 def navigation_list(request):
     """
     Helper that returns the admin navigation as list of NavListEntry objects.
     """
     return navigation_registry(request).get_navlist(request)

rhodecode/apps/admin/views/open_source_licenses.py

@@ -1,48 +1,48 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import collections
 import logging
 
 from pylons import tmpl_context as c
 from pyramid.view import view_config
 
-from rhodecode.admin.views.base import AdminSettingsView
-from rhodecode.admin.navigation import navigation_list
+from rhodecode.apps._base import BaseAppView
+from rhodecode.apps.admin.navigation import navigation_list
 from rhodecode.lib.auth import (LoginRequired, HasPermissionAllDecorator)
 from rhodecode.lib.utils import read_opensource_licenses
 
 log = logging.getLogger(__name__)
 
 
-class OpenSourceLicensesAdminSettingsView(AdminSettingsView):
+class OpenSourceLicensesAdminSettingsView(BaseAppView):
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_open_source', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def open_source_licenses(self):
         c.active = 'open_source'
         c.navlist = navigation_list(self.request)
         c.opensource_licenses = collections.OrderedDict(
             sorted(read_opensource_licenses().items(), key=lambda t: t[0]))
 
         return {}

rhodecode/apps/admin/views/sessions.py

@@ -1,100 +1,96 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 
 from pylons import tmpl_context as c
 from pyramid.view import view_config
 from pyramid.httpexceptions import HTTPFound
 
-from rhodecode.translation import _
-
-from rhodecode.admin.views.base import AdminSettingsView
+from rhodecode.apps._base import BaseAppView
+from rhodecode.apps.admin.navigation import navigation_list
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator, CSRFRequired)
 from rhodecode.lib.utils2 import safe_int
 from rhodecode.lib import system_info
 from rhodecode.lib import user_sessions
 
 
-from rhodecode.admin.navigation import navigation_list
-
-
 log = logging.getLogger(__name__)
 
 
-class AdminSessionSettingsView(AdminSettingsView):
+class AdminSessionSettingsView(BaseAppView):
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_sessions', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_sessions(self):
         c.active = 'sessions'
         c.navlist = navigation_list(self.request)
 
         c.cleanup_older_days = 60
         older_than_seconds = 60 * 60 * 24 * c.cleanup_older_days
 
         config = system_info.rhodecode_config().get_value()['value']['config']
         c.session_model = user_sessions.get_session_handler(
             config.get('beaker.session.type', 'memory'))(config)
 
         c.session_conf = c.session_model.config
         c.session_count = c.session_model.get_count()
         c.session_expired_count = c.session_model.get_expired_count(
             older_than_seconds)
 
         return {}
 
     @LoginRequired()
     @CSRFRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_sessions_cleanup', request_method='POST')
     def settings_sessions_cleanup(self):
         _ = self.request.translate
         expire_days = safe_int(self.request.params.get('expire_days'))
 
         if expire_days is None:
             expire_days = 60
 
         older_than_seconds = 60 * 60 * 24 * expire_days
 
         config = system_info.rhodecode_config().get_value()['value']['config']
         session_model = user_sessions.get_session_handler(
             config.get('beaker.session.type', 'memory'))(config)
 
         try:
             session_model.clean_sessions(
                 older_than_seconds=older_than_seconds)
             self.request.session.flash(
                 _('Cleaned up old sessions'), queue='success')
         except user_sessions.CleanupCommand as msg:
             self.request.session.flash(msg.message, queue='warning')
         except Exception as e:
             log.exception('Failed session cleanup')
             self.request.session.flash(
                 _('Failed to cleanup up old sessions'), queue='error')
 
         redirect_to = self.request.resource_path(
             self.context, route_name='admin_settings_sessions')
         return HTTPFound(redirect_to)

rhodecode/apps/admin/views/svn_config.py

@@ -1,60 +1,59 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 
 from pyramid.view import view_config
 
-from rhodecode.svn_support.utils import generate_mod_dav_svn_config
-
-from rhodecode.admin.views.base import AdminSettingsView
+from rhodecode.apps._base import BaseAppView
+from rhodecode.apps.svn_support.utils import generate_mod_dav_svn_config
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator, CSRFRequired)
 
 log = logging.getLogger(__name__)
 
 
-class SvnConfigAdminSettingsView(AdminSettingsView):
+class SvnConfigAdminSettingsView(BaseAppView):
 
     @LoginRequired()
     @CSRFRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_vcs_svn_generate_cfg',
         request_method='POST', renderer='json')
     def vcs_svn_generate_config(self):
         _ = self.request.translate
         try:
             generate_mod_dav_svn_config(self.request.registry)
             msg = {
                 'message': _('Apache configuration for Subversion generated.'),
                 'level': 'success',
             }
         except Exception:
             log.exception(
                 'Exception while generating the Apache '
                 'configuration for Subversion.')
             msg = {
                 'message': _('Failed to generate the Apache configuration for Subversion.'),
                 'level': 'error',
             }
 
         data = {'message': msg}
         return data

rhodecode/apps/admin/views/system_info.py

@@ -1,202 +1,203 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 import urllib2
 import packaging.version
 
 from pylons import tmpl_context as c
 from pyramid.view import view_config
 
 import rhodecode
+from rhodecode.apps._base import BaseAppView
+from rhodecode.apps.admin.navigation import navigation_list
 from rhodecode.lib import helpers as h
 from rhodecode.lib.auth import (LoginRequired, HasPermissionAllDecorator)
 from rhodecode.lib.utils2 import str2bool
 from rhodecode.lib import system_info
 from rhodecode.lib.ext_json import json
-
-from rhodecode.admin.views.base import AdminSettingsView
-from rhodecode.admin.navigation import navigation_list
 from rhodecode.model.settings import SettingsModel
 
 log = logging.getLogger(__name__)
 
 
-class AdminSystemInfoSettingsView(AdminSettingsView):
+class AdminSystemInfoSettingsView(BaseAppView):
 
     @staticmethod
     def get_update_data(update_url):
         """Return the JSON update data."""
         ver = rhodecode.__version__
         log.debug('Checking for upgrade on `%s` server', update_url)
         opener = urllib2.build_opener()
         opener.addheaders = [('User-agent', 'RhodeCode-SCM/%s' % ver)]
         response = opener.open(update_url)
         response_data = response.read()
         data = json.loads(response_data)
 
         return data
 
     def get_update_url(self):
         settings = SettingsModel().get_all_settings()
         return settings.get('rhodecode_update_url')
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_system', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_system_info(self):
         _ = self.request.translate
 
         c.active = 'system'
         c.navlist = navigation_list(self.request)
 
         # TODO(marcink), figure out how to allow only selected users to do this
         c.allowed_to_snapshot = self._rhodecode_user.admin
 
         snapshot = str2bool(self.request.params.get('snapshot'))
 
         c.rhodecode_update_url = self.get_update_url()
         server_info = system_info.get_system_info(self.request.environ)
 
         for key, val in server_info.items():
             setattr(c, key, val)
 
         def val(name, subkey='human_value'):
             return server_info[name][subkey]
 
         def state(name):
             return server_info[name]['state']
 
         def val2(name):
             val = server_info[name]['human_value']
             state = server_info[name]['state']
             return val, state
 
         update_info_msg = _('Note: please make sure this server can '
                             'access `${url}` for the update link to work',
                             mapping=dict(url=c.rhodecode_update_url))
         c.data_items = [
             # update info
             (_('Update info'), h.literal(
                 '<span class="link" id="check_for_update" >%s.</span>' % (
                 _('Check for updates')) +
                 '<br/> <span >%s.</span>' % (update_info_msg)
             ), ''),
 
             # RhodeCode specific
             (_('RhodeCode Version'), val('rhodecode_app')['text'], state('rhodecode_app')),
             (_('RhodeCode Server IP'), val('server')['server_ip'], state('server')),
             (_('RhodeCode Server ID'), val('server')['server_id'], state('server')),
             (_('RhodeCode Configuration'), val('rhodecode_config')['path'], state('rhodecode_config')),
+            (_('Workers'), val('rhodecode_config')['config']['server:main'].get('workers', '?'), state('rhodecode_config')),
+            (_('Worker Type'), val('rhodecode_config')['config']['server:main'].get('worker_class', 'sync'), state('rhodecode_config')),
             ('', '', ''),  # spacer
 
             # Database
             (_('Database'), val('database')['url'], state('database')),
             (_('Database version'), val('database')['version'], state('database')),
             ('', '', ''),  # spacer
 
             # Platform/Python
             (_('Platform'), val('platform')['name'], state('platform')),
             (_('Platform UUID'), val('platform')['uuid'], state('platform')),
             (_('Python version'), val('python')['version'], state('python')),
             (_('Python path'), val('python')['executable'], state('python')),
             ('', '', ''),  # spacer
 
             # Systems stats
-            (_('CPU'), val('cpu'), state('cpu')),
+            (_('CPU'), val('cpu')['text'], state('cpu')),
             (_('Load'), val('load')['text'], state('load')),
             (_('Memory'), val('memory')['text'], state('memory')),
             (_('Uptime'), val('uptime')['text'], state('uptime')),
             ('', '', ''),  # spacer
 
             # Repo storage
             (_('Storage location'), val('storage')['path'], state('storage')),
             (_('Storage info'), val('storage')['text'], state('storage')),
             (_('Storage inodes'), val('storage_inodes')['text'], state('storage_inodes')),
 
             (_('Gist storage location'), val('storage_gist')['path'], state('storage_gist')),
             (_('Gist storage info'), val('storage_gist')['text'], state('storage_gist')),
 
             (_('Archive cache storage location'), val('storage_archive')['path'], state('storage_archive')),
             (_('Archive cache info'), val('storage_archive')['text'], state('storage_archive')),
 
             (_('Temp storage location'), val('storage_temp')['path'], state('storage_temp')),
             (_('Temp storage info'), val('storage_temp')['text'], state('storage_temp')),
 
             (_('Search info'), val('search')['text'], state('search')),
             (_('Search location'), val('search')['location'], state('search')),
             ('', '', ''),  # spacer
 
             # VCS specific
             (_('VCS Backends'), val('vcs_backends'), state('vcs_backends')),
             (_('VCS Server'), val('vcs_server')['text'], state('vcs_server')),
             (_('GIT'), val('git'), state('git')),
             (_('HG'), val('hg'), state('hg')),
             (_('SVN'), val('svn'), state('svn')),
 
         ]
 
         if snapshot:
             if c.allowed_to_snapshot:
                 c.data_items.pop(0)  # remove server info
                 self.request.override_renderer = 'admin/settings/settings_system_snapshot.mako'
             else:
                 self.request.session.flash(
                     'You are not allowed to do this', queue='warning')
         return {}
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_system_update', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings_system_update.mako')
     def settings_system_info_check_update(self):
         _ = self.request.translate
 
         update_url = self.get_update_url()
 
         _err = lambda s: '<div style="color:#ff8888; padding:4px 0px">{}</div>'.format(s)
         try:
             data = self.get_update_data(update_url)
         except urllib2.URLError as e:
             log.exception("Exception contacting upgrade server")
             self.request.override_renderer = 'string'
             return _err('Failed to contact upgrade server: %r' % e)
         except ValueError as e:
             log.exception("Bad data sent from update server")
             self.request.override_renderer = 'string'
             return _err('Bad data sent from update server')
 
         latest = data['versions'][0]
 
         c.update_url = update_url
         c.latest_data = latest
         c.latest_ver = latest['version']
         c.cur_ver = rhodecode.__version__
         c.should_upgrade = False
 
         if (packaging.version.Version(c.latest_ver) >
                 packaging.version.Version(c.cur_ver)):
             c.should_upgrade = True
         c.important_notices = latest['general']
 
         return {}

rhodecode/apps/channelstream/__init__.py

@@ -1,88 +1,90 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import os
 
 from pyramid.settings import asbool
 
 from rhodecode.config.routing import ADMIN_PREFIX
 from rhodecode.lib.ext_json import json
 
 
 def url_gen(request):
     registry = request.registry
     longpoll_url = registry.settings.get('channelstream.longpoll_url', '')
     ws_url = registry.settings.get('channelstream.ws_url', '')
     proxy_url = request.route_url('channelstream_proxy')
     urls = {
         'connect': request.route_path('channelstream_connect'),
         'subscribe': request.route_path('channelstream_subscribe'),
         'longpoll': longpoll_url or proxy_url,
         'ws': ws_url or proxy_url.replace('http', 'ws')
     }
     return json.dumps(urls)
 
 
 PLUGIN_DEFINITION = {
     'name': 'channelstream',
     'config': {
         'javascript': [],
         'css': [],
         'template_hooks': {
             'plugin_init_template': 'rhodecode:templates/channelstream/plugin_init.mako'
         },
         'url_gen': url_gen,
         'static': None,
         'enabled': False,
         'server': '',
         'secret': ''
     }
 }
 
 
 def includeme(config):
     settings = config.registry.settings
     PLUGIN_DEFINITION['config']['enabled'] = asbool(
         settings.get('channelstream.enabled'))
     PLUGIN_DEFINITION['config']['server'] = settings.get(
         'channelstream.server', '')
     PLUGIN_DEFINITION['config']['secret'] = settings.get(
         'channelstream.secret', '')
     PLUGIN_DEFINITION['config']['history.location'] = settings.get(
         'channelstream.history.location', '')
     config.register_rhodecode_plugin(
         PLUGIN_DEFINITION['name'],
         PLUGIN_DEFINITION['config']
     )
     # create plugin history location
     history_dir = PLUGIN_DEFINITION['config']['history.location']
     if history_dir and not os.path.exists(history_dir):
         os.makedirs(history_dir, 0750)
 
     config.add_route(
         name='channelstream_connect',
         pattern=ADMIN_PREFIX + '/channelstream/connect')
     config.add_route(
         name='channelstream_subscribe',
         pattern=ADMIN_PREFIX + '/channelstream/subscribe')
     config.add_route(
         name='channelstream_proxy',
         pattern=settings.get('channelstream.proxy_path') or '/_channelstream')
-    config.scan('rhodecode.channelstream')
+
+    # Scan module for configuration decorators.
+    config.scan()

rhodecode/apps/login/tests/test_register_captcha.py

@@ -1,128 +1,133 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import mock
 import pytest
 
+from rhodecode.apps.login.views import LoginView, CaptchaData
 from rhodecode.config.routing import ADMIN_PREFIX
-from rhodecode.login.views import LoginView, CaptchaData
+from rhodecode.lib.utils2 import AttributeDict
 from rhodecode.model.settings import SettingsModel
 from rhodecode.tests.utils import AssertResponse
 
 
 class RhodeCodeSetting(object):
     def __init__(self, name, value):
         self.name = name
         self.value = value
 
     def __enter__(self):
         from rhodecode.model.settings import SettingsModel
         model = SettingsModel()
         self.old_setting = model.get_setting_by_name(self.name)
         model.create_or_update_setting(name=self.name, val=self.value)
         return self
 
-    def __exit__(self, type, value, traceback):
+    def __exit__(self, exc_type, exc_val, exc_tb):
         model = SettingsModel()
         if self.old_setting:
             model.create_or_update_setting(
                 name=self.name, val=self.old_setting.app_settings_value)
         else:
             model.create_or_update_setting(name=self.name)
 
 
 class TestRegisterCaptcha(object):
 
     @pytest.mark.parametrize('private_key, public_key, expected', [
         ('',        '',       CaptchaData(False, '',        '')),
         ('',        'pubkey', CaptchaData(False, '',        'pubkey')),
         ('privkey', '',       CaptchaData(True,  'privkey', '')),
         ('privkey', 'pubkey', CaptchaData(True,  'privkey', 'pubkey')),
     ])
-    def test_get_captcha_data(self, private_key, public_key, expected, db):
-        login_view = LoginView(mock.Mock(), mock.Mock())
+    def test_get_captcha_data(self, private_key, public_key, expected, db,
+                              request_stub, user_util):
+        request_stub.user = user_util.create_user().AuthUser
+        request_stub.matched_route = AttributeDict({'name': 'login'})
+        login_view = LoginView(mock.Mock(), request_stub)
+
         with RhodeCodeSetting('captcha_private_key', private_key):
             with RhodeCodeSetting('captcha_public_key', public_key):
                 captcha = login_view._get_captcha_data()
                 assert captcha == expected
 
     @pytest.mark.parametrize('active', [False, True])
     @mock.patch.object(LoginView, '_get_captcha_data')
     def test_private_key_does_not_leak_to_html(
             self, m_get_captcha_data, active, app):
         captcha = CaptchaData(
             active=active, private_key='PRIVATE_KEY', public_key='PUBLIC_KEY')
         m_get_captcha_data.return_value = captcha
 
         response = app.get(ADMIN_PREFIX + '/register')
         assert 'PRIVATE_KEY' not in response
 
     @pytest.mark.parametrize('active', [False, True])
     @mock.patch.object(LoginView, '_get_captcha_data')
     def test_register_view_renders_captcha(
             self, m_get_captcha_data, active, app):
         captcha = CaptchaData(
             active=active, private_key='PRIVATE_KEY', public_key='PUBLIC_KEY')
         m_get_captcha_data.return_value = captcha
 
         response = app.get(ADMIN_PREFIX + '/register')
 
         assertr = AssertResponse(response)
         if active:
             assertr.one_element_exists('#recaptcha_field')
         else:
             assertr.no_element_exists('#recaptcha_field')
 
     @pytest.mark.parametrize('valid', [False, True])
-    @mock.patch('rhodecode.login.views.submit')
+    @mock.patch('rhodecode.apps.login.views.submit')
     @mock.patch.object(LoginView, '_get_captcha_data')
     def test_register_with_active_captcha(
             self, m_get_captcha_data, m_submit, valid, app, csrf_token):
         captcha = CaptchaData(
             active=True, private_key='PRIVATE_KEY', public_key='PUBLIC_KEY')
         m_get_captcha_data.return_value = captcha
         m_response = mock.Mock()
         m_response.is_valid = valid
         m_submit.return_value = m_response
 
         params = {
             'csrf_token': csrf_token,
             'email': '[e-mail]',
             'firstname': 'pytest-firstname',
             'lastname': 'pytest-lastname',
             'password': 'secret',
             'password_confirmation': 'secret',
             'username': 'pytest',
         }
         response = app.post(ADMIN_PREFIX + '/register', params=params)
 
         if valid:
             # If we provided a valid captcha input we expect a successful
             # registration and redirect to the login page.
             assert response.status_int == 302
             assert 'location' in response.headers
             assert ADMIN_PREFIX + '/login' in response.headers['location']
         else:
             # If captche input is invalid we expect to stay on the registration
             # page with an error message displayed.
             assertr = AssertResponse(response)
             assert response.status_int == 200
             assertr.one_element_exists('#recaptcha_field ~ span.error-message')

rhodecode/apps/login/views.py

@@ -1,358 +1,396 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
+import time
 import collections
 import datetime
 import formencode
 import logging
 import urlparse
 
 from pylons import url
 from pyramid.httpexceptions import HTTPFound
 from pyramid.view import view_config
 from recaptcha.client.captcha import submit
 
+from rhodecode.apps._base import BaseAppView
 from rhodecode.authentication.base import authenticate, HTTP_TYPE
 from rhodecode.events import UserRegistered
 from rhodecode.lib import helpers as h
 from rhodecode.lib.auth import (
     AuthUser, HasPermissionAnyDecorator, CSRFRequired)
 from rhodecode.lib.base import get_ip_addr
 from rhodecode.lib.exceptions import UserCreationError
 from rhodecode.lib.utils2 import safe_str
-from rhodecode.model.db import User
+from rhodecode.model.db import User, UserApiKeys
 from rhodecode.model.forms import LoginForm, RegisterForm, PasswordResetForm
 from rhodecode.model.meta import Session
+from rhodecode.model.auth_token import AuthTokenModel
 from rhodecode.model.settings import SettingsModel
 from rhodecode.model.user import UserModel
 from rhodecode.translation import _
 
 
 log = logging.getLogger(__name__)
 
 CaptchaData = collections.namedtuple(
     'CaptchaData', 'active, private_key, public_key')
 
 
 def _store_user_in_session(session, username, remember=False):
     user = User.get_by_username(username, case_insensitive=True)
     auth_user = AuthUser(user.user_id)
     auth_user.set_authenticated()
     cs = auth_user.get_cookie_store()
     session['rhodecode_user'] = cs
     user.update_lastlogin()
     Session().commit()
 
     # If they want to be remembered, update the cookie
     if remember:
         _year = (datetime.datetime.now() +
                  datetime.timedelta(seconds=60 * 60 * 24 * 365))
         session._set_cookie_expires(_year)
 
     session.save()
 
     safe_cs = cs.copy()
     safe_cs['password'] = '****'
     log.info('user %s is now authenticated and stored in '
              'session, session attrs %s', username, safe_cs)
 
     # dumps session attrs back to cookie
     session._update_cookie_out()
     # we set new cookie
     headers = None
     if session.request['set_cookie']:
         # send set-cookie headers back to response to update cookie
         headers = [('Set-Cookie', session.request['cookie_out'])]
     return headers
 
 
 def get_came_from(request):
     came_from = safe_str(request.GET.get('came_from', ''))
     parsed = urlparse.urlparse(came_from)
     allowed_schemes = ['http', 'https']
     if parsed.scheme and parsed.scheme not in allowed_schemes:
         log.error('Suspicious URL scheme detected %s for url %s' %
                   (parsed.scheme, parsed))
         came_from = url('home')
     elif parsed.netloc and request.host != parsed.netloc:
         log.error('Suspicious NETLOC detected %s for url %s server url '
                   'is: %s' % (parsed.netloc, parsed, request.host))
         came_from = url('home')
     elif any(bad_str in parsed.path for bad_str in ('\r', '\n')):
         log.error('Header injection detected `%s` for url %s server url ' %
                   (parsed.path, parsed))
         came_from = url('home')
 
     return came_from or url('home')
 
 
-class LoginView(object):
+class LoginView(BaseAppView):
 
-    def __init__(self, context, request):
-        self.request = request
-        self.context = context
-        self.session = request.session
-        self._rhodecode_user = request.user
-
-    def _get_template_context(self):
-        return {
-            'came_from': get_came_from(self.request),
-            'defaults': {},
-            'errors': {},
-        }
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+        c.came_from = get_came_from(self.request)
+        self._register_global_c(c)
+        return c
 
     def _get_captcha_data(self):
         settings = SettingsModel().get_all_settings()
         private_key = settings.get('rhodecode_captcha_private_key')
         public_key = settings.get('rhodecode_captcha_public_key')
         active = bool(private_key)
         return CaptchaData(
             active=active, private_key=private_key, public_key=public_key)
 
     @view_config(
         route_name='login', request_method='GET',
         renderer='rhodecode:templates/login.mako')
     def login(self):
-        came_from = get_came_from(self.request)
-        user = self.request.user
+        c = self.load_default_context()
+        auth_user = self._rhodecode_user
 
         # redirect if already logged in
-        if user.is_authenticated and not user.is_default and user.ip_allowed:
-            raise HTTPFound(came_from)
+        if (auth_user.is_authenticated and
+                not auth_user.is_default and auth_user.ip_allowed):
+            raise HTTPFound(c.came_from)
 
         # check if we use headers plugin, and try to login using it.
         try:
             log.debug('Running PRE-AUTH for headers based authentication')
             auth_info = authenticate(
                 '', '', self.request.environ, HTTP_TYPE, skip_missing=True)
             if auth_info:
                 headers = _store_user_in_session(
                     self.session, auth_info.get('username'))
-                raise HTTPFound(came_from, headers=headers)
+                raise HTTPFound(c.came_from, headers=headers)
         except UserCreationError as e:
             log.error(e)
             self.session.flash(e, queue='error')
 
-        return self._get_template_context()
+        return self._get_template_context(c)
 
     @view_config(
         route_name='login', request_method='POST',
         renderer='rhodecode:templates/login.mako')
     def login_post(self):
-        came_from = get_came_from(self.request)
+        c = self.load_default_context()
 
         login_form = LoginForm()()
 
         try:
             self.session.invalidate()
             form_result = login_form.to_python(self.request.params)
             # form checks for username/password, now we're authenticated
             headers = _store_user_in_session(
                 self.session,
                 username=form_result['username'],
                 remember=form_result['remember'])
-            log.debug('Redirecting to "%s" after login.', came_from)
-            raise HTTPFound(came_from, headers=headers)
+            log.debug('Redirecting to "%s" after login.', c.came_from)
+            raise HTTPFound(c.came_from, headers=headers)
         except formencode.Invalid as errors:
             defaults = errors.value
             # remove password from filling in form again
             defaults.pop('password', None)
-            render_ctx = self._get_template_context()
+            render_ctx = self._get_template_context(c)
             render_ctx.update({
                 'errors': errors.error_dict,
                 'defaults': defaults,
             })
             return render_ctx
 
         except UserCreationError as e:
             # headers auth or other auth functions that create users on
             # the fly can throw this exception signaling that there's issue
             # with user creation, explanation should be provided in
             # Exception itself
             self.session.flash(e, queue='error')
-            return self._get_template_context()
+            return self._get_template_context(c)
 
     @CSRFRequired()
     @view_config(route_name='logout', request_method='POST')
     def logout(self):
-        user = self.request.user
-        log.info('Deleting session for user: `%s`', user)
+        auth_user = self._rhodecode_user
+        log.info('Deleting session for user: `%s`', auth_user)
         self.session.delete()
         return HTTPFound(url('home'))
 
     @HasPermissionAnyDecorator(
         'hg.admin', 'hg.register.auto_activate', 'hg.register.manual_activate')
     @view_config(
         route_name='register', request_method='GET',
         renderer='rhodecode:templates/register.mako',)
     def register(self, defaults=None, errors=None):
+        c = self.load_default_context()
         defaults = defaults or {}
         errors = errors or {}
 
         settings = SettingsModel().get_all_settings()
         register_message = settings.get('rhodecode_register_message') or ''
         captcha = self._get_captcha_data()
         auto_active = 'hg.register.auto_activate' in User.get_default_user()\
             .AuthUser.permissions['global']
 
-        render_ctx = self._get_template_context()
+        render_ctx = self._get_template_context(c)
         render_ctx.update({
             'defaults': defaults,
             'errors': errors,
             'auto_active': auto_active,
             'captcha_active': captcha.active,
             'captcha_public_key': captcha.public_key,
             'register_message': register_message,
         })
         return render_ctx
 
     @HasPermissionAnyDecorator(
         'hg.admin', 'hg.register.auto_activate', 'hg.register.manual_activate')
     @view_config(
         route_name='register', request_method='POST',
         renderer='rhodecode:templates/register.mako')
     def register_post(self):
         captcha = self._get_captcha_data()
         auto_active = 'hg.register.auto_activate' in User.get_default_user()\
             .AuthUser.permissions['global']
 
         register_form = RegisterForm()()
         try:
             form_result = register_form.to_python(self.request.params)
             form_result['active'] = auto_active
 
             if captcha.active:
                 response = submit(
                     self.request.params.get('recaptcha_challenge_field'),
                     self.request.params.get('recaptcha_response_field'),
                     private_key=captcha.private_key,
                     remoteip=get_ip_addr(self.request.environ))
                 if not response.is_valid:
                     _value = form_result
                     _msg = _('Bad captcha')
                     error_dict = {'recaptcha_field': _msg}
                     raise formencode.Invalid(_msg, _value, None,
                                              error_dict=error_dict)
 
             new_user = UserModel().create_registration(form_result)
             event = UserRegistered(user=new_user, session=self.session)
             self.request.registry.notify(event)
             self.session.flash(
                 _('You have successfully registered with RhodeCode'),
                 queue='success')
             Session().commit()
 
             redirect_ro = self.request.route_path('login')
             raise HTTPFound(redirect_ro)
 
         except formencode.Invalid as errors:
             errors.value.pop('password', None)
             errors.value.pop('password_confirmation', None)
             return self.register(
                 defaults=errors.value, errors=errors.error_dict)
 
         except UserCreationError as e:
             # container auth or other auth functions that create users on
             # the fly can throw this exception signaling that there's issue
             # with user creation, explanation should be provided in
             # Exception itself
             self.session.flash(e, queue='error')
             return self.register()
 
     @view_config(
         route_name='reset_password', request_method=('GET', 'POST'),
         renderer='rhodecode:templates/password_reset.mako')
     def password_reset(self):
         captcha = self._get_captcha_data()
 
         render_ctx = {
             'captcha_active': captcha.active,
             'captcha_public_key': captcha.public_key,
             'defaults': {},
             'errors': {},
         }
 
+        # always send implicit message to prevent from discovery of
+        # matching emails
+        msg = _('If such email exists, a password reset link was sent to it.')
+
         if self.request.POST:
+            if h.HasPermissionAny('hg.password_reset.disabled')():
+                _email = self.request.POST.get('email', '')
+                log.error('Failed attempt to reset password for `%s`.', _email)
+                self.session.flash(_('Password reset has been disabled.'),
+                                   queue='error')
+                return HTTPFound(self.request.route_path('reset_password'))
+
             password_reset_form = PasswordResetForm()()
             try:
                 form_result = password_reset_form.to_python(
                     self.request.params)
-                if h.HasPermissionAny('hg.password_reset.disabled')():
-                    log.error('Failed attempt to reset password for %s.', form_result['email'] )
-                    self.session.flash(
-                        _('Password reset has been disabled.'),
-                        queue='error')
-                    return HTTPFound(self.request.route_path('reset_password'))
+                user_email = form_result['email']
+
                 if captcha.active:
                     response = submit(
                         self.request.params.get('recaptcha_challenge_field'),
                         self.request.params.get('recaptcha_response_field'),
                         private_key=captcha.private_key,
                         remoteip=get_ip_addr(self.request.environ))
                     if not response.is_valid:
                         _value = form_result
                         _msg = _('Bad captcha')
                         error_dict = {'recaptcha_field': _msg}
-                        raise formencode.Invalid(_msg, _value, None,
-                                                 error_dict=error_dict)
+                        raise formencode.Invalid(
+                            _msg, _value, None, error_dict=error_dict)
 
                 # Generate reset URL and send mail.
-                user_email = form_result['email']
                 user = User.get_by_email(user_email)
+
+                # generate password reset token that expires in 10minutes
+                desc = 'Generated token for password reset from {}'.format(
+                    datetime.datetime.now().isoformat())
+                reset_token = AuthTokenModel().create(
+                    user, lifetime=10,
+                    description=desc,
+                    role=UserApiKeys.ROLE_PASSWORD_RESET)
+                Session().commit()
+
+                log.debug('Successfully created password recovery token')
                 password_reset_url = self.request.route_url(
                     'reset_password_confirmation',
-                    _query={'key': user.api_key})
+                    _query={'key': reset_token.api_key})
                 UserModel().reset_password_link(
                     form_result, password_reset_url)
-
                 # Display success message and redirect.
-                self.session.flash(
-                    _('Your password reset link was sent'),
-                    queue='success')
-                return HTTPFound(self.request.route_path('login'))
+                self.session.flash(msg, queue='success')
+                return HTTPFound(self.request.route_path('reset_password'))
 
             except formencode.Invalid as errors:
                 render_ctx.update({
                     'defaults': errors.value,
                     'errors': errors.error_dict,
                 })
+                if not self.request.params.get('email'):
+                    # case of empty email, we want to report that
+                    return render_ctx
+
+                if 'recaptcha_field' in errors.error_dict:
+                    # case of failed captcha
+                    return render_ctx
+
+            log.debug('faking response on invalid password reset')
+            # make this take 2s, to prevent brute forcing.
+            time.sleep(2)
+            self.session.flash(msg, queue='success')
+            return HTTPFound(self.request.route_path('reset_password'))
 
         return render_ctx
 
     @view_config(route_name='reset_password_confirmation',
                  request_method='GET')
     def password_reset_confirmation(self):
+
         if self.request.GET and self.request.GET.get('key'):
+            # make this take 2s, to prevent brute forcing.
+            time.sleep(2)
+
+            token = AuthTokenModel().get_auth_token(
+                self.request.GET.get('key'))
+
+            # verify token is the correct role
+            if token is None or token.role != UserApiKeys.ROLE_PASSWORD_RESET:
+                log.debug('Got token with role:%s expected is %s',
+                          getattr(token, 'role', 'EMPTY_TOKEN'),
+                          UserApiKeys.ROLE_PASSWORD_RESET)
+                self.session.flash(
+                    _('Given reset token is invalid'), queue='error')
+                return HTTPFound(self.request.route_path('reset_password'))
+
             try:
-                user = User.get_by_auth_token(self.request.GET.get('key'))
-                password_reset_url = self.request.route_url(
-                    'reset_password_confirmation',
-                    _query={'key': user.api_key})
-                data = {'email': user.email}
-                UserModel().reset_password(data, password_reset_url)
+                owner = token.user
+                data = {'email': owner.email, 'token': token.api_key}
+                UserModel().reset_password(data)
                 self.session.flash(
                     _('Your password reset was successful, '
                       'a new password has been sent to your email'),
                     queue='success')
             except Exception as e:
                 log.error(e)
                 return HTTPFound(self.request.route_path('reset_password'))
 
         return HTTPFound(self.request.route_path('login'))

rhodecode/apps/svn_support/tests/test_mod_dav_svn_config.py

@@ -1,107 +1,107 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import mock
 import pytest
 import re
 
 from pyramid import testing
 
-from rhodecode.svn_support import utils
+from rhodecode.apps.svn_support import utils
 
 
 class TestModDavSvnConfig(object):
 
     @classmethod
     def setup_class(cls):
         # Make mako renderer available in tests.
         config = testing.setUp()
         config.include('pyramid_mako')
 
         cls.location_root = u'/location/root/çµäö'
         cls.parent_path_root = u'/parent/path/çµäö'
         cls.realm = u'Dummy Realm (äöüçµ)'
 
     @classmethod
     def get_repo_group_mocks(cls, count=1):
         repo_groups = []
         for num in range(0, count):
             full_path = u'/path/to/RepöGröúp-°µ {}'.format(num)
             repo_group_mock = mock.MagicMock()
             repo_group_mock.full_path = full_path
             repo_group_mock.full_path_splitted = full_path.split('/')
             repo_groups.append(repo_group_mock)
         return repo_groups
 
     def assert_root_location_directive(self, config):
         pattern = u'<Location "{location}">'.format(
             location=self.location_root)
         assert len(re.findall(pattern, config)) == 1
 
     def assert_group_location_directive(self, config, group_path):
         pattern = u'<Location "{location}{group_path}">'.format(
             location=self.location_root, group_path=group_path)
         assert len(re.findall(pattern, config)) == 1
 
     def test_render_mod_dav_svn_config(self):
         repo_groups = self.get_repo_group_mocks(count=10)
         generated_config = utils._render_mod_dav_svn_config(
             parent_path_root=self.parent_path_root,
             list_parent_path=True,
             location_root=self.location_root,
             repo_groups=repo_groups,
             realm=self.realm,
             use_ssl=True
         )
         # Assert that one location directive exists for each repository group.
         for group in repo_groups:
             self.assert_group_location_directive(
                 generated_config, group.full_path)
 
         # Assert that the root location directive exists.
         self.assert_root_location_directive(generated_config)
 
     @pytest.mark.parametrize('list_parent_path', [True, False])
     @pytest.mark.parametrize('use_ssl', [True, False])
     def test_list_parent_path(self, list_parent_path, use_ssl):
         generated_config = utils._render_mod_dav_svn_config(
             parent_path_root=self.parent_path_root,
             list_parent_path=list_parent_path,
             location_root=self.location_root,
             repo_groups=self.get_repo_group_mocks(count=10),
             realm=self.realm,
             use_ssl=use_ssl
         )
 
         # Assert that correct configuration directive is present.
         if list_parent_path:
             assert not re.search('SVNListParentPath\s+Off', generated_config)
             assert re.search('SVNListParentPath\s+On', generated_config)
         else:
             assert re.search('SVNListParentPath\s+Off', generated_config)
             assert not re.search('SVNListParentPath\s+On', generated_config)
 
         if use_ssl:
             assert 'RequestHeader edit Destination ^https: http: early' \
                    in generated_config
         else:
             assert '#RequestHeader edit Destination ^https: http: early' \
                    in generated_config

rhodecode/apps/svn_support/utils.py

@@ -1,93 +1,93 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2017 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import codecs
 import logging
 import os
 from pyramid.renderers import render
 
 from rhodecode.events import trigger
 from rhodecode.lib.utils import get_rhodecode_realm, get_rhodecode_base_path
 from rhodecode.lib.utils2 import str2bool
 from rhodecode.model.db import RepoGroup
 
 from . import config_keys
 from .events import ModDavSvnConfigChange
 
 
 log = logging.getLogger(__name__)
 
 
 def generate_mod_dav_svn_config(registry):
     """
     Generate the configuration file for use with subversion's mod_dav_svn
     module. The configuration has to contain a <Location> block for each
     available repository group because the mod_dav_svn module does not support
     repositories organized in sub folders.
     """
     settings = registry.settings
     use_ssl = str2bool(registry.settings['force_https'])
 
     config = _render_mod_dav_svn_config(
         use_ssl=use_ssl,
         parent_path_root=get_rhodecode_base_path(),
         list_parent_path=settings[config_keys.list_parent_path],
         location_root=settings[config_keys.location_root],
         repo_groups=RepoGroup.get_all_repo_groups(),
         realm=get_rhodecode_realm())
     _write_mod_dav_svn_config(config, settings[config_keys.config_file_path])
 
     # Trigger an event on mod dav svn configuration change.
     trigger(ModDavSvnConfigChange(), registry)
 
 
 def _render_mod_dav_svn_config(
         parent_path_root, list_parent_path, location_root, repo_groups, realm,
         use_ssl):
     """
     Render mod_dav_svn configuration to string.
     """
     repo_group_paths = []
     for repo_group in repo_groups:
         group_path = repo_group.full_path_splitted
         location = os.path.join(location_root, *group_path)
         parent_path = os.path.join(parent_path_root, *group_path)
         repo_group_paths.append((location, parent_path))
 
     context = {
         'location_root': location_root,
         'parent_path_root': parent_path_root,
         'repo_group_paths': repo_group_paths,
         'svn_list_parent_path': list_parent_path,
         'rhodecode_realm': realm,
         'use_https': use_ssl
     }
 
     # Render the configuration template to string.
-    template = 'rhodecode:svn_support/templates/mod-dav-svn.conf.mako'
+    template = 'rhodecode:apps/svn_support/templates/mod-dav-svn.conf.mako'
     return render(template, context)
 
 
 def _write_mod_dav_svn_config(config, filepath):
     """
     Write mod_dav_svn config to file.
     """
     with codecs.open(filepath, 'w', encoding='utf-8') as f:
         f.write(config)