release: Merge default into stable for release preparation

docs/admin/svn-http.rst

@@ -0,0 +1,137 b''
+.. _svn-http:
+
+|svn| With Write Over HTTP
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To use |svn| with read/write support over the |svn| HTTP protocol, you have to
+configure the HTTP |svn| backend.
+
+Prerequisites
+=============
+
+- Enable HTTP support inside the admin VCS settings on your |RCE| instance
+- You need to install the following tools on the machine that is running an
+  instance of |RCE|:
+  ``Apache HTTP Server`` and ``mod_dav_svn``.
+
+
+Using Ubuntu 14.04 Distribution as an example execute the following:
+
+.. code-block:: bash
+
+    $ sudo apt-get install apache2 libapache2-mod-svn
+
+Once installed you need to enable ``dav_svn``:
+
+.. code-block:: bash
+
+    $ sudo a2enmod dav_svn
+    $ sudo a2enmod headers
+
+
+Configuring Apache Setup
+========================
+
+.. tip::
+
+   It is recommended to run Apache on a port other than 80, due to possible
+   conflicts with other HTTP servers like nginx. To do this, set the
+   ``Listen`` parameter in the ``/etc/apache2/ports.conf`` file, for example
+   ``Listen 8090``.
+
+
+.. warning::
+
+   Make sure your Apache instance which runs the mod_dav_svn module is
+   only accessible by |RCE|. Otherwise everyone is able to browse
+   the repositories or run subversion operations (checkout/commit/etc.).
+
+It is also recommended to run apache as the same user as |RCE|, otherwise
+permission issues could occur. To do this edit the ``/etc/apache2/envvars``
+
+   .. code-block:: apache
+
+      export APACHE_RUN_USER=rhodecode
+      export APACHE_RUN_GROUP=rhodecode
+
+1. To configure Apache, create and edit a virtual hosts file, for example
+   :file:`/etc/apache2/sites-available/default.conf`. Below is an example
+   how to use one with auto-generated config ```mod_dav_svn.conf```
+   from configured |RCE| instance.
+
+.. code-block:: apache
+
+    <VirtualHost *:8090>
+        ServerAdmin rhodecode-admin@localhost
+        DocumentRoot /var/www/html
+        ErrorLog ${'${APACHE_LOG_DIR}'}/error.log
+        CustomLog ${'${APACHE_LOG_DIR}'}/access.log combined
+        Include /home/user/.rccontrol/enterprise-1/mod_dav_svn.conf
+    </VirtualHost>
+
+
+2. Go to the :menuselection:`Admin --> Settings --> VCS` page, and
+   enable :guilabel:`Proxy Subversion HTTP requests`, and specify the
+   :guilabel:`Subversion HTTP Server URL`.
+
+3. Open the |RCE| configuration file,
+   :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
+
+4. Add the following configuration option in the ``[app:main]``
+   section if you don't have it yet.
+
+   This enables mapping of the created |RCE| repo groups into special
+   |svn| paths. Each time a new repository group is created, the system will
+   update the template file and create new mapping. Apache web server needs to
+   be reloaded to pick up the changes on this file.
+   To do this, simply configure `svn.proxy.reload_cmd` inside the .ini file.
+   Example configuration:
+
+
+.. code-block:: ini
+
+    ############################################################
+    ### Subversion proxy support (mod_dav_svn)               ###
+    ### Maps RhodeCode repo groups into SVN paths for Apache ###
+    ############################################################
+    ## Enable or disable the config file generation.
+    svn.proxy.generate_config = true
+    ## Generate config file with `SVNListParentPath` set to `On`.
+    svn.proxy.list_parent_path = true
+    ## Set location and file name of generated config file.
+    svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
+    ## Used as a prefix to the <Location> block in the generated config file.
+    ## In most cases it should be set to `/`.
+    svn.proxy.location_root = /
+    ## Command to reload the mod dav svn configuration on change.
+    ## Example: `/etc/init.d/apache2 reload`
+    svn.proxy.reload_cmd = /etc/init.d/apache2 reload
+    ## If the timeout expires before the reload command finishes, the command will
+    ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
+    #svn.proxy.reload_timeout = 10
+
+
+This would create a special template file called ```mod_dav_svn.conf```. We
+used that file path in the apache config above inside the Include statement.
+It's also possible to generate the config from the
+:menuselection:`Admin --> Settings --> VCS` page.
+
+
+Using |svn|
+===========
+
+Once |svn| has been enabled on your instance, you can use it with the
+following examples. For more |svn| information, see the `Subversion Red Book`_
+
+.. code-block:: bash
+
+    # To clone a repository
+    svn checkout http://my-svn-server.example.com/my-svn-repo
+
+    # svn commit
+    svn commit
+
+
+.. _Subversion Red Book: http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn
+
+.. _Ask Ubuntu: http://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue
  No newline at end of file

docs/contributing/checklist-pull-request.rst

@@ -0,0 +1,17 b''
+.. _checklist-pull-request:
+
+=======================
+Pull Request Checklists
+=======================
+
+
+
+Checklists for Pull Request
+===========================
+
+
+- Informative description
+- Linear commit history
+- Rebased on top of latest changes
+- Add ticket references. eg fixes #123, references #123 etc.
+

docs/release-notes/release-notes-4.5.0.rst

@@ -0,0 +1,159 b''
+|RCE| 4.5.0 |RNS|
+-----------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2016-12-02
+
+
+New Features
+^^^^^^^^^^^^
+
+- Diffs: re-implemented diff engine. Added: Syntax highlighting inside diffs,
+  new side-by-side view with commenting and live chat. Enabled soft-wrapping of
+  long lines and much improved rendering speed for large diffs.
+- File source view: new file display engine. File view now
+  soft wraps long lines. Double click inside file view show occurrences of
+  clicked item. Added pygments-markdown-lexer for highlighting markdown syntax.
+- Files annotation: Added new grouped annotations. Color all related commits
+  by double clicking singe commit in annotation view.
+- Pull request reviewers (EE only): added new default reviewers functionality.
+  Allows picking users or user groups defined as reviewers for new pull request.
+  Picking reviewers can be based on branch name, changed file name patterns or
+  original author of changed source code. eg *.css -> design team.
+  Master branch -> repo owner, fixes #1131.
+- Pull request reviewers: store and show reasons why given person is a reviewer.
+  Manually adding reviewers after creating a PR will now be also indicated
+  together with who added a given person to review.
+- Integrations: Webhooks integration now allows to use variables inside the
+  call URL. Currently supported variables are ${repo_name}, ${repo_type},
+  ${repo_id}, ${repo_url}, ${branch}, ${commit_id}, ${pull_request_id},
+  ${pull_request_url}. Commits are now grouped by branches as well.
+  Allows much easier integration with CI systems.
+- Integrations (EE only): allow wildcard * project key in Jira integration
+  settings to allow referencing multiple projects per commit, fixes #4267.
+- Live notifications: RhodeCode sends live notification to online
+  users on certain events and pages. Currently this works on: invite to chat,
+  update pull request, commit/inline comment. Part of live code review system.
+  Allows users to update the reviewed code while doing the review and never
+  miss any updates or comment replies as they happen. Requires channelstream
+  to be enabled.
+- Repository groups: added default personal repository groups. Personal groups
+  are isolated playground for users allowing them to create projects or forks.
+  Adds new setting to automatically create personal repo groups for newly
+  created users. New groups are created from specified pattern, for example
+  /u/{username}. Implements #4003.
+- Security: It's now possible to disable password reset functionality.
+  This is useful for cases when users only use LDAP or similar types of
+  authentication. Implements #3944
+- Pull requests: exposed shadow repositories to end users. Users are now given
+  access to the shadow repository which represents state after merge performed.
+  In this way users or especially CI servers can much easier perform code
+  analysis of the final merged code.
+- Pull requests: My account > pull request page now uses datagrid.
+  It's faster, filterable and sortable. Fixes #4297.
+- Pull requests: delete pull request action was moved from my account
+  into pull request view itself. This is where everyone was looking for it.
+- Pull requests: improve showing failed merges with proper status in pull
+  request page.
+- User groups: overhaul of edit user group page. Added new selector for
+  adding new user group members.
+- Licensing (EE only): exposed unlock link to deactivate users that are over
+  license limit, to unlock full functionality. This might happen when migrating
+  from CE into EE, and your license supports less active users then allowed.
+- Global settings: add a new header/footer template to allow flash filtering.
+  In case a license warning appears and admin wants to hide it for some time.
+  The new template can be used to do this.
+- System info: added create snapshot button to easily generate system state
+  report. Comes in handy for support and reporting. System state holds
+  information such as free disk/memory, CPU load and some of RhodeCode settings.
+- System info: fetch and show vcs settings from vcsserver. Fixes #4276.
+- System info: use real memory usage based on new psutil api available.
+- System info: added info about temporary storage.
+- System info: expose inode limits and usage. Fixes #4282.
+- Ui: added new icon for merge commit.
+
+
+
+General
+^^^^^^^
+
+- Notifications: move all notifications into polymer for consistency.
+  Fixes #4201.
+- Live chat (EE): Improved UI for live-chat. Use Codemirror editor as
+  input for text box.
+- Api: WARNING DEPRECATION, refactor repository group schemas. Fixes #4133.
+  When using create_repo, create_repo_group, update_repo, update_repo_group
+  the *_name parameter now takes full path including sub repository groups.
+  This is the only way to add resource under another repository group.
+  Furthermore giving non-existing path will no longer create the missing
+  structure. This change makes the api more consistent, it better validates
+  the errors in the data sent to given api call.
+- Pull requests: disable subrepo handling on pull requests. It means users can
+  now use more types of repositories with subrepos to create pull requests.
+  Since handling is disabled, repositories behind authentication, or outside
+  of network can be used.
+- VCSServer: fetch backend info from vcsserver including git/hg/svn versions
+  and connection information.
+- Svn support: it's no longer required to put in repo root path to
+  generate mod-dav-svn config. Fixes #4203.
+- Svn support: Add reload command option (svn.proxy.reload_cmd) to ini files.
+  Apache can now be automatically reloaded when the mod_dav_svn config changes.
+- Svn support: Add a view to trigger the (re)generation of Apache mod_dav_svn
+  configuration file. Users are able to generate such file manually by clicking
+  that button.
+- Dependency: updated subversion library to 1.9.
+- Dependency: updated ipython to 5.1.0.
+- Dependency: updated psutil to 4.3.1.
+
+
+Security
+^^^^^^^^
+
+- Hipchat: escape user entered data to avoid xss/formatting problems.
+- VCSServer: obfuscate credentials added into remote url during remote
+  repository creation. Prevents leaking of those credentials inside
+  RhodeCode logs.
+
+
+Performance
+^^^^^^^^^^^
+
+- Diffs: new diff engine is much smarter when it comes to showing huge diffs.
+  The rendering speed should be much improved in such cases, however showing
+  full diff is still supported.
+- VCS backends: when using a repo object from database, re-use this information
+  instead of trying to detect a backend. Reduces the traffic to vcsserver.
+- Pull requests: Add a column to hold the last merge revision. This will skip
+  heavy recalculation of merge state if nothing changed inside a pull request.
+- File source view: don't load the file if it is over the size limit since it
+  won't be displayed anyway. This increases speed of loading the page when a
+  file is above cut-off limit defined.
+
+
+Fixes
+^^^^^
+
+- Users admin: fixed search filter in user admin page.
+- Autocomplete: improve the lookup of users with non-ascii characters. In case
+  of unicode email the previous method could generate wrong data, and
+  make search not show up such users.
+- Svn: added request header downgrade for COPY command to work on
+  https setup. Fixes #4307.
+- Svn: add handling of renamed files inside our generated changes metadata.
+  Fixes #4258.
+- Pull requests: fixed problem with creating pull requests on empty repositories.
+- Events: use branch from previous commit for repo push event commits data so
+  that per-branch grouping works. Fixes #4233.
+- Login: make sure recaptcha data is always validated. Fixes #4279.
+- Vcs: Use commit date as modification time when creating archives.
+  Fixes problem with unstable hashes for archives. Fixes #4247.
+- Issue trackers: fixed bug where saving empty issue tracker via form was
+  causing exception. Fixes #4278.
+- Styling: fixed gravatar size for pull request reviewers.
+- Ldap: fixed email extraction typo. An empty email from LDAP server will now
+  not overwrite the stored one.
+- Integrations: use consistent formatting of users data in Slack integration.
+- Meta-tags: meta tags are not taken into account when truncating descriptions
+  that are too long. Fixes #4305.
  No newline at end of file

licenses/diff_match_patch_license.txt

@@ -0,0 +1,14 b''
+Copyright 2006 Google Inc.
+http://code.google.com/p/google-diff-match-patch/
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

licenses/pygments_markdown_lexer_license.txt

@@ -0,0 +1,12 b''
+    Copyright ©  2015 Jürgen Hermann <[e-mail]>
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+        http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.

rhodecode/lib/codeblocks.py

@@ -0,0 +1,665 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2011-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import logging
+import difflib
+from itertools import groupby
+
+from pygments import lex
+from pygments.formatters.html import _get_ttype_class as pygment_token_class
+from rhodecode.lib.helpers import (
+    get_lexer_for_filenode, get_lexer_safe, html_escape)
+from rhodecode.lib.utils2 import AttributeDict
+from rhodecode.lib.vcs.nodes import FileNode
+from rhodecode.lib.diff_match_patch import diff_match_patch
+from rhodecode.lib.diffs import LimitedDiffContainer
+from pygments.lexers import get_lexer_by_name
+
+plain_text_lexer = get_lexer_by_name(
+    'text', stripall=False, stripnl=False, ensurenl=False)
+
+
+log = logging.getLogger()
+
+
+def filenode_as_lines_tokens(filenode, lexer=None):
+    lexer = lexer or get_lexer_for_filenode(filenode)
+    log.debug('Generating file node pygment tokens for %s, %s', lexer, filenode)
+    tokens = tokenize_string(filenode.content, lexer)
+    lines = split_token_stream(tokens, split_string='\n')
+    rv = list(lines)
+    return rv
+
+
+def tokenize_string(content, lexer):
+    """
+    Use pygments to tokenize some content based on a lexer
+    ensuring all original new lines and whitespace is preserved
+    """
+
+    lexer.stripall = False
+    lexer.stripnl = False
+    lexer.ensurenl = False
+    for token_type, token_text in lex(content, lexer):
+        yield pygment_token_class(token_type), token_text
+
+
+def split_token_stream(tokens, split_string=u'\n'):
+    """
+    Take a list of (TokenType, text) tuples and split them by a string
+
+    >>> split_token_stream([(TEXT, 'some\ntext'), (TEXT, 'more\n')])
+    [(TEXT, 'some'), (TEXT, 'text'),
+     (TEXT, 'more'), (TEXT, 'text')]
+    """
+
+    buffer = []
+    for token_class, token_text in tokens:
+        parts = token_text.split(split_string)
+        for part in parts[:-1]:
+            buffer.append((token_class, part))
+            yield buffer
+            buffer = []
+
+        buffer.append((token_class, parts[-1]))
+
+    if buffer:
+        yield buffer
+
+
+def filenode_as_annotated_lines_tokens(filenode):
+    """
+    Take a file node and return a list of annotations => lines, if no annotation
+    is found, it will be None.
+
+    eg:
+
+    [
+        (annotation1, [
+            (1, line1_tokens_list),
+            (2, line2_tokens_list),
+        ]),
+        (annotation2, [
+            (3, line1_tokens_list),
+        ]),
+        (None, [
+            (4, line1_tokens_list),
+        ]),
+        (annotation1, [
+            (5, line1_tokens_list),
+            (6, line2_tokens_list),
+        ])
+    ]
+    """
+
+    commit_cache = {} # cache commit_getter lookups
+
+    def _get_annotation(commit_id, commit_getter):
+        if commit_id not in commit_cache:
+            commit_cache[commit_id] = commit_getter()
+        return commit_cache[commit_id]
+
+    annotation_lookup = {
+        line_no: _get_annotation(commit_id, commit_getter)
+        for line_no, commit_id, commit_getter, line_content
+        in filenode.annotate
+    }
+
+    annotations_lines = ((annotation_lookup.get(line_no), line_no, tokens)
+                          for line_no, tokens
+                          in enumerate(filenode_as_lines_tokens(filenode), 1))
+
+    grouped_annotations_lines = groupby(annotations_lines, lambda x: x[0])
+
+    for annotation, group in grouped_annotations_lines:
+        yield (
+            annotation, [(line_no, tokens)
+                          for (_, line_no, tokens) in group]
+        )
+
+
+def render_tokenstream(tokenstream):
+    result = []
+    for token_class, token_ops_texts in rollup_tokenstream(tokenstream):
+
+        if token_class:
+            result.append(u'<span class="%s">' % token_class)
+        else:
+            result.append(u'<span>')
+
+        for op_tag, token_text in token_ops_texts:
+
+            if op_tag:
+                result.append(u'<%s>' % op_tag)
+
+            escaped_text = html_escape(token_text)
+
+            # TODO: dan: investigate showing hidden characters like space/nl/tab
+            # escaped_text = escaped_text.replace(' ', '<sp> </sp>')
+            # escaped_text = escaped_text.replace('\n', '<nl>\n</nl>')
+            # escaped_text = escaped_text.replace('\t', '<tab>\t</tab>')
+
+            result.append(escaped_text)
+
+            if op_tag:
+                result.append(u'</%s>' % op_tag)
+
+        result.append(u'</span>')
+
+    html = ''.join(result)
+    return html
+
+
+def rollup_tokenstream(tokenstream):
+    """
+    Group a token stream of the format:
+
+        ('class', 'op', 'text')
+    or
+        ('class', 'text')
+
+    into
+
+        [('class1',
+            [('op1', 'text'),
+             ('op2', 'text')]),
+         ('class2',
+            [('op3', 'text')])]
+
+    This is used to get the minimal tags necessary when
+    rendering to html eg for a token stream ie.
+
+    <span class="A"><ins>he</ins>llo</span>
+    vs
+    <span class="A"><ins>he</ins></span><span class="A">llo</span>
+
+    If a 2 tuple is passed in, the output op will be an empty string.
+
+    eg:
+
+    >>> rollup_tokenstream([('classA', '',      'h'),
+                            ('classA', 'del',   'ell'),
+                            ('classA', '',      'o'),
+                            ('classB', '',      ' '),
+                            ('classA', '',      'the'),
+                            ('classA', '',      're'),
+                            ])
+
+    [('classA', [('', 'h'), ('del', 'ell'), ('', 'o')],
+     ('classB', [('', ' ')],
+     ('classA', [('', 'there')]]
+
+    """
+    if tokenstream and len(tokenstream[0]) == 2:
+        tokenstream = ((t[0], '', t[1]) for t in tokenstream)
+
+    result = []
+    for token_class, op_list in groupby(tokenstream, lambda t: t[0]):
+        ops = []
+        for token_op, token_text_list in groupby(op_list, lambda o: o[1]):
+            text_buffer = []
+            for t_class, t_op, t_text in token_text_list:
+                text_buffer.append(t_text)
+            ops.append((token_op, ''.join(text_buffer)))
+        result.append((token_class, ops))
+    return result
+
+
+def tokens_diff(old_tokens, new_tokens, use_diff_match_patch=True):
+    """
+    Converts a list of (token_class, token_text) tuples to a list of
+    (token_class, token_op, token_text) tuples where token_op is one of
+    ('ins', 'del', '')
+
+    :param old_tokens: list of (token_class, token_text) tuples of old line
+    :param new_tokens: list of (token_class, token_text) tuples of new line
+    :param use_diff_match_patch: boolean, will use google's diff match patch
+        library which has options to 'smooth' out the character by character
+        differences making nicer ins/del blocks
+    """
+
+    old_tokens_result = []
+    new_tokens_result = []
+
+    similarity = difflib.SequenceMatcher(None,
+        ''.join(token_text for token_class, token_text in old_tokens),
+        ''.join(token_text for token_class, token_text in new_tokens)
+    ).ratio()
+
+    if similarity < 0.6: # return, the blocks are too different
+        for token_class, token_text in old_tokens:
+            old_tokens_result.append((token_class, '', token_text))
+        for token_class, token_text in new_tokens:
+            new_tokens_result.append((token_class, '', token_text))
+        return old_tokens_result, new_tokens_result, similarity
+
+    token_sequence_matcher = difflib.SequenceMatcher(None,
+        [x[1] for x in old_tokens],
+        [x[1] for x in new_tokens])
+
+    for tag, o1, o2, n1, n2 in token_sequence_matcher.get_opcodes():
+        # check the differences by token block types first to give a more
+        # nicer "block" level replacement vs character diffs
+
+        if tag == 'equal':
+            for token_class, token_text in old_tokens[o1:o2]:
+                old_tokens_result.append((token_class, '', token_text))
+            for token_class, token_text in new_tokens[n1:n2]:
+                new_tokens_result.append((token_class, '', token_text))
+        elif tag == 'delete':
+            for token_class, token_text in old_tokens[o1:o2]:
+                old_tokens_result.append((token_class, 'del', token_text))
+        elif tag == 'insert':
+            for token_class, token_text in new_tokens[n1:n2]:
+                new_tokens_result.append((token_class, 'ins', token_text))
+        elif tag == 'replace':
+            # if same type token blocks must be replaced, do a diff on the
+            # characters in the token blocks to show individual changes
+
+            old_char_tokens = []
+            new_char_tokens = []
+            for token_class, token_text in old_tokens[o1:o2]:
+                for char in token_text:
+                    old_char_tokens.append((token_class, char))
+
+            for token_class, token_text in new_tokens[n1:n2]:
+                for char in token_text:
+                    new_char_tokens.append((token_class, char))
+
+            old_string = ''.join([token_text for
+                token_class, token_text in old_char_tokens])
+            new_string = ''.join([token_text for
+                token_class, token_text in new_char_tokens])
+
+            char_sequence = difflib.SequenceMatcher(
+                None, old_string, new_string)
+            copcodes = char_sequence.get_opcodes()
+            obuffer, nbuffer = [], []
+
+            if use_diff_match_patch:
+                dmp = diff_match_patch()
+                dmp.Diff_EditCost = 11 # TODO: dan: extract this to a setting
+                reps = dmp.diff_main(old_string, new_string)
+                dmp.diff_cleanupEfficiency(reps)
+
+                a, b = 0, 0
+                for op, rep in reps:
+                    l = len(rep)
+                    if op == 0:
+                        for i, c in enumerate(rep):
+                            obuffer.append((old_char_tokens[a+i][0], '', c))
+                            nbuffer.append((new_char_tokens[b+i][0], '', c))
+                        a += l
+                        b += l
+                    elif op == -1:
+                        for i, c in enumerate(rep):
+                            obuffer.append((old_char_tokens[a+i][0], 'del', c))
+                        a += l
+                    elif op == 1:
+                        for i, c in enumerate(rep):
+                            nbuffer.append((new_char_tokens[b+i][0], 'ins', c))
+                        b += l
+            else:
+                for ctag, co1, co2, cn1, cn2 in copcodes:
+                    if ctag == 'equal':
+                        for token_class, token_text in old_char_tokens[co1:co2]:
+                            obuffer.append((token_class, '', token_text))
+                        for token_class, token_text in new_char_tokens[cn1:cn2]:
+                            nbuffer.append((token_class, '', token_text))
+                    elif ctag == 'delete':
+                        for token_class, token_text in old_char_tokens[co1:co2]:
+                            obuffer.append((token_class, 'del', token_text))
+                    elif ctag == 'insert':
+                        for token_class, token_text in new_char_tokens[cn1:cn2]:
+                            nbuffer.append((token_class, 'ins', token_text))
+                    elif ctag == 'replace':
+                        for token_class, token_text in old_char_tokens[co1:co2]:
+                            obuffer.append((token_class, 'del', token_text))
+                        for token_class, token_text in new_char_tokens[cn1:cn2]:
+                            nbuffer.append((token_class, 'ins', token_text))
+
+            old_tokens_result.extend(obuffer)
+            new_tokens_result.extend(nbuffer)
+
+    return old_tokens_result, new_tokens_result, similarity
+
+
+class DiffSet(object):
+    """
+    An object for parsing the diff result from diffs.DiffProcessor and
+    adding highlighting, side by side/unified renderings and line diffs
+    """
+
+    HL_REAL = 'REAL' # highlights using original file, slow
+    HL_FAST = 'FAST' # highlights using just the line, fast but not correct
+                     # in the case of multiline code
+    HL_NONE = 'NONE' # no highlighting, fastest
+
+    def __init__(self, highlight_mode=HL_REAL, repo_name=None,
+                 source_node_getter=lambda filename: None,
+                 target_node_getter=lambda filename: None,
+                 source_nodes=None, target_nodes=None,
+                 max_file_size_limit=150 * 1024, # files over this size will
+                                                 # use fast highlighting
+                 comments=None,
+                 ):
+
+        self.highlight_mode = highlight_mode
+        self.highlighted_filenodes = {}
+        self.source_node_getter = source_node_getter
+        self.target_node_getter = target_node_getter
+        self.source_nodes = source_nodes or {}
+        self.target_nodes = target_nodes or {}
+        self.repo_name = repo_name
+        self.comments = comments or {}
+        self.max_file_size_limit = max_file_size_limit
+
+    def render_patchset(self, patchset, source_ref=None, target_ref=None):
+        diffset = AttributeDict(dict(
+            lines_added=0,
+            lines_deleted=0,
+            changed_files=0,
+            files=[],
+            limited_diff=isinstance(patchset, LimitedDiffContainer),
+            repo_name=self.repo_name,
+            source_ref=source_ref,
+            target_ref=target_ref,
+        ))
+        for patch in patchset:
+            filediff = self.render_patch(patch)
+            filediff.diffset = diffset
+            diffset.files.append(filediff)
+            diffset.changed_files += 1
+            if not patch['stats']['binary']:
+                diffset.lines_added += patch['stats']['added']
+                diffset.lines_deleted += patch['stats']['deleted']
+
+        return diffset
+
+    _lexer_cache = {}
+    def _get_lexer_for_filename(self, filename):
+        # cached because we might need to call it twice for source/target
+        if filename not in self._lexer_cache:
+            self._lexer_cache[filename] = get_lexer_safe(filepath=filename)
+        return self._lexer_cache[filename]
+
+    def render_patch(self, patch):
+        log.debug('rendering diff for %r' % patch['filename'])
+
+        source_filename = patch['original_filename']
+        target_filename = patch['filename']
+
+        source_lexer = plain_text_lexer
+        target_lexer = plain_text_lexer
+
+        if not patch['stats']['binary']:
+            if self.highlight_mode == self.HL_REAL:
+                if (source_filename and patch['operation'] in ('D', 'M')
+                    and source_filename not in self.source_nodes):
+                        self.source_nodes[source_filename] = (
+                            self.source_node_getter(source_filename))
+
+                if (target_filename and patch['operation'] in ('A', 'M')
+                    and target_filename not in self.target_nodes):
+                        self.target_nodes[target_filename] = (
+                            self.target_node_getter(target_filename))
+
+            elif self.highlight_mode == self.HL_FAST:
+                source_lexer = self._get_lexer_for_filename(source_filename)
+                target_lexer = self._get_lexer_for_filename(target_filename)
+
+        source_file = self.source_nodes.get(source_filename, source_filename)
+        target_file = self.target_nodes.get(target_filename, target_filename)
+
+        source_filenode, target_filenode = None, None
+
+        # TODO: dan: FileNode.lexer works on the content of the file - which
+        # can be slow - issue #4289 explains a lexer clean up - which once
+        # done can allow caching a lexer for a filenode to avoid the file lookup
+        if isinstance(source_file, FileNode):
+            source_filenode = source_file
+            source_lexer = source_file.lexer
+        if isinstance(target_file, FileNode):
+            target_filenode = target_file
+            target_lexer = target_file.lexer
+
+        source_file_path, target_file_path = None, None
+
+        if source_filename != '/dev/null':
+            source_file_path = source_filename
+        if target_filename != '/dev/null':
+            target_file_path = target_filename
+
+        source_file_type = source_lexer.name
+        target_file_type = target_lexer.name
+
+        op_hunks = patch['chunks'][0]
+        hunks = patch['chunks'][1:]
+
+        filediff = AttributeDict({
+            'source_file_path': source_file_path,
+            'target_file_path': target_file_path,
+            'source_filenode': source_filenode,
+            'target_filenode': target_filenode,
+            'hunks': [],
+            'source_file_type': target_file_type,
+            'target_file_type': source_file_type,
+            'patch': patch,
+            'source_mode': patch['stats']['old_mode'],
+            'target_mode': patch['stats']['new_mode'],
+            'limited_diff': isinstance(patch, LimitedDiffContainer),
+            'diffset': self,
+        })
+
+        for hunk in hunks:
+            hunkbit = self.parse_hunk(hunk, source_file, target_file)
+            hunkbit.filediff = filediff
+            filediff.hunks.append(hunkbit)
+        return filediff
+
+    def parse_hunk(self, hunk, source_file, target_file):
+        result = AttributeDict(dict(
+            source_start=hunk['source_start'],
+            source_length=hunk['source_length'],
+            target_start=hunk['target_start'],
+            target_length=hunk['target_length'],
+            section_header=hunk['section_header'],
+            lines=[],
+        ))
+        before, after = [], []
+
+        for line in hunk['lines']:
+            if line['action'] == 'unmod':
+                result.lines.extend(
+                    self.parse_lines(before, after, source_file, target_file))
+                after.append(line)
+                before.append(line)
+            elif line['action'] == 'add':
+                after.append(line)
+            elif line['action'] == 'del':
+                before.append(line)
+            elif line['action'] == 'old-no-nl':
+                before.append(line)
+            elif line['action'] == 'new-no-nl':
+                after.append(line)
+
+        result.lines.extend(
+            self.parse_lines(before, after, source_file, target_file))
+        result.unified = self.as_unified(result.lines)
+        result.sideside = result.lines
+        return result
+
+    def parse_lines(self, before_lines, after_lines, source_file, target_file):
+        # TODO: dan: investigate doing the diff comparison and fast highlighting
+        # on the entire before and after buffered block lines rather than by
+        # line, this means we can get better 'fast' highlighting if the context
+        # allows it - eg.
+        # line 4: """
+        # line 5: this gets highlighted as a string
+        # line 6: """
+
+        lines = []
+        while before_lines or after_lines:
+            before, after = None, None
+            before_tokens, after_tokens = None, None
+
+            if before_lines:
+                before = before_lines.pop(0)
+            if after_lines:
+                after = after_lines.pop(0)
+
+            original = AttributeDict()
+            modified = AttributeDict()
+
+            if before:
+                if before['action'] == 'old-no-nl':
+                    before_tokens = [('nonl', before['line'])]
+                else:
+                    before_tokens = self.get_line_tokens(
+                        line_text=before['line'], line_number=before['old_lineno'],
+                        file=source_file)
+                original.lineno = before['old_lineno']
+                original.content = before['line']
+                original.action = self.action_to_op(before['action'])
+                original.comments = self.get_comments_for('old',
+                    source_file, before['old_lineno'])
+
+            if after:
+                if after['action'] == 'new-no-nl':
+                    after_tokens = [('nonl', after['line'])]
+                else:
+                    after_tokens = self.get_line_tokens(
+                        line_text=after['line'], line_number=after['new_lineno'],
+                        file=target_file)
+                modified.lineno = after['new_lineno']
+                modified.content = after['line']
+                modified.action = self.action_to_op(after['action'])
+                modified.comments = self.get_comments_for('new',
+                    target_file, after['new_lineno'])
+
+            # diff the lines
+            if before_tokens and after_tokens:
+                o_tokens, m_tokens, similarity = tokens_diff(
+                    before_tokens, after_tokens)
+                original.content = render_tokenstream(o_tokens)
+                modified.content = render_tokenstream(m_tokens)
+            elif before_tokens:
+                original.content = render_tokenstream(
+                    [(x[0], '', x[1]) for x in before_tokens])
+            elif after_tokens:
+                modified.content = render_tokenstream(
+                    [(x[0], '', x[1]) for x in after_tokens])
+
+            lines.append(AttributeDict({
+                'original': original,
+                'modified': modified,
+            }))
+
+        return lines
+
+    def get_comments_for(self, version, file, line_number):
+        if hasattr(file, 'unicode_path'):
+            file = file.unicode_path
+
+        if not isinstance(file, basestring):
+            return None
+
+        line_key = {
+            'old': 'o',
+            'new': 'n',
+        }[version] + str(line_number)
+
+        return self.comments.get(file, {}).get(line_key)
+
+    def get_line_tokens(self, line_text, line_number, file=None):
+        filenode = None
+        filename = None
+
+        if isinstance(file, basestring):
+            filename = file
+        elif isinstance(file, FileNode):
+            filenode = file
+            filename = file.unicode_path
+
+        if self.highlight_mode == self.HL_REAL and filenode:
+            if line_number and file.size < self.max_file_size_limit:
+                return self.get_tokenized_filenode_line(file, line_number)
+
+        if self.highlight_mode in (self.HL_REAL, self.HL_FAST) and filename:
+            lexer = self._get_lexer_for_filename(filename)
+            return list(tokenize_string(line_text, lexer))
+
+        return list(tokenize_string(line_text, plain_text_lexer))
+
+    def get_tokenized_filenode_line(self, filenode, line_number):
+
+        if filenode not in self.highlighted_filenodes:
+            tokenized_lines = filenode_as_lines_tokens(filenode, filenode.lexer)
+            self.highlighted_filenodes[filenode] = tokenized_lines
+        return self.highlighted_filenodes[filenode][line_number - 1]
+
+    def action_to_op(self, action):
+        return {
+            'add': '+',
+            'del': '-',
+            'unmod': ' ',
+            'old-no-nl': ' ',
+            'new-no-nl': ' ',
+        }.get(action, action)
+
+    def as_unified(self, lines):
+        """ Return a generator that yields the lines of a diff in unified order """
+        def generator():
+            buf = []
+            for line in lines:
+
+                if buf and not line.original or line.original.action == ' ':
+                    for b in buf:
+                        yield b
+                    buf = []
+
+                if line.original:
+                    if line.original.action == ' ':
+                        yield (line.original.lineno, line.modified.lineno,
+                               line.original.action, line.original.content,
+                               line.original.comments)
+                        continue
+
+                    if line.original.action == '-':
+                        yield (line.original.lineno, None,
+                               line.original.action, line.original.content,
+                               line.original.comments)
+
+                    if line.modified.action == '+':
+                        buf.append((
+                            None, line.modified.lineno,
+                            line.modified.action, line.modified.content,
+                            line.modified.comments))
+                        continue
+
+                if line.modified:
+                    yield (None, line.modified.lineno,
+                           line.modified.action, line.modified.content,
+                           line.modified.comments)
+
+            for b in buf:
+                yield b
+
+        return generator()

rhodecode/lib/dbmigrate/schema/db_4_4_0_2.py

@@ -0,0 +1,3640 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2010-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+"""
+Database Models for RhodeCode Enterprise
+"""
+
+import re
+import os
+import sys
+import time
+import hashlib
+import logging
+import datetime
+import warnings
+import ipaddress
+import functools
+import traceback
+import collections
+
+
+from sqlalchemy import *
+from sqlalchemy.exc import IntegrityError
+from sqlalchemy.ext.declarative import declared_attr
+from sqlalchemy.ext.hybrid import hybrid_property
+from sqlalchemy.orm import (
+    relationship, joinedload, class_mapper, validates, aliased)
+from sqlalchemy.sql.expression import true
+from beaker.cache import cache_region, region_invalidate
+from webob.exc import HTTPNotFound
+from zope.cachedescriptors.property import Lazy as LazyProperty
+
+from pylons import url
+from pylons.i18n.translation import lazy_ugettext as _
+
+from rhodecode.lib.vcs import get_backend, get_vcs_instance
+from rhodecode.lib.vcs.utils.helpers import get_scm
+from rhodecode.lib.vcs.exceptions import VCSError
+from rhodecode.lib.vcs.backends.base import (
+    EmptyCommit, Reference, MergeFailureReason)
+from rhodecode.lib.utils2 import (
+    str2bool, safe_str, get_commit_safe, safe_unicode, remove_prefix, md5_safe,
+    time_to_datetime, aslist, Optional, safe_int, get_clone_url, AttributeDict,
+    glob2re)
+from rhodecode.lib.jsonalchemy import MutationObj, JsonType, JSONDict
+from rhodecode.lib.ext_json import json
+from rhodecode.lib.caching_query import FromCache
+from rhodecode.lib.encrypt import AESCipher
+
+from rhodecode.model.meta import Base, Session
+
+URL_SEP = '/'
+log = logging.getLogger(__name__)
+
+# =============================================================================
+# BASE CLASSES
+# =============================================================================
+
+# this is propagated from .ini file rhodecode.encrypted_values.secret or
+# beaker.session.secret if first is not set.
+# and initialized at environment.py
+ENCRYPTION_KEY = None
+
+# used to sort permissions by types, '#' used here is not allowed to be in
+# usernames, and it's very early in sorted string.printable table.
+PERMISSION_TYPE_SORT = {
+    'admin': '####',
+    'write': '###',
+    'read':  '##',
+    'none':  '#',
+}
+
+
+def display_sort(obj):
+    """
+    Sort function used to sort permissions in .permissions() function of
+    Repository, RepoGroup, UserGroup. Also it put the default user in front
+    of all other resources
+    """
+
+    if obj.username == User.DEFAULT_USER:
+        return '#####'
+    prefix = PERMISSION_TYPE_SORT.get(obj.permission.split('.')[-1], '')
+    return prefix + obj.username
+
+
+def _hash_key(k):
+    return md5_safe(k)
+
+
+class EncryptedTextValue(TypeDecorator):
+    """
+    Special column for encrypted long text data, use like::
+
+        value = Column("encrypted_value", EncryptedValue(), nullable=False)
+
+    This column is intelligent so if value is in unencrypted form it return
+    unencrypted form, but on save it always encrypts
+    """
+    impl = Text
+
+    def process_bind_param(self, value, dialect):
+        if not value:
+            return value
+        if value.startswith('enc$aes$') or value.startswith('enc$aes_hmac$'):
+            # protect against double encrypting if someone manually starts
+            # doing
+            raise ValueError('value needs to be in unencrypted format, ie. '
+                             'not starting with enc$aes')
+        return 'enc$aes_hmac$%s' % AESCipher(
+            ENCRYPTION_KEY, hmac=True).encrypt(value)
+
+    def process_result_value(self, value, dialect):
+        import rhodecode
+
+        if not value:
+            return value
+
+        parts = value.split('$', 3)
+        if not len(parts) == 3:
+            # probably not encrypted values
+            return value
+        else:
+            if parts[0] != 'enc':
+                # parts ok but without our header ?
+                return value
+            enc_strict_mode = str2bool(rhodecode.CONFIG.get(
+                'rhodecode.encrypted_values.strict') or True)
+            # at that stage we know it's our encryption
+            if parts[1] == 'aes':
+                decrypted_data = AESCipher(ENCRYPTION_KEY).decrypt(parts[2])
+            elif parts[1] == 'aes_hmac':
+                decrypted_data = AESCipher(
+                    ENCRYPTION_KEY, hmac=True,
+                    strict_verification=enc_strict_mode).decrypt(parts[2])
+            else:
+                raise ValueError(
+                    'Encryption type part is wrong, must be `aes` '
+                    'or `aes_hmac`, got `%s` instead' % (parts[1]))
+            return decrypted_data
+
+
+class BaseModel(object):
+    """
+    Base Model for all classes
+    """
+
+    @classmethod
+    def _get_keys(cls):
+        """return column names for this model """
+        return class_mapper(cls).c.keys()
+
+    def get_dict(self):
+        """
+        return dict with keys and values corresponding
+        to this model data """
+
+        d = {}
+        for k in self._get_keys():
+            d[k] = getattr(self, k)
+
+        # also use __json__() if present to get additional fields
+        _json_attr = getattr(self, '__json__', None)
+        if _json_attr:
+            # update with attributes from __json__
+            if callable(_json_attr):
+                _json_attr = _json_attr()
+            for k, val in _json_attr.iteritems():
+                d[k] = val
+        return d
+
+    def get_appstruct(self):
+        """return list with keys and values tuples corresponding
+        to this model data """
+
+        l = []
+        for k in self._get_keys():
+            l.append((k, getattr(self, k),))
+        return l
+
+    def populate_obj(self, populate_dict):
+        """populate model with data from given populate_dict"""
+
+        for k in self._get_keys():
+            if k in populate_dict:
+                setattr(self, k, populate_dict[k])
+
+    @classmethod
+    def query(cls):
+        return Session().query(cls)
+
+    @classmethod
+    def get(cls, id_):
+        if id_:
+            return cls.query().get(id_)
+
+    @classmethod
+    def get_or_404(cls, id_):
+        try:
+            id_ = int(id_)
+        except (TypeError, ValueError):
+            raise HTTPNotFound
+
+        res = cls.query().get(id_)
+        if not res:
+            raise HTTPNotFound
+        return res
+
+    @classmethod
+    def getAll(cls):
+        # deprecated and left for backward compatibility
+        return cls.get_all()
+
+    @classmethod
+    def get_all(cls):
+        return cls.query().all()
+
+    @classmethod
+    def delete(cls, id_):
+        obj = cls.query().get(id_)
+        Session().delete(obj)
+
+    @classmethod
+    def identity_cache(cls, session, attr_name, value):
+        exist_in_session = []
+        for (item_cls, pkey), instance in session.identity_map.items():
+            if cls == item_cls and getattr(instance, attr_name) == value:
+                exist_in_session.append(instance)
+        if exist_in_session:
+            if len(exist_in_session) == 1:
+                return exist_in_session[0]
+            log.exception(
+                'multiple objects with attr %s and '
+                'value %s found with same name: %r',
+                attr_name, value, exist_in_session)
+
+    def __repr__(self):
+        if hasattr(self, '__unicode__'):
+            # python repr needs to return str
+            try:
+                return safe_str(self.__unicode__())
+            except UnicodeDecodeError:
+                pass
+        return '<DB:%s>' % (self.__class__.__name__)
+
+
+class RhodeCodeSetting(Base, BaseModel):
+    __tablename__ = 'rhodecode_settings'
+    __table_args__ = (
+        UniqueConstraint('app_settings_name'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    SETTINGS_TYPES = {
+        'str': safe_str,
+        'int': safe_int,
+        'unicode': safe_unicode,
+        'bool': str2bool,
+        'list': functools.partial(aslist, sep=',')
+    }
+    DEFAULT_UPDATE_URL = 'https://rhodecode.com/api/v1/info/versions'
+    GLOBAL_CONF_KEY = 'app_settings'
+
+    app_settings_id = Column("app_settings_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    app_settings_name = Column("app_settings_name", String(255), nullable=True, unique=None, default=None)
+    _app_settings_value = Column("app_settings_value", String(4096), nullable=True, unique=None, default=None)
+    _app_settings_type = Column("app_settings_type", String(255), nullable=True, unique=None, default=None)
+
+    def __init__(self, key='', val='', type='unicode'):
+        self.app_settings_name = key
+        self.app_settings_type = type
+        self.app_settings_value = val
+
+    @validates('_app_settings_value')
+    def validate_settings_value(self, key, val):
+        assert type(val) == unicode
+        return val
+
+    @hybrid_property
+    def app_settings_value(self):
+        v = self._app_settings_value
+        _type = self.app_settings_type
+        if _type:
+            _type = self.app_settings_type.split('.')[0]
+            # decode the encrypted value
+            if 'encrypted' in self.app_settings_type:
+                cipher = EncryptedTextValue()
+                v = safe_unicode(cipher.process_result_value(v, None))
+
+        converter = self.SETTINGS_TYPES.get(_type) or \
+            self.SETTINGS_TYPES['unicode']
+        return converter(v)
+
+    @app_settings_value.setter
+    def app_settings_value(self, val):
+        """
+        Setter that will always make sure we use unicode in app_settings_value
+
+        :param val:
+        """
+        val = safe_unicode(val)
+        # encode the encrypted value
+        if 'encrypted' in self.app_settings_type:
+            cipher = EncryptedTextValue()
+            val = safe_unicode(cipher.process_bind_param(val, None))
+        self._app_settings_value = val
+
+    @hybrid_property
+    def app_settings_type(self):
+        return self._app_settings_type
+
+    @app_settings_type.setter
+    def app_settings_type(self, val):
+        if val.split('.')[0] not in self.SETTINGS_TYPES:
+            raise Exception('type must be one of %s got %s'
+                            % (self.SETTINGS_TYPES.keys(), val))
+        self._app_settings_type = val
+
+    def __unicode__(self):
+        return u"<%s('%s:%s[%s]')>" % (
+            self.__class__.__name__,
+            self.app_settings_name, self.app_settings_value,
+            self.app_settings_type
+        )
+
+
+class RhodeCodeUi(Base, BaseModel):
+    __tablename__ = 'rhodecode_ui'
+    __table_args__ = (
+        UniqueConstraint('ui_key'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    HOOK_REPO_SIZE = 'changegroup.repo_size'
+    # HG
+    HOOK_PRE_PULL = 'preoutgoing.pre_pull'
+    HOOK_PULL = 'outgoing.pull_logger'
+    HOOK_PRE_PUSH = 'prechangegroup.pre_push'
+    HOOK_PUSH = 'changegroup.push_logger'
+
+    # TODO: johbo: Unify way how hooks are configured for git and hg,
+    # git part is currently hardcoded.
+
+    # SVN PATTERNS
+    SVN_BRANCH_ID = 'vcs_svn_branch'
+    SVN_TAG_ID = 'vcs_svn_tag'
+
+    ui_id = Column(
+        "ui_id", Integer(), nullable=False, unique=True, default=None,
+        primary_key=True)
+    ui_section = Column(
+        "ui_section", String(255), nullable=True, unique=None, default=None)
+    ui_key = Column(
+        "ui_key", String(255), nullable=True, unique=None, default=None)
+    ui_value = Column(
+        "ui_value", String(255), nullable=True, unique=None, default=None)
+    ui_active = Column(
+        "ui_active", Boolean(), nullable=True, unique=None, default=True)
+
+    def __repr__(self):
+        return '<%s[%s]%s=>%s]>' % (self.__class__.__name__, self.ui_section,
+                                    self.ui_key, self.ui_value)
+
+
+class RepoRhodeCodeSetting(Base, BaseModel):
+    __tablename__ = 'repo_rhodecode_settings'
+    __table_args__ = (
+        UniqueConstraint(
+            'app_settings_name', 'repository_id',
+            name='uq_repo_rhodecode_setting_name_repo_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    repository_id = Column(
+        "repository_id", Integer(), ForeignKey('repositories.repo_id'),
+        nullable=False)
+    app_settings_id = Column(
+        "app_settings_id", Integer(), nullable=False, unique=True,
+        default=None, primary_key=True)
+    app_settings_name = Column(
+        "app_settings_name", String(255), nullable=True, unique=None,
+        default=None)
+    _app_settings_value = Column(
+        "app_settings_value", String(4096), nullable=True, unique=None,
+        default=None)
+    _app_settings_type = Column(
+        "app_settings_type", String(255), nullable=True, unique=None,
+        default=None)
+
+    repository = relationship('Repository')
+
+    def __init__(self, repository_id, key='', val='', type='unicode'):
+        self.repository_id = repository_id
+        self.app_settings_name = key
+        self.app_settings_type = type
+        self.app_settings_value = val
+
+    @validates('_app_settings_value')
+    def validate_settings_value(self, key, val):
+        assert type(val) == unicode
+        return val
+
+    @hybrid_property
+    def app_settings_value(self):
+        v = self._app_settings_value
+        type_ = self.app_settings_type
+        SETTINGS_TYPES = RhodeCodeSetting.SETTINGS_TYPES
+        converter = SETTINGS_TYPES.get(type_) or SETTINGS_TYPES['unicode']
+        return converter(v)
+
+    @app_settings_value.setter
+    def app_settings_value(self, val):
+        """
+        Setter that will always make sure we use unicode in app_settings_value
+
+        :param val:
+        """
+        self._app_settings_value = safe_unicode(val)
+
+    @hybrid_property
+    def app_settings_type(self):
+        return self._app_settings_type
+
+    @app_settings_type.setter
+    def app_settings_type(self, val):
+        SETTINGS_TYPES = RhodeCodeSetting.SETTINGS_TYPES
+        if val not in SETTINGS_TYPES:
+            raise Exception('type must be one of %s got %s'
+                            % (SETTINGS_TYPES.keys(), val))
+        self._app_settings_type = val
+
+    def __unicode__(self):
+        return u"<%s('%s:%s:%s[%s]')>" % (
+            self.__class__.__name__, self.repository.repo_name,
+            self.app_settings_name, self.app_settings_value,
+            self.app_settings_type
+        )
+
+
+class RepoRhodeCodeUi(Base, BaseModel):
+    __tablename__ = 'repo_rhodecode_ui'
+    __table_args__ = (
+        UniqueConstraint(
+            'repository_id', 'ui_section', 'ui_key',
+            name='uq_repo_rhodecode_ui_repository_id_section_key'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    repository_id = Column(
+        "repository_id", Integer(), ForeignKey('repositories.repo_id'),
+        nullable=False)
+    ui_id = Column(
+        "ui_id", Integer(), nullable=False, unique=True, default=None,
+        primary_key=True)
+    ui_section = Column(
+        "ui_section", String(255), nullable=True, unique=None, default=None)
+    ui_key = Column(
+        "ui_key", String(255), nullable=True, unique=None, default=None)
+    ui_value = Column(
+        "ui_value", String(255), nullable=True, unique=None, default=None)
+    ui_active = Column(
+        "ui_active", Boolean(), nullable=True, unique=None, default=True)
+
+    repository = relationship('Repository')
+
+    def __repr__(self):
+        return '<%s[%s:%s]%s=>%s]>' % (
+            self.__class__.__name__, self.repository.repo_name,
+            self.ui_section, self.ui_key, self.ui_value)
+
+
+class User(Base, BaseModel):
+    __tablename__ = 'users'
+    __table_args__ = (
+        UniqueConstraint('username'), UniqueConstraint('email'),
+        Index('u_username_idx', 'username'),
+        Index('u_email_idx', 'email'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    DEFAULT_USER = 'default'
+    DEFAULT_USER_EMAIL = '[e-mail]'
+    DEFAULT_GRAVATAR_URL = 'https://secure.gravatar.com/avatar/{md5email}?d=identicon&s={size}'
+
+    user_id = Column("user_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    username = Column("username", String(255), nullable=True, unique=None, default=None)
+    password = Column("password", String(255), nullable=True, unique=None, default=None)
+    active = Column("active", Boolean(), nullable=True, unique=None, default=True)
+    admin = Column("admin", Boolean(), nullable=True, unique=None, default=False)
+    name = Column("firstname", String(255), nullable=True, unique=None, default=None)
+    lastname = Column("lastname", String(255), nullable=True, unique=None, default=None)
+    _email = Column("email", String(255), nullable=True, unique=None, default=None)
+    last_login = Column("last_login", DateTime(timezone=False), nullable=True, unique=None, default=None)
+    extern_type = Column("extern_type", String(255), nullable=True, unique=None, default=None)
+    extern_name = Column("extern_name", String(255), nullable=True, unique=None, default=None)
+    api_key = Column("api_key", String(255), nullable=True, unique=None, default=None)
+    inherit_default_permissions = Column("inherit_default_permissions", Boolean(), nullable=False, unique=None, default=True)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+    _user_data = Column("user_data", LargeBinary(), nullable=True)  # JSON data
+
+    user_log = relationship('UserLog')
+    user_perms = relationship('UserToPerm', primaryjoin="User.user_id==UserToPerm.user_id", cascade='all')
+
+    repositories = relationship('Repository')
+    repository_groups = relationship('RepoGroup')
+    user_groups = relationship('UserGroup')
+
+    user_followers = relationship('UserFollowing', primaryjoin='UserFollowing.follows_user_id==User.user_id', cascade='all')
+    followings = relationship('UserFollowing', primaryjoin='UserFollowing.user_id==User.user_id', cascade='all')
+
+    repo_to_perm = relationship('UserRepoToPerm', primaryjoin='UserRepoToPerm.user_id==User.user_id', cascade='all')
+    repo_group_to_perm = relationship('UserRepoGroupToPerm', primaryjoin='UserRepoGroupToPerm.user_id==User.user_id', cascade='all')
+    user_group_to_perm = relationship('UserUserGroupToPerm', primaryjoin='UserUserGroupToPerm.user_id==User.user_id', cascade='all')
+
+    group_member = relationship('UserGroupMember', cascade='all')
+
+    notifications = relationship('UserNotification', cascade='all')
+    # notifications assigned to this user
+    user_created_notifications = relationship('Notification', cascade='all')
+    # comments created by this user
+    user_comments = relationship('ChangesetComment', cascade='all')
+    # user profile extra info
+    user_emails = relationship('UserEmailMap', cascade='all')
+    user_ip_map = relationship('UserIpMap', cascade='all')
+    user_auth_tokens = relationship('UserApiKeys', cascade='all')
+    # gists
+    user_gists = relationship('Gist', cascade='all')
+    # user pull requests
+    user_pull_requests = relationship('PullRequest', cascade='all')
+    # external identities
+    extenal_identities = relationship(
+        'ExternalIdentity',
+        primaryjoin="User.user_id==ExternalIdentity.local_user_id",
+        cascade='all')
+
+    def __unicode__(self):
+        return u"<%s('id:%s:%s')>" % (self.__class__.__name__,
+                                      self.user_id, self.username)
+
+    @hybrid_property
+    def email(self):
+        return self._email
+
+    @email.setter
+    def email(self, val):
+        self._email = val.lower() if val else None
+
+    @property
+    def firstname(self):
+        # alias for future
+        return self.name
+
+    @property
+    def emails(self):
+        other = UserEmailMap.query().filter(UserEmailMap.user==self).all()
+        return [self.email] + [x.email for x in other]
+
+    @property
+    def auth_tokens(self):
+        return [self.api_key] + [x.api_key for x in self.extra_auth_tokens]
+
+    @property
+    def extra_auth_tokens(self):
+        return UserApiKeys.query().filter(UserApiKeys.user == self).all()
+
+    @property
+    def feed_token(self):
+        feed_tokens = UserApiKeys.query()\
+            .filter(UserApiKeys.user == self)\
+            .filter(UserApiKeys.role == UserApiKeys.ROLE_FEED)\
+            .all()
+        if feed_tokens:
+            return feed_tokens[0].api_key
+        else:
+            # use the main token so we don't end up with nothing...
+            return self.api_key
+
+    @classmethod
+    def extra_valid_auth_tokens(cls, user, role=None):
+        tokens = UserApiKeys.query().filter(UserApiKeys.user == user)\
+                .filter(or_(UserApiKeys.expires == -1,
+                            UserApiKeys.expires >= time.time()))
+        if role:
+            tokens = tokens.filter(or_(UserApiKeys.role == role,
+                                       UserApiKeys.role == UserApiKeys.ROLE_ALL))
+        return tokens.all()
+
+    @property
+    def ip_addresses(self):
+        ret = UserIpMap.query().filter(UserIpMap.user == self).all()
+        return [x.ip_addr for x in ret]
+
+    @property
+    def username_and_name(self):
+        return '%s (%s %s)' % (self.username, self.firstname, self.lastname)
+
+    @property
+    def username_or_name_or_email(self):
+        full_name = self.full_name if self.full_name is not ' ' else None
+        return self.username or full_name or self.email
+
+    @property
+    def full_name(self):
+        return '%s %s' % (self.firstname, self.lastname)
+
+    @property
+    def full_name_or_username(self):
+        return ('%s %s' % (self.firstname, self.lastname)
+                if (self.firstname and self.lastname) else self.username)
+
+    @property
+    def full_contact(self):
+        return '%s %s <%s>' % (self.firstname, self.lastname, self.email)
+
+    @property
+    def short_contact(self):
+        return '%s %s' % (self.firstname, self.lastname)
+
+    @property
+    def is_admin(self):
+        return self.admin
+
+    @property
+    def AuthUser(self):
+        """
+        Returns instance of AuthUser for this user
+        """
+        from rhodecode.lib.auth import AuthUser
+        return AuthUser(user_id=self.user_id, api_key=self.api_key,
+                        username=self.username)
+
+    @hybrid_property
+    def user_data(self):
+        if not self._user_data:
+            return {}
+
+        try:
+            return json.loads(self._user_data)
+        except TypeError:
+            return {}
+
+    @user_data.setter
+    def user_data(self, val):
+        if not isinstance(val, dict):
+            raise Exception('user_data must be dict, got %s' % type(val))
+        try:
+            self._user_data = json.dumps(val)
+        except Exception:
+            log.error(traceback.format_exc())
+
+    @classmethod
+    def get_by_username(cls, username, case_insensitive=False,
+                        cache=False, identity_cache=False):
+        session = Session()
+
+        if case_insensitive:
+            q = cls.query().filter(
+                func.lower(cls.username) == func.lower(username))
+        else:
+            q = cls.query().filter(cls.username == username)
+
+        if cache:
+            if identity_cache:
+                val = cls.identity_cache(session, 'username', username)
+                if val:
+                    return val
+            else:
+                q = q.options(
+                    FromCache("sql_cache_short",
+                              "get_user_by_name_%s" % _hash_key(username)))
+
+        return q.scalar()
+
+    @classmethod
+    def get_by_auth_token(cls, auth_token, cache=False, fallback=True):
+        q = cls.query().filter(cls.api_key == auth_token)
+
+        if cache:
+            q = q.options(FromCache("sql_cache_short",
+                                    "get_auth_token_%s" % auth_token))
+        res = q.scalar()
+
+        if fallback and not res:
+            #fallback to additional keys
+            _res = UserApiKeys.query()\
+                .filter(UserApiKeys.api_key == auth_token)\
+                .filter(or_(UserApiKeys.expires == -1,
+                            UserApiKeys.expires >= time.time()))\
+                .first()
+            if _res:
+                res = _res.user
+        return res
+
+    @classmethod
+    def get_by_email(cls, email, case_insensitive=False, cache=False):
+
+        if case_insensitive:
+            q = cls.query().filter(func.lower(cls.email) == func.lower(email))
+
+        else:
+            q = cls.query().filter(cls.email == email)
+
+        if cache:
+            q = q.options(FromCache("sql_cache_short",
+                                    "get_email_key_%s" % _hash_key(email)))
+
+        ret = q.scalar()
+        if ret is None:
+            q = UserEmailMap.query()
+            # try fetching in alternate email map
+            if case_insensitive:
+                q = q.filter(func.lower(UserEmailMap.email) == func.lower(email))
+            else:
+                q = q.filter(UserEmailMap.email == email)
+            q = q.options(joinedload(UserEmailMap.user))
+            if cache:
+                q = q.options(FromCache("sql_cache_short",
+                                        "get_email_map_key_%s" % email))
+            ret = getattr(q.scalar(), 'user', None)
+
+        return ret
+
+    @classmethod
+    def get_from_cs_author(cls, author):
+        """
+        Tries to get User objects out of commit author string
+
+        :param author:
+        """
+        from rhodecode.lib.helpers import email, author_name
+        # Valid email in the attribute passed, see if they're in the system
+        _email = email(author)
+        if _email:
+            user = cls.get_by_email(_email, case_insensitive=True)
+            if user:
+                return user
+        # Maybe we can match by username?
+        _author = author_name(author)
+        user = cls.get_by_username(_author, case_insensitive=True)
+        if user:
+            return user
+
+    def update_userdata(self, **kwargs):
+        usr = self
+        old = usr.user_data
+        old.update(**kwargs)
+        usr.user_data = old
+        Session().add(usr)
+        log.debug('updated userdata with ', kwargs)
+
+    def update_lastlogin(self):
+        """Update user lastlogin"""
+        self.last_login = datetime.datetime.now()
+        Session().add(self)
+        log.debug('updated user %s lastlogin', self.username)
+
+    def update_lastactivity(self):
+        """Update user lastactivity"""
+        usr = self
+        old = usr.user_data
+        old.update({'last_activity': time.time()})
+        usr.user_data = old
+        Session().add(usr)
+        log.debug('updated user %s lastactivity', usr.username)
+
+    def update_password(self, new_password, change_api_key=False):
+        from rhodecode.lib.auth import get_crypt_password,generate_auth_token
+
+        self.password = get_crypt_password(new_password)
+        if change_api_key:
+            self.api_key = generate_auth_token(self.username)
+        Session().add(self)
+
+    @classmethod
+    def get_first_super_admin(cls):
+        user = User.query().filter(User.admin == true()).first()
+        if user is None:
+            raise Exception('FATAL: Missing administrative account!')
+        return user
+
+    @classmethod
+    def get_all_super_admins(cls):
+        """
+        Returns all admin accounts sorted by username
+        """
+        return User.query().filter(User.admin == true())\
+            .order_by(User.username.asc()).all()
+
+    @classmethod
+    def get_default_user(cls, cache=False):
+        user = User.get_by_username(User.DEFAULT_USER, cache=cache)
+        if user is None:
+            raise Exception('FATAL: Missing default account!')
+        return user
+
+    def _get_default_perms(self, user, suffix=''):
+        from rhodecode.model.permission import PermissionModel
+        return PermissionModel().get_default_perms(user.user_perms, suffix)
+
+    def get_default_perms(self, suffix=''):
+        return self._get_default_perms(self, suffix)
+
+    def get_api_data(self, include_secrets=False, details='full'):
+        """
+        Common function for generating user related data for API
+
+        :param include_secrets: By default secrets in the API data will be replaced
+           by a placeholder value to prevent exposing this data by accident. In case
+           this data shall be exposed, set this flag to ``True``.
+
+        :param details: details can be 'basic|full' basic gives only a subset of
+           the available user information that includes user_id, name and emails.
+        """
+        user = self
+        user_data = self.user_data
+        data = {
+            'user_id': user.user_id,
+            'username': user.username,
+            'firstname': user.name,
+            'lastname': user.lastname,
+            'email': user.email,
+            'emails': user.emails,
+        }
+        if details == 'basic':
+            return data
+
+        api_key_length = 40
+        api_key_replacement = '*' * api_key_length
+
+        extras = {
+            'api_key': api_key_replacement,
+            'api_keys': [api_key_replacement],
+            'active': user.active,
+            'admin': user.admin,
+            'extern_type': user.extern_type,
+            'extern_name': user.extern_name,
+            'last_login': user.last_login,
+            'ip_addresses': user.ip_addresses,
+            'language': user_data.get('language')
+        }
+        data.update(extras)
+
+        if include_secrets:
+            data['api_key'] = user.api_key
+            data['api_keys'] = user.auth_tokens
+        return data
+
+    def __json__(self):
+        data = {
+            'full_name': self.full_name,
+            'full_name_or_username': self.full_name_or_username,
+            'short_contact': self.short_contact,
+            'full_contact': self.full_contact,
+        }
+        data.update(self.get_api_data())
+        return data
+
+
+class UserApiKeys(Base, BaseModel):
+    __tablename__ = 'user_api_keys'
+    __table_args__ = (
+        Index('uak_api_key_idx', 'api_key'),
+        Index('uak_api_key_expires_idx', 'api_key', 'expires'),
+        UniqueConstraint('api_key'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    __mapper_args__ = {}
+
+    # ApiKey role
+    ROLE_ALL = 'token_role_all'
+    ROLE_HTTP = 'token_role_http'
+    ROLE_VCS = 'token_role_vcs'
+    ROLE_API = 'token_role_api'
+    ROLE_FEED = 'token_role_feed'
+    ROLES = [ROLE_ALL, ROLE_HTTP, ROLE_VCS, ROLE_API, ROLE_FEED]
+
+    user_api_key_id = Column("user_api_key_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=True, unique=None, default=None)
+    api_key = Column("api_key", String(255), nullable=False, unique=True)
+    description = Column('description', UnicodeText().with_variant(UnicodeText(1024), 'mysql'))
+    expires = Column('expires', Float(53), nullable=False)
+    role = Column('role', String(255), nullable=True)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+
+    user = relationship('User', lazy='joined')
+
+    @classmethod
+    def _get_role_name(cls, role):
+        return {
+            cls.ROLE_ALL: _('all'),
+            cls.ROLE_HTTP: _('http/web interface'),
+            cls.ROLE_VCS: _('vcs (git/hg/svn protocol)'),
+            cls.ROLE_API: _('api calls'),
+            cls.ROLE_FEED: _('feed access'),
+        }.get(role, role)
+
+    @property
+    def expired(self):
+        if self.expires == -1:
+            return False
+        return time.time() > self.expires
+
+    @property
+    def role_humanized(self):
+        return self._get_role_name(self.role)
+
+
+class UserEmailMap(Base, BaseModel):
+    __tablename__ = 'user_email_map'
+    __table_args__ = (
+        Index('uem_email_idx', 'email'),
+        UniqueConstraint('email'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    __mapper_args__ = {}
+
+    email_id = Column("email_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=True, unique=None, default=None)
+    _email = Column("email", String(255), nullable=True, unique=False, default=None)
+    user = relationship('User', lazy='joined')
+
+    @validates('_email')
+    def validate_email(self, key, email):
+        # check if this email is not main one
+        main_email = Session().query(User).filter(User.email == email).scalar()
+        if main_email is not None:
+            raise AttributeError('email %s is present is user table' % email)
+        return email
+
+    @hybrid_property
+    def email(self):
+        return self._email
+
+    @email.setter
+    def email(self, val):
+        self._email = val.lower() if val else None
+
+
+class UserIpMap(Base, BaseModel):
+    __tablename__ = 'user_ip_map'
+    __table_args__ = (
+        UniqueConstraint('user_id', 'ip_addr'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    __mapper_args__ = {}
+
+    ip_id = Column("ip_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=True, unique=None, default=None)
+    ip_addr = Column("ip_addr", String(255), nullable=True, unique=False, default=None)
+    active = Column("active", Boolean(), nullable=True, unique=None, default=True)
+    description = Column("description", String(10000), nullable=True, unique=None, default=None)
+    user = relationship('User', lazy='joined')
+
+    @classmethod
+    def _get_ip_range(cls, ip_addr):
+        net = ipaddress.ip_network(ip_addr, strict=False)
+        return [str(net.network_address), str(net.broadcast_address)]
+
+    def __json__(self):
+        return {
+          'ip_addr': self.ip_addr,
+          'ip_range': self._get_ip_range(self.ip_addr),
+        }
+
+    def __unicode__(self):
+        return u"<%s('user_id:%s=>%s')>" % (self.__class__.__name__,
+                                            self.user_id, self.ip_addr)
+
+class UserLog(Base, BaseModel):
+    __tablename__ = 'user_logs'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+    user_log_id = Column("user_log_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=True, unique=None, default=None)
+    username = Column("username", String(255), nullable=True, unique=None, default=None)
+    repository_id = Column("repository_id", Integer(), ForeignKey('repositories.repo_id'), nullable=True)
+    repository_name = Column("repository_name", String(255), nullable=True, unique=None, default=None)
+    user_ip = Column("user_ip", String(255), nullable=True, unique=None, default=None)
+    action = Column("action", Text().with_variant(Text(1200000), 'mysql'), nullable=True, unique=None, default=None)
+    action_date = Column("action_date", DateTime(timezone=False), nullable=True, unique=None, default=None)
+
+    def __unicode__(self):
+        return u"<%s('id:%s:%s')>" % (self.__class__.__name__,
+                                      self.repository_name,
+                                      self.action)
+
+    @property
+    def action_as_day(self):
+        return datetime.date(*self.action_date.timetuple()[:3])
+
+    user = relationship('User')
+    repository = relationship('Repository', cascade='')
+
+
+class UserGroup(Base, BaseModel):
+    __tablename__ = 'users_groups'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+
+    users_group_id = Column("users_group_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    users_group_name = Column("users_group_name", String(255), nullable=False, unique=True, default=None)
+    user_group_description = Column("user_group_description", String(10000), nullable=True, unique=None, default=None)
+    users_group_active = Column("users_group_active", Boolean(), nullable=True, unique=None, default=None)
+    inherit_default_permissions = Column("users_group_inherit_default_permissions", Boolean(), nullable=False, unique=None, default=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=False, default=None)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+    _group_data = Column("group_data", LargeBinary(), nullable=True)  # JSON data
+
+    members = relationship('UserGroupMember', cascade="all, delete, delete-orphan", lazy="joined")
+    users_group_to_perm = relationship('UserGroupToPerm', cascade='all')
+    users_group_repo_to_perm = relationship('UserGroupRepoToPerm', cascade='all')
+    users_group_repo_group_to_perm = relationship('UserGroupRepoGroupToPerm', cascade='all')
+    user_user_group_to_perm = relationship('UserUserGroupToPerm', cascade='all')
+    user_group_user_group_to_perm = relationship('UserGroupUserGroupToPerm ', primaryjoin="UserGroupUserGroupToPerm.target_user_group_id==UserGroup.users_group_id", cascade='all')
+
+    user = relationship('User')
+
+    @hybrid_property
+    def group_data(self):
+        if not self._group_data:
+            return {}
+
+        try:
+            return json.loads(self._group_data)
+        except TypeError:
+            return {}
+
+    @group_data.setter
+    def group_data(self, val):
+        try:
+            self._group_data = json.dumps(val)
+        except Exception:
+            log.error(traceback.format_exc())
+
+    def __unicode__(self):
+        return u"<%s('id:%s:%s')>" % (self.__class__.__name__,
+                                      self.users_group_id,
+                                      self.users_group_name)
+
+    @classmethod
+    def get_by_group_name(cls, group_name, cache=False,
+                          case_insensitive=False):
+        if case_insensitive:
+            q = cls.query().filter(func.lower(cls.users_group_name) ==
+                                   func.lower(group_name))
+
+        else:
+            q = cls.query().filter(cls.users_group_name == group_name)
+        if cache:
+            q = q.options(FromCache(
+                            "sql_cache_short",
+                            "get_group_%s" % _hash_key(group_name)))
+        return q.scalar()
+
+    @classmethod
+    def get(cls, user_group_id, cache=False):
+        user_group = cls.query()
+        if cache:
+            user_group = user_group.options(FromCache("sql_cache_short",
+                                    "get_users_group_%s" % user_group_id))
+        return user_group.get(user_group_id)
+
+    def permissions(self, with_admins=True, with_owner=True):
+        q = UserUserGroupToPerm.query().filter(UserUserGroupToPerm.user_group == self)
+        q = q.options(joinedload(UserUserGroupToPerm.user_group),
+                      joinedload(UserUserGroupToPerm.user),
+                      joinedload(UserUserGroupToPerm.permission),)
+
+        # get owners and admins and permissions. We do a trick of re-writing
+        # objects from sqlalchemy to named-tuples due to sqlalchemy session
+        # has a global reference and changing one object propagates to all
+        # others. This means if admin is also an owner admin_row that change
+        # would propagate to both objects
+        perm_rows = []
+        for _usr in q.all():
+            usr = AttributeDict(_usr.user.get_dict())
+            usr.permission = _usr.permission.permission_name
+            perm_rows.append(usr)
+
+        # filter the perm rows by 'default' first and then sort them by
+        # admin,write,read,none permissions sorted again alphabetically in
+        # each group
+        perm_rows = sorted(perm_rows, key=display_sort)
+
+        _admin_perm = 'usergroup.admin'
+        owner_row = []
+        if with_owner:
+            usr = AttributeDict(self.user.get_dict())
+            usr.owner_row = True
+            usr.permission = _admin_perm
+            owner_row.append(usr)
+
+        super_admin_rows = []
+        if with_admins:
+            for usr in User.get_all_super_admins():
+                # if this admin is also owner, don't double the record
+                if usr.user_id == owner_row[0].user_id:
+                    owner_row[0].admin_row = True
+                else:
+                    usr = AttributeDict(usr.get_dict())
+                    usr.admin_row = True
+                    usr.permission = _admin_perm
+                    super_admin_rows.append(usr)
+
+        return super_admin_rows + owner_row + perm_rows
+
+    def permission_user_groups(self):
+        q = UserGroupUserGroupToPerm.query().filter(UserGroupUserGroupToPerm.target_user_group == self)
+        q = q.options(joinedload(UserGroupUserGroupToPerm.user_group),
+                      joinedload(UserGroupUserGroupToPerm.target_user_group),
+                      joinedload(UserGroupUserGroupToPerm.permission),)
+
+        perm_rows = []
+        for _user_group in q.all():
+            usr = AttributeDict(_user_group.user_group.get_dict())
+            usr.permission = _user_group.permission.permission_name
+            perm_rows.append(usr)
+
+        return perm_rows
+
+    def _get_default_perms(self, user_group, suffix=''):
+        from rhodecode.model.permission import PermissionModel
+        return PermissionModel().get_default_perms(user_group.users_group_to_perm, suffix)
+
+    def get_default_perms(self, suffix=''):
+        return self._get_default_perms(self, suffix)
+
+    def get_api_data(self, with_group_members=True, include_secrets=False):
+        """
+        :param include_secrets: See :meth:`User.get_api_data`, this parameter is
+           basically forwarded.
+
+        """
+        user_group = self
+
+        data = {
+            'users_group_id': user_group.users_group_id,
+            'group_name': user_group.users_group_name,
+            'group_description': user_group.user_group_description,
+            'active': user_group.users_group_active,
+            'owner': user_group.user.username,
+        }
+        if with_group_members:
+            users = []
+            for user in user_group.members:
+                user = user.user
+                users.append(user.get_api_data(include_secrets=include_secrets))
+            data['users'] = users
+
+        return data
+
+
+class UserGroupMember(Base, BaseModel):
+    __tablename__ = 'users_groups_members'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+
+    users_group_member_id = Column("users_group_member_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    users_group_id = Column("users_group_id", Integer(), ForeignKey('users_groups.users_group_id'), nullable=False, unique=None, default=None)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=None, default=None)
+
+    user = relationship('User', lazy='joined')
+    users_group = relationship('UserGroup')
+
+    def __init__(self, gr_id='', u_id=''):
+        self.users_group_id = gr_id
+        self.user_id = u_id
+
+
+class RepositoryField(Base, BaseModel):
+    __tablename__ = 'repositories_fields'
+    __table_args__ = (
+        UniqueConstraint('repository_id', 'field_key'),  # no-multi field
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+    PREFIX = 'ex_'  # prefix used in form to not conflict with already existing fields
+
+    repo_field_id = Column("repo_field_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    repository_id = Column("repository_id", Integer(), ForeignKey('repositories.repo_id'), nullable=False, unique=None, default=None)
+    field_key = Column("field_key", String(250))
+    field_label = Column("field_label", String(1024), nullable=False)
+    field_value = Column("field_value", String(10000), nullable=False)
+    field_desc = Column("field_desc", String(1024), nullable=False)
+    field_type = Column("field_type", String(255), nullable=False, unique=None)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+
+    repository = relationship('Repository')
+
+    @property
+    def field_key_prefixed(self):
+        return 'ex_%s' % self.field_key
+
+    @classmethod
+    def un_prefix_key(cls, key):
+        if key.startswith(cls.PREFIX):
+            return key[len(cls.PREFIX):]
+        return key
+
+    @classmethod
+    def get_by_key_name(cls, key, repo):
+        row = cls.query()\
+                .filter(cls.repository == repo)\
+                .filter(cls.field_key == key).scalar()
+        return row
+
+
+class Repository(Base, BaseModel):
+    __tablename__ = 'repositories'
+    __table_args__ = (
+        Index('r_repo_name_idx', 'repo_name', mysql_length=255),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+    DEFAULT_CLONE_URI = '{scheme}://{user}@{netloc}/{repo}'
+    DEFAULT_CLONE_URI_ID = '{scheme}://{user}@{netloc}/_{repoid}'
+
+    STATE_CREATED = 'repo_state_created'
+    STATE_PENDING = 'repo_state_pending'
+    STATE_ERROR = 'repo_state_error'
+
+    LOCK_AUTOMATIC = 'lock_auto'
+    LOCK_API = 'lock_api'
+    LOCK_WEB = 'lock_web'
+    LOCK_PULL = 'lock_pull'
+
+    NAME_SEP = URL_SEP
+
+    repo_id = Column(
+        "repo_id", Integer(), nullable=False, unique=True, default=None,
+        primary_key=True)
+    _repo_name = Column(
+        "repo_name", Text(), nullable=False, default=None)
+    _repo_name_hash = Column(
+        "repo_name_hash", String(255), nullable=False, unique=True)
+    repo_state = Column("repo_state", String(255), nullable=True)
+
+    clone_uri = Column(
+        "clone_uri", EncryptedTextValue(), nullable=True, unique=False,
+        default=None)
+    repo_type = Column(
+        "repo_type", String(255), nullable=False, unique=False, default=None)
+    user_id = Column(
+        "user_id", Integer(), ForeignKey('users.user_id'), nullable=False,
+        unique=False, default=None)
+    private = Column(
+        "private", Boolean(), nullable=True, unique=None, default=None)
+    enable_statistics = Column(
+        "statistics", Boolean(), nullable=True, unique=None, default=True)
+    enable_downloads = Column(
+        "downloads", Boolean(), nullable=True, unique=None, default=True)
+    description = Column(
+        "description", String(10000), nullable=True, unique=None, default=None)
+    created_on = Column(
+        'created_on', DateTime(timezone=False), nullable=True, unique=None,
+        default=datetime.datetime.now)
+    updated_on = Column(
+        'updated_on', DateTime(timezone=False), nullable=True, unique=None,
+        default=datetime.datetime.now)
+    _landing_revision = Column(
+        "landing_revision", String(255), nullable=False, unique=False,
+        default=None)
+    enable_locking = Column(
+        "enable_locking", Boolean(), nullable=False, unique=None,
+        default=False)
+    _locked = Column(
+        "locked", String(255), nullable=True, unique=False, default=None)
+    _changeset_cache = Column(
+        "changeset_cache", LargeBinary(), nullable=True)  # JSON data
+
+    fork_id = Column(
+        "fork_id", Integer(), ForeignKey('repositories.repo_id'),
+        nullable=True, unique=False, default=None)
+    group_id = Column(
+        "group_id", Integer(), ForeignKey('groups.group_id'), nullable=True,
+        unique=False, default=None)
+
+    user = relationship('User', lazy='joined')
+    fork = relationship('Repository', remote_side=repo_id, lazy='joined')
+    group = relationship('RepoGroup', lazy='joined')
+    repo_to_perm = relationship(
+        'UserRepoToPerm', cascade='all',
+        order_by='UserRepoToPerm.repo_to_perm_id')
+    users_group_to_perm = relationship('UserGroupRepoToPerm', cascade='all')
+    stats = relationship('Statistics', cascade='all', uselist=False)
+
+    followers = relationship(
+        'UserFollowing',
+        primaryjoin='UserFollowing.follows_repo_id==Repository.repo_id',
+        cascade='all')
+    extra_fields = relationship(
+        'RepositoryField', cascade="all, delete, delete-orphan")
+    logs = relationship('UserLog')
+    comments = relationship(
+        'ChangesetComment', cascade="all, delete, delete-orphan")
+    pull_requests_source = relationship(
+        'PullRequest',
+        primaryjoin='PullRequest.source_repo_id==Repository.repo_id',
+        cascade="all, delete, delete-orphan")
+    pull_requests_target = relationship(
+        'PullRequest',
+        primaryjoin='PullRequest.target_repo_id==Repository.repo_id',
+        cascade="all, delete, delete-orphan")
+    ui = relationship('RepoRhodeCodeUi', cascade="all")
+    settings = relationship('RepoRhodeCodeSetting', cascade="all")
+    integrations = relationship('Integration',
+                                cascade="all, delete, delete-orphan")
+
+    def __unicode__(self):
+        return u"<%s('%s:%s')>" % (self.__class__.__name__, self.repo_id,
+                                   safe_unicode(self.repo_name))
+
+    @hybrid_property
+    def landing_rev(self):
+        # always should return [rev_type, rev]
+        if self._landing_revision:
+            _rev_info = self._landing_revision.split(':')
+            if len(_rev_info) < 2:
+                _rev_info.insert(0, 'rev')
+            return [_rev_info[0], _rev_info[1]]
+        return [None, None]
+
+    @landing_rev.setter
+    def landing_rev(self, val):
+        if ':' not in val:
+            raise ValueError('value must be delimited with `:` and consist '
+                             'of <rev_type>:<rev>, got %s instead' % val)
+        self._landing_revision = val
+
+    @hybrid_property
+    def locked(self):
+        if self._locked:
+            user_id, timelocked, reason = self._locked.split(':')
+            lock_values = int(user_id), timelocked, reason
+        else:
+            lock_values = [None, None, None]
+        return lock_values
+
+    @locked.setter
+    def locked(self, val):
+        if val and isinstance(val, (list, tuple)):
+            self._locked = ':'.join(map(str, val))
+        else:
+            self._locked = None
+
+    @hybrid_property
+    def changeset_cache(self):
+        from rhodecode.lib.vcs.backends.base import EmptyCommit
+        dummy = EmptyCommit().__json__()
+        if not self._changeset_cache:
+            return dummy
+        try:
+            return json.loads(self._changeset_cache)
+        except TypeError:
+            return dummy
+        except Exception:
+            log.error(traceback.format_exc())
+            return dummy
+
+    @changeset_cache.setter
+    def changeset_cache(self, val):
+        try:
+            self._changeset_cache = json.dumps(val)
+        except Exception:
+            log.error(traceback.format_exc())
+
+    @hybrid_property
+    def repo_name(self):
+        return self._repo_name
+
+    @repo_name.setter
+    def repo_name(self, value):
+        self._repo_name = value
+        self._repo_name_hash = hashlib.sha1(safe_str(value)).hexdigest()
+
+    @classmethod
+    def normalize_repo_name(cls, repo_name):
+        """
+        Normalizes os specific repo_name to the format internally stored inside
+        database using URL_SEP
+
+        :param cls:
+        :param repo_name:
+        """
+        return cls.NAME_SEP.join(repo_name.split(os.sep))
+
+    @classmethod
+    def get_by_repo_name(cls, repo_name, cache=False, identity_cache=False):
+        session = Session()
+        q = session.query(cls).filter(cls.repo_name == repo_name)
+
+        if cache:
+            if identity_cache:
+                val = cls.identity_cache(session, 'repo_name', repo_name)
+                if val:
+                    return val
+            else:
+                q = q.options(
+                    FromCache("sql_cache_short",
+                              "get_repo_by_name_%s" % _hash_key(repo_name)))
+
+        return q.scalar()
+
+    @classmethod
+    def get_by_full_path(cls, repo_full_path):
+        repo_name = repo_full_path.split(cls.base_path(), 1)[-1]
+        repo_name = cls.normalize_repo_name(repo_name)
+        return cls.get_by_repo_name(repo_name.strip(URL_SEP))
+
+    @classmethod
+    def get_repo_forks(cls, repo_id):
+        return cls.query().filter(Repository.fork_id == repo_id)
+
+    @classmethod
+    def base_path(cls):
+        """
+        Returns base path when all repos are stored
+
+        :param cls:
+        """
+        q = Session().query(RhodeCodeUi)\
+            .filter(RhodeCodeUi.ui_key == cls.NAME_SEP)
+        q = q.options(FromCache("sql_cache_short", "repository_repo_path"))
+        return q.one().ui_value
+
+    @classmethod
+    def is_valid(cls, repo_name):
+        """
+        returns True if given repo name is a valid filesystem repository
+
+        :param cls:
+        :param repo_name:
+        """
+        from rhodecode.lib.utils import is_valid_repo
+
+        return is_valid_repo(repo_name, cls.base_path())
+
+    @classmethod
+    def get_all_repos(cls, user_id=Optional(None), group_id=Optional(None),
+                      case_insensitive=True):
+        q = Repository.query()
+
+        if not isinstance(user_id, Optional):
+            q = q.filter(Repository.user_id == user_id)
+
+        if not isinstance(group_id, Optional):
+            q = q.filter(Repository.group_id == group_id)
+
+        if case_insensitive:
+            q = q.order_by(func.lower(Repository.repo_name))
+        else:
+            q = q.order_by(Repository.repo_name)
+        return q.all()
+
+    @property
+    def forks(self):
+        """
+        Return forks of this repo
+        """
+        return Repository.get_repo_forks(self.repo_id)
+
+    @property
+    def parent(self):
+        """
+        Returns fork parent
+        """
+        return self.fork
+
+    @property
+    def just_name(self):
+        return self.repo_name.split(self.NAME_SEP)[-1]
+
+    @property
+    def groups_with_parents(self):
+        groups = []
+        if self.group is None:
+            return groups
+
+        cur_gr = self.group
+        groups.insert(0, cur_gr)
+        while 1:
+            gr = getattr(cur_gr, 'parent_group', None)
+            cur_gr = cur_gr.parent_group
+            if gr is None:
+                break
+            groups.insert(0, gr)
+
+        return groups
+
+    @property
+    def groups_and_repo(self):
+        return self.groups_with_parents, self
+
+    @LazyProperty
+    def repo_path(self):
+        """
+        Returns base full path for that repository means where it actually
+        exists on a filesystem
+        """
+        q = Session().query(RhodeCodeUi).filter(
+            RhodeCodeUi.ui_key == self.NAME_SEP)
+        q = q.options(FromCache("sql_cache_short", "repository_repo_path"))
+        return q.one().ui_value
+
+    @property
+    def repo_full_path(self):
+        p = [self.repo_path]
+        # we need to split the name by / since this is how we store the
+        # names in the database, but that eventually needs to be converted
+        # into a valid system path
+        p += self.repo_name.split(self.NAME_SEP)
+        return os.path.join(*map(safe_unicode, p))
+
+    @property
+    def cache_keys(self):
+        """
+        Returns associated cache keys for that repo
+        """
+        return CacheKey.query()\
+            .filter(CacheKey.cache_args == self.repo_name)\
+            .order_by(CacheKey.cache_key)\
+            .all()
+
+    def get_new_name(self, repo_name):
+        """
+        returns new full repository name based on assigned group and new new
+
+        :param group_name:
+        """
+        path_prefix = self.group.full_path_splitted if self.group else []
+        return self.NAME_SEP.join(path_prefix + [repo_name])
+
+    @property
+    def _config(self):
+        """
+        Returns db based config object.
+        """
+        from rhodecode.lib.utils import make_db_config
+        return make_db_config(clear_session=False, repo=self)
+
+    def permissions(self, with_admins=True, with_owner=True):
+        q = UserRepoToPerm.query().filter(UserRepoToPerm.repository == self)
+        q = q.options(joinedload(UserRepoToPerm.repository),
+                      joinedload(UserRepoToPerm.user),
+                      joinedload(UserRepoToPerm.permission),)
+
+        # get owners and admins and permissions. We do a trick of re-writing
+        # objects from sqlalchemy to named-tuples due to sqlalchemy session
+        # has a global reference and changing one object propagates to all
+        # others. This means if admin is also an owner admin_row that change
+        # would propagate to both objects
+        perm_rows = []
+        for _usr in q.all():
+            usr = AttributeDict(_usr.user.get_dict())
+            usr.permission = _usr.permission.permission_name
+            perm_rows.append(usr)
+
+        # filter the perm rows by 'default' first and then sort them by
+        # admin,write,read,none permissions sorted again alphabetically in
+        # each group
+        perm_rows = sorted(perm_rows, key=display_sort)
+
+        _admin_perm = 'repository.admin'
+        owner_row = []
+        if with_owner:
+            usr = AttributeDict(self.user.get_dict())
+            usr.owner_row = True
+            usr.permission = _admin_perm
+            owner_row.append(usr)
+
+        super_admin_rows = []
+        if with_admins:
+            for usr in User.get_all_super_admins():
+                # if this admin is also owner, don't double the record
+                if usr.user_id == owner_row[0].user_id:
+                    owner_row[0].admin_row = True
+                else:
+                    usr = AttributeDict(usr.get_dict())
+                    usr.admin_row = True
+                    usr.permission = _admin_perm
+                    super_admin_rows.append(usr)
+
+        return super_admin_rows + owner_row + perm_rows
+
+    def permission_user_groups(self):
+        q = UserGroupRepoToPerm.query().filter(
+            UserGroupRepoToPerm.repository == self)
+        q = q.options(joinedload(UserGroupRepoToPerm.repository),
+                      joinedload(UserGroupRepoToPerm.users_group),
+                      joinedload(UserGroupRepoToPerm.permission),)
+
+        perm_rows = []
+        for _user_group in q.all():
+            usr = AttributeDict(_user_group.users_group.get_dict())
+            usr.permission = _user_group.permission.permission_name
+            perm_rows.append(usr)
+
+        return perm_rows
+
+    def get_api_data(self, include_secrets=False):
+        """
+        Common function for generating repo api data
+
+        :param include_secrets: See :meth:`User.get_api_data`.
+
+        """
+        # TODO: mikhail: Here there is an anti-pattern, we probably need to
+        # move this methods on models level.
+        from rhodecode.model.settings import SettingsModel
+
+        repo = self
+        _user_id, _time, _reason = self.locked
+
+        data = {
+            'repo_id': repo.repo_id,
+            'repo_name': repo.repo_name,
+            'repo_type': repo.repo_type,
+            'clone_uri': repo.clone_uri or '',
+            'url': url('summary_home', repo_name=self.repo_name, qualified=True),
+            'private': repo.private,
+            'created_on': repo.created_on,
+            'description': repo.description,
+            'landing_rev': repo.landing_rev,
+            'owner': repo.user.username,
+            'fork_of': repo.fork.repo_name if repo.fork else None,
+            'enable_statistics': repo.enable_statistics,
+            'enable_locking': repo.enable_locking,
+            'enable_downloads': repo.enable_downloads,
+            'last_changeset': repo.changeset_cache,
+            'locked_by': User.get(_user_id).get_api_data(
+                include_secrets=include_secrets) if _user_id else None,
+            'locked_date': time_to_datetime(_time) if _time else None,
+            'lock_reason': _reason if _reason else None,
+        }
+
+        # TODO: mikhail: should be per-repo settings here
+        rc_config = SettingsModel().get_all_settings()
+        repository_fields = str2bool(
+            rc_config.get('rhodecode_repository_fields'))
+        if repository_fields:
+            for f in self.extra_fields:
+                data[f.field_key_prefixed] = f.field_value
+
+        return data
+
+    @classmethod
+    def lock(cls, repo, user_id, lock_time=None, lock_reason=None):
+        if not lock_time:
+            lock_time = time.time()
+        if not lock_reason:
+            lock_reason = cls.LOCK_AUTOMATIC
+        repo.locked = [user_id, lock_time, lock_reason]
+        Session().add(repo)
+        Session().commit()
+
+    @classmethod
+    def unlock(cls, repo):
+        repo.locked = None
+        Session().add(repo)
+        Session().commit()
+
+    @classmethod
+    def getlock(cls, repo):
+        return repo.locked
+
+    def is_user_lock(self, user_id):
+        if self.lock[0]:
+            lock_user_id = safe_int(self.lock[0])
+            user_id = safe_int(user_id)
+            # both are ints, and they are equal
+            return all([lock_user_id, user_id]) and lock_user_id == user_id
+
+        return False
+
+    def get_locking_state(self, action, user_id, only_when_enabled=True):
+        """
+        Checks locking on this repository, if locking is enabled and lock is
+        present returns a tuple of make_lock, locked, locked_by.
+        make_lock can have 3 states None (do nothing) True, make lock
+        False release lock, This value is later propagated to hooks, which
+        do the locking. Think about this as signals passed to hooks what to do.
+
+        """
+        # TODO: johbo: This is part of the business logic and should be moved
+        # into the RepositoryModel.
+
+        if action not in ('push', 'pull'):
+            raise ValueError("Invalid action value: %s" % repr(action))
+
+        # defines if locked error should be thrown to user
+        currently_locked = False
+        # defines if new lock should be made, tri-state
+        make_lock = None
+        repo = self
+        user = User.get(user_id)
+
+        lock_info = repo.locked
+
+        if repo and (repo.enable_locking or not only_when_enabled):
+            if action == 'push':
+                # check if it's already locked !, if it is compare users
+                locked_by_user_id = lock_info[0]
+                if user.user_id == locked_by_user_id:
+                    log.debug(
+                        'Got `push` action from user %s, now unlocking', user)
+                    # unlock if we have push from user who locked
+                    make_lock = False
+                else:
+                    # we're not the same user who locked, ban with
+                    # code defined in settings (default is 423 HTTP Locked) !
+                    log.debug('Repo %s is currently locked by %s', repo, user)
+                    currently_locked = True
+            elif action == 'pull':
+                # [0] user [1] date
+                if lock_info[0] and lock_info[1]:
+                    log.debug('Repo %s is currently locked by %s', repo, user)
+                    currently_locked = True
+                else:
+                    log.debug('Setting lock on repo %s by %s', repo, user)
+                    make_lock = True
+
+        else:
+            log.debug('Repository %s do not have locking enabled', repo)
+
+        log.debug('FINAL locking values make_lock:%s,locked:%s,locked_by:%s',
+                  make_lock, currently_locked, lock_info)
+
+        from rhodecode.lib.auth import HasRepoPermissionAny
+        perm_check = HasRepoPermissionAny('repository.write', 'repository.admin')
+        if make_lock and not perm_check(repo_name=repo.repo_name, user=user):
+            # if we don't have at least write permission we cannot make a lock
+            log.debug('lock state reset back to FALSE due to lack '
+                      'of at least read permission')
+            make_lock = False
+
+        return make_lock, currently_locked, lock_info
+
+    @property
+    def last_db_change(self):
+        return self.updated_on
+
+    @property
+    def clone_uri_hidden(self):
+        clone_uri = self.clone_uri
+        if clone_uri:
+            import urlobject
+            url_obj = urlobject.URLObject(clone_uri)
+            if url_obj.password:
+                clone_uri = url_obj.with_password('*****')
+        return clone_uri
+
+    def clone_url(self, **override):
+        qualified_home_url = url('home', qualified=True)
+
+        uri_tmpl = None
+        if 'with_id' in override:
+            uri_tmpl = self.DEFAULT_CLONE_URI_ID
+            del override['with_id']
+
+        if 'uri_tmpl' in override:
+            uri_tmpl = override['uri_tmpl']
+            del override['uri_tmpl']
+
+        # we didn't override our tmpl from **overrides
+        if not uri_tmpl:
+            uri_tmpl = self.DEFAULT_CLONE_URI
+            try:
+                from pylons import tmpl_context as c
+                uri_tmpl = c.clone_uri_tmpl
+            except Exception:
+                # in any case if we call this outside of request context,
+                # ie, not having tmpl_context set up
+                pass
+
+        return get_clone_url(uri_tmpl=uri_tmpl,
+                             qualifed_home_url=qualified_home_url,
+                             repo_name=self.repo_name,
+                             repo_id=self.repo_id, **override)
+
+    def set_state(self, state):
+        self.repo_state = state
+        Session().add(self)
+    #==========================================================================
+    # SCM PROPERTIES
+    #==========================================================================
+
+    def get_commit(self, commit_id=None, commit_idx=None, pre_load=None):
+        return get_commit_safe(
+            self.scm_instance(), commit_id, commit_idx, pre_load=pre_load)
+
+    def get_changeset(self, rev=None, pre_load=None):
+        warnings.warn("Use get_commit", DeprecationWarning)
+        commit_id = None
+        commit_idx = None
+        if isinstance(rev, basestring):
+            commit_id = rev
+        else:
+            commit_idx = rev
+        return self.get_commit(commit_id=commit_id, commit_idx=commit_idx,
+                               pre_load=pre_load)
+
+    def get_landing_commit(self):
+        """
+        Returns landing commit, or if that doesn't exist returns the tip
+        """
+        _rev_type, _rev = self.landing_rev
+        commit = self.get_commit(_rev)
+        if isinstance(commit, EmptyCommit):
+            return self.get_commit()
+        return commit
+
+    def update_commit_cache(self, cs_cache=None, config=None):
+        """
+        Update cache of last changeset for repository, keys should be::
+
+            short_id
+            raw_id
+            revision
+            parents
+            message
+            date
+            author
+
+        :param cs_cache:
+        """
+        from rhodecode.lib.vcs.backends.base import BaseChangeset
+        if cs_cache is None:
+            # use no-cache version here
+            scm_repo = self.scm_instance(cache=False, config=config)
+            if scm_repo:
+                cs_cache = scm_repo.get_commit(
+                    pre_load=["author", "date", "message", "parents"])
+            else:
+                cs_cache = EmptyCommit()
+
+        if isinstance(cs_cache, BaseChangeset):
+            cs_cache = cs_cache.__json__()
+
+        def is_outdated(new_cs_cache):
+            if (new_cs_cache['raw_id'] != self.changeset_cache['raw_id'] or
+                new_cs_cache['revision'] != self.changeset_cache['revision']):
+                return True
+            return False
+
+        # check if we have maybe already latest cached revision
+        if is_outdated(cs_cache) or not self.changeset_cache:
+            _default = datetime.datetime.fromtimestamp(0)
+            last_change = cs_cache.get('date') or _default
+            log.debug('updated repo %s with new cs cache %s',
+                      self.repo_name, cs_cache)
+            self.updated_on = last_change
+            self.changeset_cache = cs_cache
+            Session().add(self)
+            Session().commit()
+        else:
+            log.debug('Skipping update_commit_cache for repo:`%s` '
+                      'commit already with latest changes', self.repo_name)
+
+    @property
+    def tip(self):
+        return self.get_commit('tip')
+
+    @property
+    def author(self):
+        return self.tip.author
+
+    @property
+    def last_change(self):
+        return self.scm_instance().last_change
+
+    def get_comments(self, revisions=None):
+        """
+        Returns comments for this repository grouped by revisions
+
+        :param revisions: filter query by revisions only
+        """
+        cmts = ChangesetComment.query()\
+            .filter(ChangesetComment.repo == self)
+        if revisions:
+            cmts = cmts.filter(ChangesetComment.revision.in_(revisions))
+        grouped = collections.defaultdict(list)
+        for cmt in cmts.all():
+            grouped[cmt.revision].append(cmt)
+        return grouped
+
+    def statuses(self, revisions=None):
+        """
+        Returns statuses for this repository
+
+        :param revisions: list of revisions to get statuses for
+        """
+        statuses = ChangesetStatus.query()\
+            .filter(ChangesetStatus.repo == self)\
+            .filter(ChangesetStatus.version == 0)
+
+        if revisions:
+            # Try doing the filtering in chunks to avoid hitting limits
+            size = 500
+            status_results = []
+            for chunk in xrange(0, len(revisions), size):
+                status_results += statuses.filter(
+                    ChangesetStatus.revision.in_(
+                        revisions[chunk: chunk+size])
+                ).all()
+        else:
+            status_results = statuses.all()
+
+        grouped = {}
+
+        # maybe we have open new pullrequest without a status?
+        stat = ChangesetStatus.STATUS_UNDER_REVIEW
+        status_lbl = ChangesetStatus.get_status_lbl(stat)
+        for pr in PullRequest.query().filter(PullRequest.source_repo == self).all():
+            for rev in pr.revisions:
+                pr_id = pr.pull_request_id
+                pr_repo = pr.target_repo.repo_name
+                grouped[rev] = [stat, status_lbl, pr_id, pr_repo]
+
+        for stat in status_results:
+            pr_id = pr_repo = None
+            if stat.pull_request:
+                pr_id = stat.pull_request.pull_request_id
+                pr_repo = stat.pull_request.target_repo.repo_name
+            grouped[stat.revision] = [str(stat.status), stat.status_lbl,
+                                      pr_id, pr_repo]
+        return grouped
+
+    # ==========================================================================
+    # SCM CACHE INSTANCE
+    # ==========================================================================
+
+    def scm_instance(self, **kwargs):
+        import rhodecode
+
+        # Passing a config will not hit the cache currently only used
+        # for repo2dbmapper
+        config = kwargs.pop('config', None)
+        cache = kwargs.pop('cache', None)
+        full_cache = str2bool(rhodecode.CONFIG.get('vcs_full_cache'))
+        # if cache is NOT defined use default global, else we have a full
+        # control over cache behaviour
+        if cache is None and full_cache and not config:
+            return self._get_instance_cached()
+        return self._get_instance(cache=bool(cache), config=config)
+
+    def _get_instance_cached(self):
+        @cache_region('long_term')
+        def _get_repo(cache_key):
+            return self._get_instance()
+
+        invalidator_context = CacheKey.repo_context_cache(
+            _get_repo, self.repo_name, None, thread_scoped=True)
+
+        with invalidator_context as context:
+            context.invalidate()
+            repo = context.compute()
+
+        return repo
+
+    def _get_instance(self, cache=True, config=None):
+        config = config or self._config
+        custom_wire = {
+            'cache': cache  # controls the vcs.remote cache
+        }
+
+        repo = get_vcs_instance(
+            repo_path=safe_str(self.repo_full_path),
+            config=config,
+            with_wire=custom_wire,
+            create=False)
+
+        return repo
+
+    def __json__(self):
+        return {'landing_rev': self.landing_rev}
+
+    def get_dict(self):
+
+        # Since we transformed `repo_name` to a hybrid property, we need to
+        # keep compatibility with the code which uses `repo_name` field.
+
+        result = super(Repository, self).get_dict()
+        result['repo_name'] = result.pop('_repo_name', None)
+        return result
+
+
+class RepoGroup(Base, BaseModel):
+    __tablename__ = 'groups'
+    __table_args__ = (
+        UniqueConstraint('group_name', 'group_parent_id'),
+        CheckConstraint('group_id != group_parent_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+    __mapper_args__ = {'order_by': 'group_name'}
+
+    CHOICES_SEPARATOR = '/'  # used to generate select2 choices for nested groups
+
+    group_id = Column("group_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    group_name = Column("group_name", String(255), nullable=False, unique=True, default=None)
+    group_parent_id = Column("group_parent_id", Integer(), ForeignKey('groups.group_id'), nullable=True, unique=None, default=None)
+    group_description = Column("group_description", String(10000), nullable=True, unique=None, default=None)
+    enable_locking = Column("enable_locking", Boolean(), nullable=False, unique=None, default=False)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=False, default=None)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+
+    repo_group_to_perm = relationship('UserRepoGroupToPerm', cascade='all', order_by='UserRepoGroupToPerm.group_to_perm_id')
+    users_group_to_perm = relationship('UserGroupRepoGroupToPerm', cascade='all')
+    parent_group = relationship('RepoGroup', remote_side=group_id)
+    user = relationship('User')
+    integrations = relationship('Integration',
+                                cascade="all, delete, delete-orphan")
+
+    def __init__(self, group_name='', parent_group=None):
+        self.group_name = group_name
+        self.parent_group = parent_group
+
+    def __unicode__(self):
+        return u"<%s('id:%s:%s')>" % (self.__class__.__name__, self.group_id,
+                                      self.group_name)
+
+    @classmethod
+    def _generate_choice(cls, repo_group):
+        from webhelpers.html import literal as _literal
+        _name = lambda k: _literal(cls.CHOICES_SEPARATOR.join(k))
+        return repo_group.group_id, _name(repo_group.full_path_splitted)
+
+    @classmethod
+    def groups_choices(cls, groups=None, show_empty_group=True):
+        if not groups:
+            groups = cls.query().all()
+
+        repo_groups = []
+        if show_empty_group:
+            repo_groups = [('-1', u'-- %s --' % _('No parent'))]
+
+        repo_groups.extend([cls._generate_choice(x) for x in groups])
+
+        repo_groups = sorted(
+            repo_groups, key=lambda t: t[1].split(cls.CHOICES_SEPARATOR)[0])
+        return repo_groups
+
+    @classmethod
+    def url_sep(cls):
+        return URL_SEP
+
+    @classmethod
+    def get_by_group_name(cls, group_name, cache=False, case_insensitive=False):
+        if case_insensitive:
+            gr = cls.query().filter(func.lower(cls.group_name)
+                                    == func.lower(group_name))
+        else:
+            gr = cls.query().filter(cls.group_name == group_name)
+        if cache:
+            gr = gr.options(FromCache(
+                            "sql_cache_short",
+                            "get_group_%s" % _hash_key(group_name)))
+        return gr.scalar()
+
+    @classmethod
+    def get_all_repo_groups(cls, user_id=Optional(None), group_id=Optional(None),
+                            case_insensitive=True):
+        q = RepoGroup.query()
+
+        if not isinstance(user_id, Optional):
+            q = q.filter(RepoGroup.user_id == user_id)
+
+        if not isinstance(group_id, Optional):
+            q = q.filter(RepoGroup.group_parent_id == group_id)
+
+        if case_insensitive:
+            q = q.order_by(func.lower(RepoGroup.group_name))
+        else:
+            q = q.order_by(RepoGroup.group_name)
+        return q.all()
+
+    @property
+    def parents(self):
+        parents_recursion_limit = 10
+        groups = []
+        if self.parent_group is None:
+            return groups
+        cur_gr = self.parent_group
+        groups.insert(0, cur_gr)
+        cnt = 0
+        while 1:
+            cnt += 1
+            gr = getattr(cur_gr, 'parent_group', None)
+            cur_gr = cur_gr.parent_group
+            if gr is None:
+                break
+            if cnt == parents_recursion_limit:
+                # this will prevent accidental infinit loops
+                log.error(('more than %s parents found for group %s, stopping '
+                           'recursive parent fetching' % (parents_recursion_limit, self)))
+                break
+
+            groups.insert(0, gr)
+        return groups
+
+    @property
+    def children(self):
+        return RepoGroup.query().filter(RepoGroup.parent_group == self)
+
+    @property
+    def name(self):
+        return self.group_name.split(RepoGroup.url_sep())[-1]
+
+    @property
+    def full_path(self):
+        return self.group_name
+
+    @property
+    def full_path_splitted(self):
+        return self.group_name.split(RepoGroup.url_sep())
+
+    @property
+    def repositories(self):
+        return Repository.query()\
+                .filter(Repository.group == self)\
+                .order_by(Repository.repo_name)
+
+    @property
+    def repositories_recursive_count(self):
+        cnt = self.repositories.count()
+
+        def children_count(group):
+            cnt = 0
+            for child in group.children:
+                cnt += child.repositories.count()
+                cnt += children_count(child)
+            return cnt
+
+        return cnt + children_count(self)
+
+    def _recursive_objects(self, include_repos=True):
+        all_ = []
+
+        def _get_members(root_gr):
+            if include_repos:
+                for r in root_gr.repositories:
+                    all_.append(r)
+            childs = root_gr.children.all()
+            if childs:
+                for gr in childs:
+                    all_.append(gr)
+                    _get_members(gr)
+
+        _get_members(self)
+        return [self] + all_
+
+    def recursive_groups_and_repos(self):
+        """
+        Recursive return all groups, with repositories in those groups
+        """
+        return self._recursive_objects()
+
+    def recursive_groups(self):
+        """
+        Returns all children groups for this group including children of children
+        """
+        return self._recursive_objects(include_repos=False)
+
+    def get_new_name(self, group_name):
+        """
+        returns new full group name based on parent and new name
+
+        :param group_name:
+        """
+        path_prefix = (self.parent_group.full_path_splitted if
+                       self.parent_group else [])
+        return RepoGroup.url_sep().join(path_prefix + [group_name])
+
+    def permissions(self, with_admins=True, with_owner=True):
+        q = UserRepoGroupToPerm.query().filter(UserRepoGroupToPerm.group == self)
+        q = q.options(joinedload(UserRepoGroupToPerm.group),
+                      joinedload(UserRepoGroupToPerm.user),
+                      joinedload(UserRepoGroupToPerm.permission),)
+
+        # get owners and admins and permissions. We do a trick of re-writing
+        # objects from sqlalchemy to named-tuples due to sqlalchemy session
+        # has a global reference and changing one object propagates to all
+        # others. This means if admin is also an owner admin_row that change
+        # would propagate to both objects
+        perm_rows = []
+        for _usr in q.all():
+            usr = AttributeDict(_usr.user.get_dict())
+            usr.permission = _usr.permission.permission_name
+            perm_rows.append(usr)
+
+        # filter the perm rows by 'default' first and then sort them by
+        # admin,write,read,none permissions sorted again alphabetically in
+        # each group
+        perm_rows = sorted(perm_rows, key=display_sort)
+
+        _admin_perm = 'group.admin'
+        owner_row = []
+        if with_owner:
+            usr = AttributeDict(self.user.get_dict())
+            usr.owner_row = True
+            usr.permission = _admin_perm
+            owner_row.append(usr)
+
+        super_admin_rows = []
+        if with_admins:
+            for usr in User.get_all_super_admins():
+                # if this admin is also owner, don't double the record
+                if usr.user_id == owner_row[0].user_id:
+                    owner_row[0].admin_row = True
+                else:
+                    usr = AttributeDict(usr.get_dict())
+                    usr.admin_row = True
+                    usr.permission = _admin_perm
+                    super_admin_rows.append(usr)
+
+        return super_admin_rows + owner_row + perm_rows
+
+    def permission_user_groups(self):
+        q = UserGroupRepoGroupToPerm.query().filter(UserGroupRepoGroupToPerm.group == self)
+        q = q.options(joinedload(UserGroupRepoGroupToPerm.group),
+                      joinedload(UserGroupRepoGroupToPerm.users_group),
+                      joinedload(UserGroupRepoGroupToPerm.permission),)
+
+        perm_rows = []
+        for _user_group in q.all():
+            usr = AttributeDict(_user_group.users_group.get_dict())
+            usr.permission = _user_group.permission.permission_name
+            perm_rows.append(usr)
+
+        return perm_rows
+
+    def get_api_data(self):
+        """
+        Common function for generating api data
+
+        """
+        group = self
+        data = {
+            'group_id': group.group_id,
+            'group_name': group.group_name,
+            'group_description': group.group_description,
+            'parent_group': group.parent_group.group_name if group.parent_group else None,
+            'repositories': [x.repo_name for x in group.repositories],
+            'owner': group.user.username,
+        }
+        return data
+
+
+class Permission(Base, BaseModel):
+    __tablename__ = 'permissions'
+    __table_args__ = (
+        Index('p_perm_name_idx', 'permission_name'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+    PERMS = [
+        ('hg.admin', _('RhodeCode Super Administrator')),
+
+        ('repository.none', _('Repository no access')),
+        ('repository.read', _('Repository read access')),
+        ('repository.write', _('Repository write access')),
+        ('repository.admin', _('Repository admin access')),
+
+        ('group.none', _('Repository group no access')),
+        ('group.read', _('Repository group read access')),
+        ('group.write', _('Repository group write access')),
+        ('group.admin', _('Repository group admin access')),
+
+        ('usergroup.none', _('User group no access')),
+        ('usergroup.read', _('User group read access')),
+        ('usergroup.write', _('User group write access')),
+        ('usergroup.admin', _('User group admin access')),
+
+        ('hg.repogroup.create.false', _('Repository Group creation disabled')),
+        ('hg.repogroup.create.true', _('Repository Group creation enabled')),
+
+        ('hg.usergroup.create.false', _('User Group creation disabled')),
+        ('hg.usergroup.create.true', _('User Group creation enabled')),
+
+        ('hg.create.none', _('Repository creation disabled')),
+        ('hg.create.repository', _('Repository creation enabled')),
+        ('hg.create.write_on_repogroup.true', _('Repository creation enabled with write permission to a repository group')),
+        ('hg.create.write_on_repogroup.false', _('Repository creation disabled with write permission to a repository group')),
+
+        ('hg.fork.none', _('Repository forking disabled')),
+        ('hg.fork.repository', _('Repository forking enabled')),
+
+        ('hg.register.none', _('Registration disabled')),
+        ('hg.register.manual_activate', _('User Registration with manual account activation')),
+        ('hg.register.auto_activate', _('User Registration with automatic account activation')),
+
+        ('hg.extern_activate.manual', _('Manual activation of external account')),
+        ('hg.extern_activate.auto', _('Automatic activation of external account')),
+
+        ('hg.inherit_default_perms.false', _('Inherit object permissions from default user disabled')),
+        ('hg.inherit_default_perms.true', _('Inherit object permissions from default user enabled')),
+    ]
+
+    # definition of system default permissions for DEFAULT user
+    DEFAULT_USER_PERMISSIONS = [
+        'repository.read',
+        'group.read',
+        'usergroup.read',
+        'hg.create.repository',
+        'hg.repogroup.create.false',
+        'hg.usergroup.create.false',
+        'hg.create.write_on_repogroup.true',
+        'hg.fork.repository',
+        'hg.register.manual_activate',
+        'hg.extern_activate.auto',
+        'hg.inherit_default_perms.true',
+    ]
+
+    # defines which permissions are more important higher the more important
+    # Weight defines which permissions are more important.
+    # The higher number the more important.
+    PERM_WEIGHTS = {
+        'repository.none': 0,
+        'repository.read': 1,
+        'repository.write': 3,
+        'repository.admin': 4,
+
+        'group.none': 0,
+        'group.read': 1,
+        'group.write': 3,
+        'group.admin': 4,
+
+        'usergroup.none': 0,
+        'usergroup.read': 1,
+        'usergroup.write': 3,
+        'usergroup.admin': 4,
+
+        'hg.repogroup.create.false': 0,
+        'hg.repogroup.create.true': 1,
+
+        'hg.usergroup.create.false': 0,
+        'hg.usergroup.create.true': 1,
+
+        'hg.fork.none': 0,
+        'hg.fork.repository': 1,
+        'hg.create.none': 0,
+        'hg.create.repository': 1
+    }
+
+    permission_id = Column("permission_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    permission_name = Column("permission_name", String(255), nullable=True, unique=None, default=None)
+    permission_longname = Column("permission_longname", String(255), nullable=True, unique=None, default=None)
+
+    def __unicode__(self):
+        return u"<%s('%s:%s')>" % (
+            self.__class__.__name__, self.permission_id, self.permission_name
+        )
+
+    @classmethod
+    def get_by_key(cls, key):
+        return cls.query().filter(cls.permission_name == key).scalar()
+
+    @classmethod
+    def get_default_repo_perms(cls, user_id, repo_id=None):
+        q = Session().query(UserRepoToPerm, Repository, Permission)\
+            .join((Permission, UserRepoToPerm.permission_id == Permission.permission_id))\
+            .join((Repository, UserRepoToPerm.repository_id == Repository.repo_id))\
+            .filter(UserRepoToPerm.user_id == user_id)
+        if repo_id:
+            q = q.filter(UserRepoToPerm.repository_id == repo_id)
+        return q.all()
+
+    @classmethod
+    def get_default_repo_perms_from_user_group(cls, user_id, repo_id=None):
+        q = Session().query(UserGroupRepoToPerm, Repository, Permission)\
+            .join(
+                Permission,
+                UserGroupRepoToPerm.permission_id == Permission.permission_id)\
+            .join(
+                Repository,
+                UserGroupRepoToPerm.repository_id == Repository.repo_id)\
+            .join(
+                UserGroup,
+                UserGroupRepoToPerm.users_group_id ==
+                UserGroup.users_group_id)\
+            .join(
+                UserGroupMember,
+                UserGroupRepoToPerm.users_group_id ==
+                UserGroupMember.users_group_id)\
+            .filter(
+                UserGroupMember.user_id == user_id,
+                UserGroup.users_group_active == true())
+        if repo_id:
+            q = q.filter(UserGroupRepoToPerm.repository_id == repo_id)
+        return q.all()
+
+    @classmethod
+    def get_default_group_perms(cls, user_id, repo_group_id=None):
+        q = Session().query(UserRepoGroupToPerm, RepoGroup, Permission)\
+            .join((Permission, UserRepoGroupToPerm.permission_id == Permission.permission_id))\
+            .join((RepoGroup, UserRepoGroupToPerm.group_id == RepoGroup.group_id))\
+            .filter(UserRepoGroupToPerm.user_id == user_id)
+        if repo_group_id:
+            q = q.filter(UserRepoGroupToPerm.group_id == repo_group_id)
+        return q.all()
+
+    @classmethod
+    def get_default_group_perms_from_user_group(
+            cls, user_id, repo_group_id=None):
+        q = Session().query(UserGroupRepoGroupToPerm, RepoGroup, Permission)\
+            .join(
+                Permission,
+                UserGroupRepoGroupToPerm.permission_id ==
+                Permission.permission_id)\
+            .join(
+                RepoGroup,
+                UserGroupRepoGroupToPerm.group_id == RepoGroup.group_id)\
+            .join(
+                UserGroup,
+                UserGroupRepoGroupToPerm.users_group_id ==
+                UserGroup.users_group_id)\
+            .join(
+                UserGroupMember,
+                UserGroupRepoGroupToPerm.users_group_id ==
+                UserGroupMember.users_group_id)\
+            .filter(
+                UserGroupMember.user_id == user_id,
+                UserGroup.users_group_active == true())
+        if repo_group_id:
+            q = q.filter(UserGroupRepoGroupToPerm.group_id == repo_group_id)
+        return q.all()
+
+    @classmethod
+    def get_default_user_group_perms(cls, user_id, user_group_id=None):
+        q = Session().query(UserUserGroupToPerm, UserGroup, Permission)\
+            .join((Permission, UserUserGroupToPerm.permission_id == Permission.permission_id))\
+            .join((UserGroup, UserUserGroupToPerm.user_group_id == UserGroup.users_group_id))\
+            .filter(UserUserGroupToPerm.user_id == user_id)
+        if user_group_id:
+            q = q.filter(UserUserGroupToPerm.user_group_id == user_group_id)
+        return q.all()
+
+    @classmethod
+    def get_default_user_group_perms_from_user_group(
+            cls, user_id, user_group_id=None):
+        TargetUserGroup = aliased(UserGroup, name='target_user_group')
+        q = Session().query(UserGroupUserGroupToPerm, UserGroup, Permission)\
+            .join(
+                Permission,
+                UserGroupUserGroupToPerm.permission_id ==
+                Permission.permission_id)\
+            .join(
+                TargetUserGroup,
+                UserGroupUserGroupToPerm.target_user_group_id ==
+                TargetUserGroup.users_group_id)\
+            .join(
+                UserGroup,
+                UserGroupUserGroupToPerm.user_group_id ==
+                UserGroup.users_group_id)\
+            .join(
+                UserGroupMember,
+                UserGroupUserGroupToPerm.user_group_id ==
+                UserGroupMember.users_group_id)\
+            .filter(
+                UserGroupMember.user_id == user_id,
+                UserGroup.users_group_active == true())
+        if user_group_id:
+            q = q.filter(
+                UserGroupUserGroupToPerm.user_group_id == user_group_id)
+
+        return q.all()
+
+
+class UserRepoToPerm(Base, BaseModel):
+    __tablename__ = 'repo_to_perm'
+    __table_args__ = (
+        UniqueConstraint('user_id', 'repository_id', 'permission_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    repo_to_perm_id = Column("repo_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+    repository_id = Column("repository_id", Integer(), ForeignKey('repositories.repo_id'), nullable=False, unique=None, default=None)
+
+    user = relationship('User')
+    repository = relationship('Repository')
+    permission = relationship('Permission')
+
+    @classmethod
+    def create(cls, user, repository, permission):
+        n = cls()
+        n.user = user
+        n.repository = repository
+        n.permission = permission
+        Session().add(n)
+        return n
+
+    def __unicode__(self):
+        return u'<%s => %s >' % (self.user, self.repository)
+
+
+class UserUserGroupToPerm(Base, BaseModel):
+    __tablename__ = 'user_user_group_to_perm'
+    __table_args__ = (
+        UniqueConstraint('user_id', 'user_group_id', 'permission_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    user_user_group_to_perm_id = Column("user_user_group_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+    user_group_id = Column("user_group_id", Integer(), ForeignKey('users_groups.users_group_id'), nullable=False, unique=None, default=None)
+
+    user = relationship('User')
+    user_group = relationship('UserGroup')
+    permission = relationship('Permission')
+
+    @classmethod
+    def create(cls, user, user_group, permission):
+        n = cls()
+        n.user = user
+        n.user_group = user_group
+        n.permission = permission
+        Session().add(n)
+        return n
+
+    def __unicode__(self):
+        return u'<%s => %s >' % (self.user, self.user_group)
+
+
+class UserToPerm(Base, BaseModel):
+    __tablename__ = 'user_to_perm'
+    __table_args__ = (
+        UniqueConstraint('user_id', 'permission_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    user_to_perm_id = Column("user_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+
+    user = relationship('User')
+    permission = relationship('Permission', lazy='joined')
+
+    def __unicode__(self):
+        return u'<%s => %s >' % (self.user, self.permission)
+
+
+class UserGroupRepoToPerm(Base, BaseModel):
+    __tablename__ = 'users_group_repo_to_perm'
+    __table_args__ = (
+        UniqueConstraint('repository_id', 'users_group_id', 'permission_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    users_group_to_perm_id = Column("users_group_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    users_group_id = Column("users_group_id", Integer(), ForeignKey('users_groups.users_group_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+    repository_id = Column("repository_id", Integer(), ForeignKey('repositories.repo_id'), nullable=False, unique=None, default=None)
+
+    users_group = relationship('UserGroup')
+    permission = relationship('Permission')
+    repository = relationship('Repository')
+
+    @classmethod
+    def create(cls, users_group, repository, permission):
+        n = cls()
+        n.users_group = users_group
+        n.repository = repository
+        n.permission = permission
+        Session().add(n)
+        return n
+
+    def __unicode__(self):
+        return u'<UserGroupRepoToPerm:%s => %s >' % (self.users_group, self.repository)
+
+
+class UserGroupUserGroupToPerm(Base, BaseModel):
+    __tablename__ = 'user_group_user_group_to_perm'
+    __table_args__ = (
+        UniqueConstraint('target_user_group_id', 'user_group_id', 'permission_id'),
+        CheckConstraint('target_user_group_id != user_group_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    user_group_user_group_to_perm_id = Column("user_group_user_group_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    target_user_group_id = Column("target_user_group_id", Integer(), ForeignKey('users_groups.users_group_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+    user_group_id = Column("user_group_id", Integer(), ForeignKey('users_groups.users_group_id'), nullable=False, unique=None, default=None)
+
+    target_user_group = relationship('UserGroup', primaryjoin='UserGroupUserGroupToPerm.target_user_group_id==UserGroup.users_group_id')
+    user_group = relationship('UserGroup', primaryjoin='UserGroupUserGroupToPerm.user_group_id==UserGroup.users_group_id')
+    permission = relationship('Permission')
+
+    @classmethod
+    def create(cls, target_user_group, user_group, permission):
+        n = cls()
+        n.target_user_group = target_user_group
+        n.user_group = user_group
+        n.permission = permission
+        Session().add(n)
+        return n
+
+    def __unicode__(self):
+        return u'<UserGroupUserGroup:%s => %s >' % (self.target_user_group, self.user_group)
+
+
+class UserGroupToPerm(Base, BaseModel):
+    __tablename__ = 'users_group_to_perm'
+    __table_args__ = (
+        UniqueConstraint('users_group_id', 'permission_id',),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    users_group_to_perm_id = Column("users_group_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    users_group_id = Column("users_group_id", Integer(), ForeignKey('users_groups.users_group_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+
+    users_group = relationship('UserGroup')
+    permission = relationship('Permission')
+
+
+class UserRepoGroupToPerm(Base, BaseModel):
+    __tablename__ = 'user_repo_group_to_perm'
+    __table_args__ = (
+        UniqueConstraint('user_id', 'group_id', 'permission_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    group_to_perm_id = Column("group_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=None, default=None)
+    group_id = Column("group_id", Integer(), ForeignKey('groups.group_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+
+    user = relationship('User')
+    group = relationship('RepoGroup')
+    permission = relationship('Permission')
+
+    @classmethod
+    def create(cls, user, repository_group, permission):
+        n = cls()
+        n.user = user
+        n.group = repository_group
+        n.permission = permission
+        Session().add(n)
+        return n
+
+
+class UserGroupRepoGroupToPerm(Base, BaseModel):
+    __tablename__ = 'users_group_repo_group_to_perm'
+    __table_args__ = (
+        UniqueConstraint('users_group_id', 'group_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    users_group_repo_group_to_perm_id = Column("users_group_repo_group_to_perm_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    users_group_id = Column("users_group_id", Integer(), ForeignKey('users_groups.users_group_id'), nullable=False, unique=None, default=None)
+    group_id = Column("group_id", Integer(), ForeignKey('groups.group_id'), nullable=False, unique=None, default=None)
+    permission_id = Column("permission_id", Integer(), ForeignKey('permissions.permission_id'), nullable=False, unique=None, default=None)
+
+    users_group = relationship('UserGroup')
+    permission = relationship('Permission')
+    group = relationship('RepoGroup')
+
+    @classmethod
+    def create(cls, user_group, repository_group, permission):
+        n = cls()
+        n.users_group = user_group
+        n.group = repository_group
+        n.permission = permission
+        Session().add(n)
+        return n
+
+    def __unicode__(self):
+        return u'<UserGroupRepoGroupToPerm:%s => %s >' % (self.users_group, self.group)
+
+
+class Statistics(Base, BaseModel):
+    __tablename__ = 'statistics'
+    __table_args__ = (
+         UniqueConstraint('repository_id'),
+         {'extend_existing': True, 'mysql_engine': 'InnoDB',
+          'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    stat_id = Column("stat_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    repository_id = Column("repository_id", Integer(), ForeignKey('repositories.repo_id'), nullable=False, unique=True, default=None)
+    stat_on_revision = Column("stat_on_revision", Integer(), nullable=False)
+    commit_activity = Column("commit_activity", LargeBinary(1000000), nullable=False)#JSON data
+    commit_activity_combined = Column("commit_activity_combined", LargeBinary(), nullable=False)#JSON data
+    languages = Column("languages", LargeBinary(1000000), nullable=False)#JSON data
+
+    repository = relationship('Repository', single_parent=True)
+
+
+class UserFollowing(Base, BaseModel):
+    __tablename__ = 'user_followings'
+    __table_args__ = (
+        UniqueConstraint('user_id', 'follows_repository_id'),
+        UniqueConstraint('user_id', 'follows_user_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    user_following_id = Column("user_following_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=None, default=None)
+    follows_repo_id = Column("follows_repository_id", Integer(), ForeignKey('repositories.repo_id'), nullable=True, unique=None, default=None)
+    follows_user_id = Column("follows_user_id", Integer(), ForeignKey('users.user_id'), nullable=True, unique=None, default=None)
+    follows_from = Column('follows_from', DateTime(timezone=False), nullable=True, unique=None, default=datetime.datetime.now)
+
+    user = relationship('User', primaryjoin='User.user_id==UserFollowing.user_id')
+
+    follows_user = relationship('User', primaryjoin='User.user_id==UserFollowing.follows_user_id')
+    follows_repository = relationship('Repository', order_by='Repository.repo_name')
+
+    @classmethod
+    def get_repo_followers(cls, repo_id):
+        return cls.query().filter(cls.follows_repo_id == repo_id)
+
+
+class CacheKey(Base, BaseModel):
+    __tablename__ = 'cache_invalidation'
+    __table_args__ = (
+        UniqueConstraint('cache_key'),
+        Index('key_idx', 'cache_key'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+    CACHE_TYPE_ATOM = 'ATOM'
+    CACHE_TYPE_RSS = 'RSS'
+    CACHE_TYPE_README = 'README'
+
+    cache_id = Column("cache_id", Integer(), nullable=False, unique=True, default=None, primary_key=True)
+    cache_key = Column("cache_key", String(255), nullable=True, unique=None, default=None)
+    cache_args = Column("cache_args", String(255), nullable=True, unique=None, default=None)
+    cache_active = Column("cache_active", Boolean(), nullable=True, unique=None, default=False)
+
+    def __init__(self, cache_key, cache_args=''):
+        self.cache_key = cache_key
+        self.cache_args = cache_args
+        self.cache_active = False
+
+    def __unicode__(self):
+        return u"<%s('%s:%s[%s]')>" % (
+            self.__class__.__name__,
+            self.cache_id, self.cache_key, self.cache_active)
+
+    def _cache_key_partition(self):
+        prefix, repo_name, suffix = self.cache_key.partition(self.cache_args)
+        return prefix, repo_name, suffix
+
+    def get_prefix(self):
+        """
+        Try to extract prefix from existing cache key. The key could consist
+        of prefix, repo_name, suffix
+        """
+        # this returns prefix, repo_name, suffix
+        return self._cache_key_partition()[0]
+
+    def get_suffix(self):
+        """
+        get suffix that might have been used in _get_cache_key to
+        generate self.cache_key. Only used for informational purposes
+        in repo_edit.html.
+        """
+        # prefix, repo_name, suffix
+        return self._cache_key_partition()[2]
+
+    @classmethod
+    def delete_all_cache(cls):
+        """
+        Delete all cache keys from database.
+        Should only be run when all instances are down and all entries
+        thus stale.
+        """
+        cls.query().delete()
+        Session().commit()
+
+    @classmethod
+    def get_cache_key(cls, repo_name, cache_type):
+        """
+
+        Generate a cache key for this process of RhodeCode instance.
+        Prefix most likely will be process id or maybe explicitly set
+        instance_id from .ini file.
+        """
+        import rhodecode
+        prefix = safe_unicode(rhodecode.CONFIG.get('instance_id') or '')
+
+        repo_as_unicode = safe_unicode(repo_name)
+        key = u'{}_{}'.format(repo_as_unicode, cache_type) \
+            if cache_type else repo_as_unicode
+
+        return u'{}{}'.format(prefix, key)
+
+    @classmethod
+    def set_invalidate(cls, repo_name, delete=False):
+        """
+        Mark all caches of a repo as invalid in the database.
+        """
+
+        try:
+            qry = Session().query(cls).filter(cls.cache_args == repo_name)
+            if delete:
+                log.debug('cache objects deleted for repo %s',
+                          safe_str(repo_name))
+                qry.delete()
+            else:
+                log.debug('cache objects marked as invalid for repo %s',
+                          safe_str(repo_name))
+                qry.update({"cache_active": False})
+
+            Session().commit()
+        except Exception:
+            log.exception(
+                'Cache key invalidation failed for repository %s',
+                safe_str(repo_name))
+            Session().rollback()
+
+    @classmethod
+    def get_active_cache(cls, cache_key):
+        inv_obj = cls.query().filter(cls.cache_key == cache_key).scalar()
+        if inv_obj:
+            return inv_obj
+        return None
+
+    @classmethod
+    def repo_context_cache(cls, compute_func, repo_name, cache_type,
+                           thread_scoped=False):
+        """
+        @cache_region('long_term')
+        def _heavy_calculation(cache_key):
+            return 'result'
+
+        cache_context = CacheKey.repo_context_cache(
+            _heavy_calculation, repo_name, cache_type)
+
+        with cache_context as context:
+            context.invalidate()
+            computed = context.compute()
+
+        assert computed == 'result'
+        """
+        from rhodecode.lib import caches
+        return caches.InvalidationContext(
+            compute_func, repo_name, cache_type, thread_scoped=thread_scoped)
+
+
+class ChangesetComment(Base, BaseModel):
+    __tablename__ = 'changeset_comments'
+    __table_args__ = (
+        Index('cc_revision_idx', 'revision'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+
+    COMMENT_OUTDATED = u'comment_outdated'
+
+    comment_id = Column('comment_id', Integer(), nullable=False, primary_key=True)
+    repo_id = Column('repo_id', Integer(), ForeignKey('repositories.repo_id'), nullable=False)
+    revision = Column('revision', String(40), nullable=True)
+    pull_request_id = Column("pull_request_id", Integer(), ForeignKey('pull_requests.pull_request_id'), nullable=True)
+    pull_request_version_id = Column("pull_request_version_id", Integer(), ForeignKey('pull_request_versions.pull_request_version_id'), nullable=True)
+    line_no = Column('line_no', Unicode(10), nullable=True)
+    hl_lines = Column('hl_lines', Unicode(512), nullable=True)
+    f_path = Column('f_path', Unicode(1000), nullable=True)
+    user_id = Column('user_id', Integer(), ForeignKey('users.user_id'), nullable=False)
+    text = Column('text', UnicodeText().with_variant(UnicodeText(25000), 'mysql'), nullable=False)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+    modified_at = Column('modified_at', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+    renderer = Column('renderer', Unicode(64), nullable=True)
+    display_state = Column('display_state',  Unicode(128), nullable=True)
+
+    author = relationship('User', lazy='joined')
+    repo = relationship('Repository')
+    status_change = relationship('ChangesetStatus', cascade="all, delete, delete-orphan")
+    pull_request = relationship('PullRequest', lazy='joined')
+    pull_request_version = relationship('PullRequestVersion')
+
+    @classmethod
+    def get_users(cls, revision=None, pull_request_id=None):
+        """
+        Returns user associated with this ChangesetComment. ie those
+        who actually commented
+
+        :param cls:
+        :param revision:
+        """
+        q = Session().query(User)\
+                .join(ChangesetComment.author)
+        if revision:
+            q = q.filter(cls.revision == revision)
+        elif pull_request_id:
+            q = q.filter(cls.pull_request_id == pull_request_id)
+        return q.all()
+
+    def render(self, mentions=False):
+        from rhodecode.lib import helpers as h
+        return h.render(self.text, renderer=self.renderer, mentions=mentions)
+
+    def __repr__(self):
+        if self.comment_id:
+            return '<DB:ChangesetComment #%s>' % self.comment_id
+        else:
+            return '<DB:ChangesetComment at %#x>' % id(self)
+
+
+class ChangesetStatus(Base, BaseModel):
+    __tablename__ = 'changeset_statuses'
+    __table_args__ = (
+        Index('cs_revision_idx', 'revision'),
+        Index('cs_version_idx', 'version'),
+        UniqueConstraint('repo_id', 'revision', 'version'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    STATUS_NOT_REVIEWED = DEFAULT = 'not_reviewed'
+    STATUS_APPROVED = 'approved'
+    STATUS_REJECTED = 'rejected'
+    STATUS_UNDER_REVIEW = 'under_review'
+
+    STATUSES = [
+        (STATUS_NOT_REVIEWED, _("Not Reviewed")),  # (no icon) and default
+        (STATUS_APPROVED, _("Approved")),
+        (STATUS_REJECTED, _("Rejected")),
+        (STATUS_UNDER_REVIEW, _("Under Review")),
+    ]
+
+    changeset_status_id = Column('changeset_status_id', Integer(), nullable=False, primary_key=True)
+    repo_id = Column('repo_id', Integer(), ForeignKey('repositories.repo_id'), nullable=False)
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'), nullable=False, unique=None)
+    revision = Column('revision', String(40), nullable=False)
+    status = Column('status', String(128), nullable=False, default=DEFAULT)
+    changeset_comment_id = Column('changeset_comment_id', Integer(), ForeignKey('changeset_comments.comment_id'))
+    modified_at = Column('modified_at', DateTime(), nullable=False, default=datetime.datetime.now)
+    version = Column('version', Integer(), nullable=False, default=0)
+    pull_request_id = Column("pull_request_id", Integer(), ForeignKey('pull_requests.pull_request_id'), nullable=True)
+
+    author = relationship('User', lazy='joined')
+    repo = relationship('Repository')
+    comment = relationship('ChangesetComment', lazy='joined')
+    pull_request = relationship('PullRequest', lazy='joined')
+
+    def __unicode__(self):
+        return u"<%s('%s[%s]:%s')>" % (
+            self.__class__.__name__,
+            self.status, self.version, self.author
+        )
+
+    @classmethod
+    def get_status_lbl(cls, value):
+        return dict(cls.STATUSES).get(value)
+
+    @property
+    def status_lbl(self):
+        return ChangesetStatus.get_status_lbl(self.status)
+
+
+class _PullRequestBase(BaseModel):
+    """
+    Common attributes of pull request and version entries.
+    """
+
+    # .status values
+    STATUS_NEW = u'new'
+    STATUS_OPEN = u'open'
+    STATUS_CLOSED = u'closed'
+
+    title = Column('title', Unicode(255), nullable=True)
+    description = Column(
+        'description', UnicodeText().with_variant(UnicodeText(10240), 'mysql'),
+        nullable=True)
+    # new/open/closed status of pull request (not approve/reject/etc)
+    status = Column('status', Unicode(255), nullable=False, default=STATUS_NEW)
+    created_on = Column(
+        'created_on', DateTime(timezone=False), nullable=False,
+        default=datetime.datetime.now)
+    updated_on = Column(
+        'updated_on', DateTime(timezone=False), nullable=False,
+        default=datetime.datetime.now)
+
+    @declared_attr
+    def user_id(cls):
+        return Column(
+            "user_id", Integer(), ForeignKey('users.user_id'), nullable=False,
+            unique=None)
+
+    # 500 revisions max
+    _revisions = Column(
+        'revisions', UnicodeText().with_variant(UnicodeText(20500), 'mysql'))
+
+    @declared_attr
+    def source_repo_id(cls):
+        # TODO: dan: rename column to source_repo_id
+        return Column(
+            'org_repo_id', Integer(), ForeignKey('repositories.repo_id'),
+            nullable=False)
+
+    source_ref = Column('org_ref', Unicode(255), nullable=False)
+
+    @declared_attr
+    def target_repo_id(cls):
+        # TODO: dan: rename column to target_repo_id
+        return Column(
+            'other_repo_id', Integer(), ForeignKey('repositories.repo_id'),
+            nullable=False)
+
+    target_ref = Column('other_ref', Unicode(255), nullable=False)
+
+    # TODO: dan: rename column to last_merge_source_rev
+    _last_merge_source_rev = Column(
+        'last_merge_org_rev', String(40), nullable=True)
+    # TODO: dan: rename column to last_merge_target_rev
+    _last_merge_target_rev = Column(
+        'last_merge_other_rev', String(40), nullable=True)
+    _last_merge_status = Column('merge_status', Integer(), nullable=True)
+    merge_rev = Column('merge_rev', String(40), nullable=True)
+
+    @hybrid_property
+    def revisions(self):
+        return self._revisions.split(':') if self._revisions else []
+
+    @revisions.setter
+    def revisions(self, val):
+        self._revisions = ':'.join(val)
+
+    @declared_attr
+    def author(cls):
+        return relationship('User', lazy='joined')
+
+    @declared_attr
+    def source_repo(cls):
+        return relationship(
+            'Repository',
+            primaryjoin='%s.source_repo_id==Repository.repo_id' % cls.__name__)
+
+    @property
+    def source_ref_parts(self):
+        refs = self.source_ref.split(':')
+        return Reference(refs[0], refs[1], refs[2])
+
+    @declared_attr
+    def target_repo(cls):
+        return relationship(
+            'Repository',
+            primaryjoin='%s.target_repo_id==Repository.repo_id' % cls.__name__)
+
+    @property
+    def target_ref_parts(self):
+        refs = self.target_ref.split(':')
+        return Reference(refs[0], refs[1], refs[2])
+
+
+class PullRequest(Base, _PullRequestBase):
+    __tablename__ = 'pull_requests'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+
+    pull_request_id = Column(
+        'pull_request_id', Integer(), nullable=False, primary_key=True)
+
+    def __repr__(self):
+        if self.pull_request_id:
+            return '<DB:PullRequest #%s>' % self.pull_request_id
+        else:
+            return '<DB:PullRequest at %#x>' % id(self)
+
+    reviewers = relationship('PullRequestReviewers',
+                             cascade="all, delete, delete-orphan")
+    statuses = relationship('ChangesetStatus')
+    comments = relationship('ChangesetComment',
+                            cascade="all, delete, delete-orphan")
+    versions = relationship('PullRequestVersion',
+                            cascade="all, delete, delete-orphan")
+
+    def is_closed(self):
+        return self.status == self.STATUS_CLOSED
+
+    def get_api_data(self):
+        from rhodecode.model.pull_request import PullRequestModel
+        pull_request = self
+        merge_status = PullRequestModel().merge_status(pull_request)
+        data = {
+            'pull_request_id': pull_request.pull_request_id,
+            'url': url('pullrequest_show', repo_name=self.target_repo.repo_name,
+                                       pull_request_id=self.pull_request_id,
+                                       qualified=True),
+            'title': pull_request.title,
+            'description': pull_request.description,
+            'status': pull_request.status,
+            'created_on': pull_request.created_on,
+            'updated_on': pull_request.updated_on,
+            'commit_ids': pull_request.revisions,
+            'review_status': pull_request.calculated_review_status(),
+            'mergeable': {
+                'status': merge_status[0],
+                'message': unicode(merge_status[1]),
+            },
+            'source': {
+                'clone_url': pull_request.source_repo.clone_url(),
+                'repository': pull_request.source_repo.repo_name,
+                'reference': {
+                    'name': pull_request.source_ref_parts.name,
+                    'type': pull_request.source_ref_parts.type,
+                    'commit_id': pull_request.source_ref_parts.commit_id,
+                },
+            },
+            'target': {
+                'clone_url': pull_request.target_repo.clone_url(),
+                'repository': pull_request.target_repo.repo_name,
+                'reference': {
+                    'name': pull_request.target_ref_parts.name,
+                    'type': pull_request.target_ref_parts.type,
+                    'commit_id': pull_request.target_ref_parts.commit_id,
+                },
+            },
+            'author': pull_request.author.get_api_data(include_secrets=False,
+                                                       details='basic'),
+            'reviewers': [
+                {
+                    'user': reviewer.get_api_data(include_secrets=False,
+                                                  details='basic'),
+                    'review_status': st[0][1].status if st else 'not_reviewed',
+                }
+                for reviewer, st in pull_request.reviewers_statuses()
+            ]
+        }
+
+        return data
+
+    def __json__(self):
+        return {
+            'revisions': self.revisions,
+        }
+
+    def calculated_review_status(self):
+        # TODO: anderson: 13.05.15 Used only on templates/my_account_pullrequests.html
+        # because it's tricky on how to use ChangesetStatusModel from there
+        warnings.warn("Use calculated_review_status from ChangesetStatusModel", DeprecationWarning)
+        from rhodecode.model.changeset_status import ChangesetStatusModel
+        return ChangesetStatusModel().calculated_review_status(self)
+
+    def reviewers_statuses(self):
+        warnings.warn("Use reviewers_statuses from ChangesetStatusModel", DeprecationWarning)
+        from rhodecode.model.changeset_status import ChangesetStatusModel
+        return ChangesetStatusModel().reviewers_statuses(self)
+
+
+class PullRequestVersion(Base, _PullRequestBase):
+    __tablename__ = 'pull_request_versions'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+
+    pull_request_version_id = Column(
+        'pull_request_version_id', Integer(), nullable=False, primary_key=True)
+    pull_request_id = Column(
+        'pull_request_id', Integer(),
+        ForeignKey('pull_requests.pull_request_id'), nullable=False)
+    pull_request = relationship('PullRequest')
+
+    def __repr__(self):
+        if self.pull_request_version_id:
+            return '<DB:PullRequestVersion #%s>' % self.pull_request_version_id
+        else:
+            return '<DB:PullRequestVersion at %#x>' % id(self)
+
+
+class PullRequestReviewers(Base, BaseModel):
+    __tablename__ = 'pull_request_reviewers'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+
+    def __init__(self, user=None, pull_request=None):
+        self.user = user
+        self.pull_request = pull_request
+
+    pull_requests_reviewers_id = Column(
+        'pull_requests_reviewers_id', Integer(), nullable=False,
+        primary_key=True)
+    pull_request_id = Column(
+        "pull_request_id", Integer(),
+        ForeignKey('pull_requests.pull_request_id'), nullable=False)
+    user_id = Column(
+        "user_id", Integer(), ForeignKey('users.user_id'), nullable=True)
+
+    user = relationship('User')
+    pull_request = relationship('PullRequest')
+
+
+class Notification(Base, BaseModel):
+    __tablename__ = 'notifications'
+    __table_args__ = (
+        Index('notification_type_idx', 'type'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+
+    TYPE_CHANGESET_COMMENT = u'cs_comment'
+    TYPE_MESSAGE = u'message'
+    TYPE_MENTION = u'mention'
+    TYPE_REGISTRATION = u'registration'
+    TYPE_PULL_REQUEST = u'pull_request'
+    TYPE_PULL_REQUEST_COMMENT = u'pull_request_comment'
+
+    notification_id = Column('notification_id', Integer(), nullable=False, primary_key=True)
+    subject = Column('subject', Unicode(512), nullable=True)
+    body = Column('body', UnicodeText().with_variant(UnicodeText(50000), 'mysql'), nullable=True)
+    created_by = Column("created_by", Integer(), ForeignKey('users.user_id'), nullable=True)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+    type_ = Column('type', Unicode(255))
+
+    created_by_user = relationship('User')
+    notifications_to_users = relationship('UserNotification', lazy='joined',
+                                          cascade="all, delete, delete-orphan")
+
+    @property
+    def recipients(self):
+        return [x.user for x in UserNotification.query()\
+                .filter(UserNotification.notification == self)\
+                .order_by(UserNotification.user_id.asc()).all()]
+
+    @classmethod
+    def create(cls, created_by, subject, body, recipients, type_=None):
+        if type_ is None:
+            type_ = Notification.TYPE_MESSAGE
+
+        notification = cls()
+        notification.created_by_user = created_by
+        notification.subject = subject
+        notification.body = body
+        notification.type_ = type_
+        notification.created_on = datetime.datetime.now()
+
+        for u in recipients:
+            assoc = UserNotification()
+            assoc.notification = notification
+
+            # if created_by is inside recipients mark his notification
+            # as read
+            if u.user_id == created_by.user_id:
+                assoc.read = True
+
+            u.notifications.append(assoc)
+        Session().add(notification)
+
+        return notification
+
+    @property
+    def description(self):
+        from rhodecode.model.notification import NotificationModel
+        return NotificationModel().make_description(self)
+
+
+class UserNotification(Base, BaseModel):
+    __tablename__ = 'user_to_notification'
+    __table_args__ = (
+        UniqueConstraint('user_id', 'notification_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    user_id = Column('user_id', Integer(), ForeignKey('users.user_id'), primary_key=True)
+    notification_id = Column("notification_id", Integer(), ForeignKey('notifications.notification_id'), primary_key=True)
+    read = Column('read', Boolean, default=False)
+    sent_on = Column('sent_on', DateTime(timezone=False), nullable=True, unique=None)
+
+    user = relationship('User', lazy="joined")
+    notification = relationship('Notification', lazy="joined",
+                                order_by=lambda: Notification.created_on.desc(),)
+
+    def mark_as_read(self):
+        self.read = True
+        Session().add(self)
+
+
+class Gist(Base, BaseModel):
+    __tablename__ = 'gists'
+    __table_args__ = (
+        Index('g_gist_access_id_idx', 'gist_access_id'),
+        Index('g_created_on_idx', 'created_on'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+    GIST_PUBLIC = u'public'
+    GIST_PRIVATE = u'private'
+    DEFAULT_FILENAME = u'gistfile1.txt'
+
+    ACL_LEVEL_PUBLIC = u'acl_public'
+    ACL_LEVEL_PRIVATE = u'acl_private'
+
+    gist_id = Column('gist_id', Integer(), primary_key=True)
+    gist_access_id = Column('gist_access_id', Unicode(250))
+    gist_description = Column('gist_description', UnicodeText().with_variant(UnicodeText(1024), 'mysql'))
+    gist_owner = Column('user_id', Integer(), ForeignKey('users.user_id'), nullable=True)
+    gist_expires = Column('gist_expires', Float(53), nullable=False)
+    gist_type = Column('gist_type', Unicode(128), nullable=False)
+    created_on = Column('created_on', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+    modified_at = Column('modified_at', DateTime(timezone=False), nullable=False, default=datetime.datetime.now)
+    acl_level = Column('acl_level', Unicode(128), nullable=True)
+
+    owner = relationship('User')
+
+    def __repr__(self):
+        return '<Gist:[%s]%s>' % (self.gist_type, self.gist_access_id)
+
+    @classmethod
+    def get_or_404(cls, id_):
+        res = cls.query().filter(cls.gist_access_id == id_).scalar()
+        if not res:
+            raise HTTPNotFound
+        return res
+
+    @classmethod
+    def get_by_access_id(cls, gist_access_id):
+        return cls.query().filter(cls.gist_access_id == gist_access_id).scalar()
+
+    def gist_url(self):
+        import rhodecode
+        alias_url = rhodecode.CONFIG.get('gist_alias_url')
+        if alias_url:
+            return alias_url.replace('{gistid}', self.gist_access_id)
+
+        return url('gist', gist_id=self.gist_access_id, qualified=True)
+
+    @classmethod
+    def base_path(cls):
+        """
+        Returns base path when all gists are stored
+
+        :param cls:
+        """
+        from rhodecode.model.gist import GIST_STORE_LOC
+        q = Session().query(RhodeCodeUi)\
+            .filter(RhodeCodeUi.ui_key == URL_SEP)
+        q = q.options(FromCache("sql_cache_short", "repository_repo_path"))
+        return os.path.join(q.one().ui_value, GIST_STORE_LOC)
+
+    def get_api_data(self):
+        """
+        Common function for generating gist related data for API
+        """
+        gist = self
+        data = {
+            'gist_id': gist.gist_id,
+            'type': gist.gist_type,
+            'access_id': gist.gist_access_id,
+            'description': gist.gist_description,
+            'url': gist.gist_url(),
+            'expires': gist.gist_expires,
+            'created_on': gist.created_on,
+            'modified_at': gist.modified_at,
+            'content': None,
+            'acl_level': gist.acl_level,
+        }
+        return data
+
+    def __json__(self):
+        data = dict(
+        )
+        data.update(self.get_api_data())
+        return data
+    # SCM functions
+
+    def scm_instance(self, **kwargs):
+        full_repo_path = os.path.join(self.base_path(), self.gist_access_id)
+        return get_vcs_instance(
+            repo_path=safe_str(full_repo_path), create=False)
+
+
+class DbMigrateVersion(Base, BaseModel):
+    __tablename__ = 'db_migrate_version'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True},
+    )
+    repository_id = Column('repository_id', String(250), primary_key=True)
+    repository_path = Column('repository_path', Text)
+    version = Column('version', Integer)
+
+
+class ExternalIdentity(Base, BaseModel):
+    __tablename__ = 'external_identities'
+    __table_args__ = (
+        Index('local_user_id_idx', 'local_user_id'),
+        Index('external_id_idx', 'external_id'),
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8'})
+
+    external_id = Column('external_id', Unicode(255), default=u'',
+                         primary_key=True)
+    external_username = Column('external_username', Unicode(1024), default=u'')
+    local_user_id = Column('local_user_id', Integer(),
+                           ForeignKey('users.user_id'), primary_key=True)
+    provider_name = Column('provider_name', Unicode(255), default=u'',
+                           primary_key=True)
+    access_token = Column('access_token', String(1024), default=u'')
+    alt_token = Column('alt_token', String(1024), default=u'')
+    token_secret = Column('token_secret', String(1024), default=u'')
+
+    @classmethod
+    def by_external_id_and_provider(cls, external_id, provider_name,
+                                    local_user_id=None):
+        """
+        Returns ExternalIdentity instance based on search params
+
+        :param external_id:
+        :param provider_name:
+        :return: ExternalIdentity
+        """
+        query = cls.query()
+        query = query.filter(cls.external_id == external_id)
+        query = query.filter(cls.provider_name == provider_name)
+        if local_user_id:
+            query = query.filter(cls.local_user_id == local_user_id)
+        return query.first()
+
+    @classmethod
+    def user_by_external_id_and_provider(cls, external_id, provider_name):
+        """
+        Returns User instance based on search params
+
+        :param external_id:
+        :param provider_name:
+        :return: User
+        """
+        query = User.query()
+        query = query.filter(cls.external_id == external_id)
+        query = query.filter(cls.provider_name == provider_name)
+        query = query.filter(User.user_id == cls.local_user_id)
+        return query.first()
+
+    @classmethod
+    def by_local_user_id(cls, local_user_id):
+        """
+        Returns all tokens for user
+
+        :param local_user_id:
+        :return: ExternalIdentity
+        """
+        query = cls.query()
+        query = query.filter(cls.local_user_id == local_user_id)
+        return query
+
+
+class Integration(Base, BaseModel):
+    __tablename__ = 'integrations'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True}
+    )
+
+    integration_id = Column('integration_id', Integer(), primary_key=True)
+    integration_type = Column('integration_type', String(255))
+    enabled = Column('enabled', Boolean(), nullable=False)
+    name = Column('name', String(255), nullable=False)
+    child_repos_only = Column('child_repos_only', Boolean(), nullable=False,
+        default=False)
+
+    settings = Column(
+        'settings_json', MutationObj.as_mutable(
+            JsonType(dialect_map=dict(mysql=UnicodeText(16384)))))
+    repo_id = Column(
+        'repo_id', Integer(), ForeignKey('repositories.repo_id'),
+        nullable=True, unique=None, default=None)
+    repo = relationship('Repository', lazy='joined')
+
+    repo_group_id = Column(
+        'repo_group_id', Integer(), ForeignKey('groups.group_id'),
+        nullable=True, unique=None, default=None)
+    repo_group = relationship('RepoGroup', lazy='joined')
+
+    @property
+    def scope(self):
+        if self.repo:
+            return repr(self.repo)
+        if self.repo_group:
+            if self.child_repos_only:
+                return repr(self.repo_group) + ' (child repos only)'
+            else:
+                return repr(self.repo_group) + ' (recursive)'
+        if self.child_repos_only:
+            return 'root_repos'
+        return 'global'
+
+    def __repr__(self):
+        return '<Integration(%r, %r)>' % (self.integration_type, self.scope)
+
+
+class RepoReviewRuleUser(Base, BaseModel):
+    __tablename__ = 'repo_review_rules_users'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True,}
+    )
+    repo_review_rule_user_id = Column(
+        'repo_review_rule_user_id', Integer(), primary_key=True)
+    repo_review_rule_id = Column("repo_review_rule_id",
+        Integer(), ForeignKey('repo_review_rules.repo_review_rule_id'))
+    user_id = Column("user_id", Integer(), ForeignKey('users.user_id'),
+        nullable=False)
+    user = relationship('User')
+
+
+class RepoReviewRuleUserGroup(Base, BaseModel):
+    __tablename__ = 'repo_review_rules_users_groups'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True,}
+    )
+    repo_review_rule_users_group_id = Column(
+        'repo_review_rule_users_group_id', Integer(), primary_key=True)
+    repo_review_rule_id = Column("repo_review_rule_id",
+        Integer(), ForeignKey('repo_review_rules.repo_review_rule_id'))
+    users_group_id = Column("users_group_id", Integer(),
+        ForeignKey('users_groups.users_group_id'), nullable=False)
+    users_group = relationship('UserGroup')
+
+
+class RepoReviewRule(Base, BaseModel):
+    __tablename__ = 'repo_review_rules'
+    __table_args__ = (
+        {'extend_existing': True, 'mysql_engine': 'InnoDB',
+         'mysql_charset': 'utf8', 'sqlite_autoincrement': True,}
+    )
+
+    repo_review_rule_id = Column(
+        'repo_review_rule_id', Integer(), primary_key=True)
+    repo_id = Column(
+        "repo_id", Integer(), ForeignKey('repositories.repo_id'))
+    repo = relationship('Repository', backref='review_rules')
+
+    _branch_pattern = Column("branch_pattern", UnicodeText().with_variant(UnicodeText(255), 'mysql'),
+        default=u'*') # glob
+    _file_pattern = Column("file_pattern", UnicodeText().with_variant(UnicodeText(255), 'mysql'),
+        default=u'*') # glob
+
+    use_authors_for_review = Column("use_authors_for_review", Boolean(),
+        nullable=False, default=False)
+    rule_users = relationship('RepoReviewRuleUser')
+    rule_user_groups = relationship('RepoReviewRuleUserGroup')
+
+    @hybrid_property
+    def branch_pattern(self):
+        return self._branch_pattern or '*'
+
+    def _validate_pattern(self, value):
+        re.compile('^' + glob2re(value) + '$')
+
+    @branch_pattern.setter
+    def branch_pattern(self, value):
+        self._validate_glob(value)
+        self._branch_pattern = value or '*'
+
+    @hybrid_property
+    def file_pattern(self):
+        return self._file_pattern or '*'
+
+    @file_pattern.setter
+    def file_pattern(self, value):
+        self._validate_glob(value)
+        self._file_pattern = value or '*'
+
+    def matches(self, branch, files_changed):
+        """
+        Check if this review rule matches a branch/files in a pull request
+
+        :param branch: branch name for the commit
+        :param files_changed: list of file paths changed in the pull request
+        """
+
+        branch = branch or ''
+        files_changed = files_changed or []
+
+        branch_matches = True
+        if branch:
+            branch_regex = re.compile('^' + glob2re(self.branch_pattern) + '$')
+            branch_matches = bool(branch_regex.search(branch))
+
+        files_matches = True
+        if self.file_pattern != '*':
+            files_matches = False
+            file_regex = re.compile(glob2re(self.file_pattern))
+            for filename in files_changed:
+                if file_regex.search(filename):
+                    files_matches = True
+                    break
+
+        return branch_matches and files_matches
+
+    @property
+    def review_users(self):
+        """ Returns the users which this rule applies to """
+
+        users = set()
+        users |= set([
+            rule_user.user for rule_user in self.rule_users
+            if rule_user.user.active])
+        users |= set(
+            member.user
+            for rule_user_group in self.rule_user_groups
+            for member in rule_user_group.users_group.members
+            if member.user.active
+        )
+        return users
+
+    def __repr__(self):
+        return '<RepoReviewerRule(id=%r, repo=%r)>' % (
+            self.repo_review_rule_id, self.repo)

.bumpversion.cfg

@@ -1,5 +1,5 b''
 [bumpversion]
-current_version = 4.4.2
+current_version = 4.5.0
 message = release: Bump version {current_version} to {new_version}
 
 [bumpversion:file:rhodecode/VERSION]

.hgignore

@@ -43,6 +43,7 b' syntax: regexp'
 ^rhodecode/public/css/style-polymer.css$
 ^rhodecode/public/js/rhodecode-components.html$
 ^rhodecode/public/js/scripts.js$
+^rhodecode/public/js/rhodecode-components.js$
 ^rhodecode/public/js/src/components/root-styles.gen.html$
 ^rhodecode/public/js/vendors/webcomponentsjs/
 ^rhodecode\.db$

.release.cfg

@@ -4,26 +4,21 b' done = false'
 [task:bump_version]
 done = true
 
-[task:rc_tools_pinned]
-done = true
-
 [task:fixes_on_stable]
-done = true
 
 [task:pip2nix_generated]
-done = true
 
 [task:changelog_updated]
-done = true
 
 [task:generate_api_docs]
-done = true
+
+[task:updated_translation]
 
 [release]
-state = prepared
-version = 4.4.2
+state = in_progress
+version = 4.5.0
 
-[task:updated_translation]
+[task:rc_tools_pinned]
 
 [task:generate_js_routes]
 

Gruntfile.js

@@ -11,5 +11,5 b' module.exports = function(grunt) {'
   grunt.loadNpmTasks('grunt-crisper');
   grunt.loadNpmTasks('grunt-contrib-copy');
 
-  grunt.registerTask('default', ['less:production', 'less:components', 'concat:polymercss', 'copy','vulcanize', 'crisper', 'concat:dist']);
+  grunt.registerTask('default', ['less:production', 'less:components', 'concat:polymercss', 'copy', 'concat:dist', 'vulcanize', 'crisper']);
 };

LICENSE.txt

@@ -12,6 +12,10 b' permission notice:'
        file:licenses/msgpack_license.txt
    Copyright (c) 2009 - tornado
        file:licenses/tornado_license.txt
+   Copyright (c) 2015 - pygments-markdown-lexer
+       file:licenses/pygments_markdown_lexer_license.txt
+   Copyright 2006 - diff_match_patch
+       file:licenses/diff_match_patch_license.txt
 
 All licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

Makefile

@@ -2,7 +2,6 b''
 WEBPACK=./node_modules/webpack/bin/webpack.js
 GRUNT=grunt
 NODE_PATH=./node_modules
-FLAKE8=flake8 setup.py pytest_pylons/ rhodecode/ --select=E124 --ignore=E711,E712,E510,E121,E122,E126,E127,E128,E501,F401 --max-line-length=100 --exclude=*rhodecode/lib/dbmigrate/*,*rhodecode/tests/*,*rhodecode/lib/vcs/utils/*
 CI_PREFIX=enterprise
 
 .PHONY: docs docs-clean ci-docs clean test test-clean test-lint test-only
@@ -25,13 +24,6 b' test: test-clean test-only'
 test-clean:
 	rm -rf coverage.xml htmlcov junit.xml pylint.log result
 
-test-lint:
-	if [ "$$IN_NIX_SHELL" = "1" ]; then \
-		$(FLAKE8); \
-	else \
-		$(FLAKE8) --format=pylint --exit-zero > pylint.log; \
-	fi
-
 test-only:
 	PYTHONHASHSEED=random py.test -vv -r xw --cov=rhodecode --cov-report=term-missing --cov-report=html rhodecode/tests/
 

bower.json

@@ -10,6 +10,8 b''
     "paper-tooltip": "PolymerElements/paper-tooltip#^1.1.2",
     "paper-toast": "PolymerElements/paper-toast#^1.3.0",
     "paper-toggle-button": "PolymerElements/paper-toggle-button#^1.2.0",
-    "iron-ajax": "PolymerElements/iron-ajax#^1.4.3"
+    "iron-ajax": "PolymerElements/iron-ajax#^1.4.3",
+    "iron-autogrow-textarea": "PolymerElements/iron-autogrow-textarea#^1.0.13",
+    "iron-a11y-keys": "PolymerElements/iron-a11y-keys#^1.0.6"
   }
 }

configs/development.ini

@@ -1,7 +1,7 b''
 
 
 ################################################################################
-##                    RHODECODE ENTERPRISE CONFIGURATION                      ##
+##                 RHODECODE COMMUNITY EDITION CONFIGURATION                  ##
 # The %(here)s variable will be replaced with the parent directory of this file#
 ################################################################################
 
@@ -64,7 +64,7 b' asyncore_use_poll = true'
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
-## run with gunicorn --log-config <inifile.ini> --paste <inifile.ini>
+## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 #use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
@@ -91,11 +91,13 b' asyncore_use_poll = true'
 #timeout = 21600
 
 
-## prefix middleware for RhodeCode, disables force_https flag.
+## prefix middleware for RhodeCode.
 ## recommended when using proxy setup.
 ## allows to set RhodeCode under a prefix in server.
-## eg https://server.com/<prefix>. Enable `filter-with =` option below as well.
-## optionally set prefix like: `prefix = /<your-prefix>`
+## eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
+## And set your prefix like: `prefix = /custom_prefix`
+## be sure to also set beaker.session.cookie_path = /custom_prefix if you need
+## to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
@@ -194,17 +196,17 b' rss_items_per_page = 10'
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
-## url that does rewrites to _admin/gists/<gistid>.
+## url that does rewrites to _admin/gists/{gistid}.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
-## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/<gistid>
+## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ## List of controllers (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
-## Adding ?auth_token = <token> to the url authenticates this request as if it
+## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ##
-## Syntax is <ControllerClass>:<function_pattern>.
+## Syntax is ControllerClass:function_pattern.
 ## To enable access to raw_files put `FilesController:raw`.
 ## To enable access to patches add `ChangesetController:changeset_patch`.
 ## The list should be "," separated and on a single line.
@@ -377,15 +379,15 b' beaker.session.lock_dir = %(here)s/data/'
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
-#beaker.session.encrypt_key = <key_for_encryption>
-#beaker.session.validate_key = <validation_key>
+#beaker.session.encrypt_key = key_for_encryption
+#beaker.session.validate_key = validation_key
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
-## Path to use for the cookie.
-#beaker.session.cookie_path = /<your-prefix>
+## Path to use for the cookie. Set to prefix if you use prefix middleware
+#beaker.session.cookie_path = /custom_prefix
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
@@ -403,8 +405,8 b' beaker.session.auto = false'
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
-# WHOOSH Backend, doesn't require additional services to run
-# it works good with few dozen repos
+## WHOOSH Backend, doesn't require additional services to run
+## it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
@@ -511,7 +513,7 b' sqlalchemy.db1.url = sqlite:///%(here)s/'
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
-## recycle the connections after this ammount of seconds
+## recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
@@ -533,19 +535,19 b' vcs.server = localhost:9900'
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
-## `pyro4` - using pyro4 server
-## `http` - using http-rpc backend
+## `pyro4` - use pyro4 server
+## `http` - use http-rpc backend (default)
 vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
-## `pyro4` - using pyro4 server
-## `rhodecode.lib.middleware.utils.scm_app_http` - Http based, recommended
-## `vcsserver.scm_app` - internal app (EE only)
-vcs.scm_app_implementation = rhodecode.lib.middleware.utils.scm_app_http
+## `pyro4` - use pyro4 server
+## `http` - use http-rpc backend (default)
+##
+vcs.scm_app_implementation = http
 
 ## Push/Pull operations hooks protocol, available options are:
-## `pyro4` - using pyro4 server
-## `http` - using http-rpc backend
+## `pyro4` - use pyro4 server
+## `http` - use http-rpc backend (default)
 vcs.hooks.protocol = http
 
 vcs.server.log_level = debug
@@ -574,12 +576,15 b' svn.proxy.generate_config = false'
 svn.proxy.list_parent_path = true
 ## Set location and file name of generated config file.
 svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
-## File system path to the directory containing the repositories served by
-## RhodeCode.
-svn.proxy.parent_path_root = /path/to/repo_store
-## Used as a prefix to the <Location> block in the generated config file. In
-## most cases it should be set to `/`.
+## Used as a prefix to the `Location` block in the generated config file.
+## In most cases it should be set to `/`.
 svn.proxy.location_root = /
+## Command to reload the mod dav svn configuration on change.
+## Example: `/etc/init.d/apache2 reload`
+#svn.proxy.reload_cmd = /etc/init.d/apache2 reload
+## If the timeout expires before the reload command finishes, the command will
+## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
+#svn.proxy.reload_timeout = 10
 
 
 ################################

configs/production.ini

@@ -1,7 +1,7 b''
 
 
 ################################################################################
-##                    RHODECODE ENTERPRISE CONFIGURATION                      ##
+##                 RHODECODE COMMUNITY EDITION CONFIGURATION                  ##
 # The %(here)s variable will be replaced with the parent directory of this file#
 ################################################################################
 
@@ -64,7 +64,7 b' port = 5000'
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
-## run with gunicorn --log-config <inifile.ini> --paste <inifile.ini>
+## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
@@ -91,11 +91,13 b' max_requests_jitter = 30'
 timeout = 21600
 
 
-## prefix middleware for RhodeCode, disables force_https flag.
+## prefix middleware for RhodeCode.
 ## recommended when using proxy setup.
 ## allows to set RhodeCode under a prefix in server.
-## eg https://server.com/<prefix>. Enable `filter-with =` option below as well.
-## optionally set prefix like: `prefix = /<your-prefix>`
+## eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
+## And set your prefix like: `prefix = /custom_prefix`
+## be sure to also set beaker.session.cookie_path = /custom_prefix if you need
+## to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
@@ -168,17 +170,17 b' rss_items_per_page = 10'
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
-## url that does rewrites to _admin/gists/<gistid>.
+## url that does rewrites to _admin/gists/{gistid}.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
-## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/<gistid>
+## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ## List of controllers (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
-## Adding ?auth_token = <token> to the url authenticates this request as if it
+## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ##
-## Syntax is <ControllerClass>:<function_pattern>.
+## Syntax is ControllerClass:function_pattern.
 ## To enable access to raw_files put `FilesController:raw`.
 ## To enable access to patches add `ChangesetController:changeset_patch`.
 ## The list should be "," separated and on a single line.
@@ -351,15 +353,15 b' beaker.session.lock_dir = %(here)s/data/'
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
-#beaker.session.encrypt_key = <key_for_encryption>
-#beaker.session.validate_key = <validation_key>
+#beaker.session.encrypt_key = key_for_encryption
+#beaker.session.validate_key = validation_key
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
-## Path to use for the cookie.
-#beaker.session.cookie_path = /<your-prefix>
+## Path to use for the cookie. Set to prefix if you use prefix middleware
+#beaker.session.cookie_path = /custom_prefix
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
@@ -377,8 +379,8 b' beaker.session.auto = false'
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
-# WHOOSH Backend, doesn't require additional services to run
-# it works good with few dozen repos
+## WHOOSH Backend, doesn't require additional services to run
+## it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
@@ -480,7 +482,7 b' sqlalchemy.db1.url = postgresql://postgr'
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
-## recycle the connections after this ammount of seconds
+## recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
@@ -502,20 +504,20 b' vcs.server = localhost:9900'
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
-## `pyro4` - using pyro4 server
-## `http` - using http-rpc backend
-#vcs.server.protocol = http
+## `pyro4` - use pyro4 server
+## `http` - use http-rpc backend (default)
+vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
-## `pyro4` - using pyro4 server
-## `rhodecode.lib.middleware.utils.scm_app_http` - Http based, recommended
-## `vcsserver.scm_app` - internal app (EE only)
-#vcs.scm_app_implementation = rhodecode.lib.middleware.utils.scm_app_http
+## `pyro4` - use pyro4 server
+## `http` - use http-rpc backend (default)
+##
+vcs.scm_app_implementation = http
 
 ## Push/Pull operations hooks protocol, available options are:
-## `pyro4` - using pyro4 server
-## `http` - using http-rpc backend
-#vcs.hooks.protocol = http
+## `pyro4` - use pyro4 server
+## `http` - use http-rpc backend (default)
+vcs.hooks.protocol = http
 
 vcs.server.log_level = info
 ## Start VCSServer with this instance as a subprocess, usefull for development
@@ -543,12 +545,15 b' svn.proxy.generate_config = false'
 svn.proxy.list_parent_path = true
 ## Set location and file name of generated config file.
 svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
-## File system path to the directory containing the repositories served by
-## RhodeCode.
-svn.proxy.parent_path_root = /path/to/repo_store
-## Used as a prefix to the <Location> block in the generated config file. In
-## most cases it should be set to `/`.
+## Used as a prefix to the `Location` block in the generated config file.
+## In most cases it should be set to `/`.
 svn.proxy.location_root = /
+## Command to reload the mod dav svn configuration on change.
+## Example: `/etc/init.d/apache2 reload`
+#svn.proxy.reload_cmd = /etc/init.d/apache2 reload
+## If the timeout expires before the reload command finishes, the command will
+## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
+#svn.proxy.reload_timeout = 10
 
 
 ################################

default.nix

@@ -4,27 +4,55 b''
 # derivation. For advanced tweaks to pimp up the development environment we use
 # "shell.nix" so that it does not have to clutter this file.
 
-{ pkgs ? (import <nixpkgs> {})
-, pythonPackages ? "python27Packages"
+args@
+{ pythonPackages ? "python27Packages"
 , pythonExternalOverrides ? self: super: {}
 , doCheck ? true
+, ...
 }:
 
-let pkgs_ = pkgs; in
-
 let
-  pkgs = pkgs_.overridePackages (self: super: {
-    # Override subversion derivation to
-    #  - activate python bindings
-    #  - set version to 1.8
-    subversion = super.subversion18.override {
-       httpSupport = true;
-       pythonBindings = true;
-       python = self.python27Packages.python;
-    };
+
+  # Use nixpkgs from args or import them. We use this indirect approach
+  # through args to be able to use the name `pkgs` for our customized packages.
+  # Otherwise we will end up with an infinite recursion.
+  nixpkgs = args.pkgs or (import <nixpkgs> { });
+
+  # johbo: Interim bridge which allows us to build with the upcoming
+  # nixos.16.09 branch (unstable at the moment of writing this note) and the
+  # current stable nixos-16.03.
+  backwardsCompatibleFetchgit = { ... }@args:
+    let
+      origSources = nixpkgs.fetchgit args;
+    in
+    nixpkgs.lib.overrideDerivation origSources (oldAttrs: {
+      NIX_PREFETCH_GIT_CHECKOUT_HOOK = ''
+        find $out -name '.git*' -print0 | xargs -0 rm -rf
+      '';
+    });
+
+  # Create a customized version of nixpkgs which should be used throughout the
+  # rest of this file.
+  pkgs = nixpkgs.overridePackages (self: super: {
+    fetchgit = backwardsCompatibleFetchgit;
   });
 
-  inherit (pkgs.lib) fix extends;
+  # Evaluates to the last segment of a file system path.
+  basename = path: with pkgs.lib; last (splitString "/" path);
+
+  # source code filter used as arugment to builtins.filterSource.
+  src-filter = path: type: with pkgs.lib;
+    let
+      ext = last (splitString "." path);
+    in
+      !builtins.elem (basename path) [
+        ".git" ".hg" "__pycache__" ".eggs"
+        "bower_components" "node_modules"
+        "build" "data" "result" "tmp"] &&
+      !builtins.elem ext ["egg-info" "pyc"] &&
+      # TODO: johbo: This check is wrong, since "path" contains an absolute path,
+      # it would still be good to restore it since we want to ignore "result-*".
+      !hasPrefix "result" path;
 
   basePythonPackages = with builtins; if isAttrs pythonPackages
     then pythonPackages
@@ -34,25 +62,6 b' let'
     pkgs.buildBowerComponents or
     (import ./pkgs/backport-16.03-build-bower-components.nix { inherit pkgs; });
 
-  elem = builtins.elem;
-  basename = path: with pkgs.lib; last (splitString "/" path);
-  startsWith = prefix: full: let
-    actualPrefix = builtins.substring 0 (builtins.stringLength prefix) full;
-  in actualPrefix == prefix;
-
-  src-filter = path: type: with pkgs.lib;
-    let
-      ext = last (splitString "." path);
-    in
-      !elem (basename path) [
-        ".git" ".hg" "__pycache__" ".eggs"
-        "bower_components" "node_modules"
-        "build" "data" "result" "tmp"] &&
-      !elem ext ["egg-info" "pyc"] &&
-      # TODO: johbo: This check is wrong, since "path" contains an absolute path,
-      # it would still be good to restore it since we want to ignore "result-*".
-      !startsWith "result" path;
-
   sources = pkgs.config.rc.sources or {};
   version = builtins.readFile ./rhodecode/VERSION;
   rhodecode-enterprise-ce-src = builtins.filterSource src-filter ./.;
@@ -147,18 +156,6 b' let'
         then "${pkgs.glibcLocales}/lib/locale/locale-archive"
         else "";
 
-      # Somewhat snappier setup of the development environment
-      # TODO: move into shell.nix
-      # TODO: think of supporting a stable path again, so that multiple shells
-      #       can share it.
-      shellHook = ''
-        tmp_path=$(mktemp -d)
-        export PATH="$tmp_path/bin:$PATH"
-        export PYTHONPATH="$tmp_path/${self.python.sitePackages}:$PYTHONPATH"
-        mkdir -p $tmp_path/${self.python.sitePackages}
-        python setup.py develop --prefix $tmp_path --allow-hosts ""
-      '' + linkNodeAndBowerPackages;
-
       preCheck = ''
         export PATH="$out/bin:$PATH"
       '';
@@ -226,16 +223,16 b' let'
   rhodecode-testdata-src = sources.rhodecode-testdata or (
     pkgs.fetchhg {
       url = "https://code.rhodecode.com/upstream/rc_testdata";
-      rev = "v0.8.0";
-      sha256 = "0hy1ba134rq2f9si85yx7j4qhc9ky0hjzdk553s3q026i7km809m";
+      rev = "v0.9.0";
+      sha256 = "0k0ccb7cncd6mmzwckfbr6l7fsymcympwcm948qc3i0f0m6bbg1y";
   });
 
   # Apply all overrides and fix the final package set
-  myPythonPackagesUnfix =
+  myPythonPackagesUnfix = with pkgs.lib;
     (extends pythonExternalOverrides
     (extends pythonLocalOverrides
     (extends pythonOverrides
              pythonGeneratedPackages)));
-  myPythonPackages = (fix myPythonPackagesUnfix);
+  myPythonPackages = (pkgs.lib.fix myPythonPackagesUnfix);
 
 in myPythonPackages.rhodecode-enterprise-ce

docs/admin/admin-tricks.rst

@@ -62,6 +62,24 b' 3. Select :guilabel:`Save`, and you will'
 
 .. _md-rst:
 
+
+Suppress license warnings or errors
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In case you're running on maximum allowed users, RhodeCode will display a
+warning message on pages that you're close to the license limits.
+It's often not desired to show that all the time. Here's how you can suppress
+the license messages.
+
+1. From the |RCE| interface, select
+   :menuselection:`Admin --> Settings --> Global`
+2. Select :guilabel:`Flash message filtering` from the drop-down menu.
+3. Select :guilabel:`Save`, and you will no longer see the license message
+   once your page refreshes.
+
+.. _admin-tricks-suppress-license-messages:
+
+
 Markdown or RST Rendering
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 

docs/admin/apache-subdirectory.rst

@@ -7,10 +7,11 b' Use the following example to configure A'
 
 .. code-block:: apache
 
-    <Location /<someprefix> > # Change <someprefix> into your chosen prefix
-      ProxyPass http://127.0.0.1:5000/<someprefix>
-      ProxyPassReverse http://127.0.0.1:5000/<someprefix>
-      SetEnvIf X-Url-Scheme https HTTPS=1
+    <Location /<someprefix>/ # Change <someprefix> into your chosen prefix
+      ProxyPreserveHost On
+      ProxyPass "http://127.0.0.1:5000/"
+      ProxyPassReverse "http://127.0.0.1:5000/"
+      Header set X-Url-Scheme https env=HTTPS
     </Location>
 
 In addition to the regular Apache setup you will need to add the following

docs/admin/backup-restore.rst

@@ -9,9 +9,14 b' Backup and Restore'
 To snapshot an instance of |RCE|, and save its settings, you need to backup the
 following parts of the system at the same time.
 
-* The |repos| managed by the instance.
+* The |repos| managed by the instance together with the stored Gists.
 * The |RCE| database.
-* Any configuration files or extensions that you've configured.
+* Any configuration files or extensions that you've configured. In most
+  cases it's only the :file:`rhodecode.ini` file.
+* Installer files such as those in `/opt/rhodecode` can be backed-up, however
+  it's not required since in case of a recovery installer simply
+  re-creates those.
+
 
 .. important::
 
@@ -29,11 +34,17 b' Repository Backup'
 ^^^^^^^^^^^^^^^^^
 
 To back up your |repos|, use the API to get a list of all |repos| managed,
-and then clone them to your backup location.
+and then clone them to your backup location. This is the most safe backup option.
+Backing up the storage directory could potentially result in a backup of
+partially committed files or commits. (Backup taking place during a big push)
+As an alternative you could use a rsync or simple `cp` commands if you can
+ensure your instance is only in read-only mode or stopped at the moment.
+
 
 Use the ``get_repos`` method to list all your managed |repos|,
 and use the ``clone_uri`` information that is returned. See the :ref:`api`
-for more information.
+for more information. Be sure to keep the structure or repositories with their
+repository groups.
 
 .. important::
 
@@ -54,13 +65,19 b' backup location:'
 .. code-block:: bash
 
    # For MySQL DBs
-   $ mysqldump -u <uname> -p <pass> db_name > mysql-db-backup
+   $ mysqldump -u <uname> -p <pass> rhodecode_db_name > mysql-db-backup
+   # MySQL restore command
+   $ mysql -u <uname> -p <pass> rhodecode_db_name < mysql-db-backup
 
    # For PostgreSQL DBs
-   $ pg_dump dbname > postgresql-db-backup
+   $ PGPASSWORD=<pass> pg_dump rhodecode_db_name > postgresql-db-backup
+   # PosgreSQL restore
+   $ PGPASSWORD=<pass> psql -U <uname> -h localhost -d rhodecode_db_name -1 -f postgresql-db-backup
 
-   # For SQLlite
+   # For SQLite
    $ sqlite3 rhodecode.db ‘.dump’ > sqlite-db-backup
+   # SQLite restore
+   $ copy sqlite-db-backup rhodecode.db
 
 
 The default |RCE| SQLite database location is
@@ -75,13 +92,18 b' Configuration File Backup'
 Depending on your setup, you could have a number of configuration files that
 should be backed up. You may have some, or all of the configuration files
 listed in the :ref:`config-rce-files` section. Ideally you should back these
-up at the same time as the database and |repos|.
+up at the same time as the database and |repos|. It really depends on if you need
+the configuration file like logs, custom modules. We always recommend backing
+those up.
 
 Gist Backup
 ^^^^^^^^^^^
 
-To backup the gists on your |RCE| instance you can use the ``get_users`` and
-``get_gists`` API methods to fetch the gists for each user on the instance.
+To backup the gists on your |RCE| instance you usually have to backup the
+gist storage path. If this haven't been changed it's located inside
+:file:`.rc_gist_store` and the metadata in :file:`.rc_gist_metadata`.
+You can use the ``get_users`` and ``get_gists`` API methods to fetch the
+gists for each user on the instance.
 
 Extension Backups
 ^^^^^^^^^^^^^^^^^
@@ -100,15 +122,17 b' the :ref:`indexing-ref` section.'
 Restoration Steps
 -----------------
 
-To restore an instance of |RCE| from its backed up components, use the
-following steps.
+To restore an instance of |RCE| from its backed up components, to a fresh
+system use the following steps.
 
-1. Install a new instance of |RCE|.
-2. Once installed, configure the instance to use the backed up
-   :file:`rhodecode.ini` file. Ensure this file points to the backed up
+1. Install a new instance of |RCE| using sqlite option as database.
+2. Restore your database.
+3. Once installed, replace you backed up the :file:`rhodecode.ini` with your
+   backup version. Ensure this file points to the restored
    database, see the :ref:`config-database` section.
-3. Restart |RCE| and remap and rescan your |repos|, see the
-   :ref:`remap-rescan` section.
+4. Restart |RCE| and remap and rescan your |repos| to verify filesystem access,
+   see the :ref:`remap-rescan` section.
+
 
 Post Restoration Steps
 ^^^^^^^^^^^^^^^^^^^^^^

docs/admin/reset-information.rst

@@ -22,8 +22,8 b' account permissions.'
 .. code-block:: bash
 
     # Open iShell from the terminal
-    $ .rccontrol/enterprise-5/profile/bin/paster \
-        ishell .rccontrol/enterprise-5/rhodecode.ini
+    $ .rccontrol/enterprise-1/profile/bin/paster \
+        ishell .rccontrol/enterprise-1/rhodecode.ini
 
 .. code-block:: mysql
 
@@ -52,11 +52,12 b' following example to make changes to thi'
 .. code-block:: mysql
 
   # Use this example to enable global .hgrc access
-  In [4]: new_option = RhodeCodeUi()
-  In [5]: new_option.ui_section='web'
-  In [6]: new_option.ui_key='allow_push'
-  In [7]: new_option.ui_value='*'
-  In [8]: Session().add(new_option);Session().commit()
+  In [1]: new_option = RhodeCodeUi()
+  In [2]: new_option.ui_section='web'
+  In [3]: new_option.ui_key='allow_push'
+  In [4]: new_option.ui_value='*'
+  In [5]: Session().add(new_option);Session().commit()
+  In [6]: exit()
 
 Manually Reset Password
 ^^^^^^^^^^^^^^^^^^^^^^^
@@ -72,24 +73,47 b' Use the following code example to carry '
 .. code-block:: bash
 
     # starts the ishell interactive prompt
-    $ .rccontrol/enterprise-5/profile/bin/paster \
-        ishell .rccontrol/enterprise-5/rhodecode.ini
+    $ .rccontrol/enterprise-1/profile/bin/paster \
+        ishell .rccontrol/enterprise-1/rhodecode.ini
 
 .. code-block:: mysql
 
-    from rhodecode.lib.auth import generate_auth_token
-    from rhodecode.lib.auth import get_crypt_password
-
+    In [1]: from rhodecode.lib.auth import generate_auth_token
+    In [2]: from rhodecode.lib.auth import get_crypt_password
     # Enter the user name whose password you wish to change
-    my_user = 'USERNAME'
-    u = User.get_by_username(my_user)
-
+    In [3]: my_user = 'USERNAME'
+    In [4]: u = User.get_by_username(my_user)
     # If this fails then the user does not exist
-    u.auth_token = generate_auth_token(my_user)
-
+    In [5]: u.auth_token = generate_auth_token(my_user)
     # Set the new password
-    u.password = get_crypt_password('PASSWORD')
+    In [6]: u.password = get_crypt_password('PASSWORD')
+    In [7]: Session().add(u);Session().commit()
+    In [8]: exit()
+
+
+
+Change user details
+^^^^^^^^^^^^^^^^^^^
+
+If you need to manually change some of users details, use the following steps.
+
+1. Navigate to your |RCE| install location.
+2. Run the interactive ``ishell`` prompt.
+3. Set a new arguments for users.
 
-    Session().add(u)
-    Session().commit()
-    exit
+Use the following code example to carry out these steps.
+
+.. code-block:: bash
+
+    # starts the ishell interactive prompt
+    $ .rccontrol/enterprise-1/profile/bin/paster \
+        ishell .rccontrol/enterprise-1/rhodecode.ini
+
+.. code-block:: mysql
+
+    # Use this example to change email and username of LDAP user
+    In [1]: my_user = User.get_by_username('some_username')
+    In [2]: my_user.email = '[e-mail]'
+    In [3]: my_user.username = 'SomeUser'
+    In [4]: Session().add(my_user);Session().commit()
+    In [5]: exit()

docs/admin/sec-your-server.rst

@@ -36,7 +36,7 b' 1. On your local machine create the publ'
     Your public key has been saved in /home/user/.ssh/id_rsa.pub.
     The key fingerprint is:
     02:82:38:95:e5:30:d2:ad:17:60:15:7f:94:17:9f:30 user@ubuntu
-    The key's randomart image is:
+    The key\'s randomart image is:
     +--[ RSA 2048]----+
 
 2. SFTP to your server, and copy the public key to the ``~/.ssh`` folder.

docs/admin/system-admin.rst

@@ -18,6 +18,7 b' The following are the most common system'
 
     config-files-overview
     vcs-server
+    svn-http
     apache-config
     nginx-config
     backup-restore

docs/admin/tuning-change-large-file-dir.rst

@@ -18,7 +18,7 b' 1. Open ishell from the terminal and use'
 2. Run the following commands, and ensure that |RCE| has write access to the
    new directory:
 
-.. code-block:: mysql
+.. code-block:: bash
 
     # Once logged into the database, use SQL to redirect
     # the large files location

docs/admin/vcs-server.rst

@@ -298,133 +298,7 b' For a more detailed explanation of the l'
     format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
     datefmt = %Y-%m-%d %H:%M:%S
 
-.. _svn-http:
-
-|svn| With Write Over HTTP
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To use |svn| with read/write support over the |svn| HTTP protocol, you have to
-configure the HTTP |svn| backend.
-
-Prerequisites
-=============
-
-- Enable HTTP support inside the admin VCS settings on your |RCE| instance
-- You need to install the following tools on the machine that is running an
-  instance of |RCE|:
-  ``Apache HTTP Server`` and
-  ``mod_dav_svn``.
-
-
-Using Ubuntu Distribution as an example you can run:
-
-.. code-block:: bash
-
-    $ sudo apt-get install apache2 libapache2-mod-svn
-
-Once installed you need to enable ``dav_svn``:
-
-.. code-block:: bash
-
-    $ sudo a2enmod dav_svn
-
-Configuring Apache Setup
-========================
-
-.. tip::
-
-   It is recommended to run Apache on a port other than 80, due to possible
-   conflicts with other HTTP servers like nginx. To do this, set the
-   ``Listen`` parameter in the ``/etc/apache2/ports.conf`` file, for example
-   ``Listen 8090``.
-
-
-.. warning::
-
-   Make sure your Apache instance which runs the mod_dav_svn module is
-   only accessible by RhodeCode. Otherwise everyone is able to browse
-   the repositories or run subversion operations (checkout/commit/etc.).
-
-It is also recommended to run apache as the same user as |RCE|, otherwise
-permission issues could occur. To do this edit the ``/etc/apache2/envvars``
-
-   .. code-block:: apache
-
-      export APACHE_RUN_USER=rhodecode
-      export APACHE_RUN_GROUP=rhodecode
-
-1. To configure Apache, create and edit a virtual hosts file, for example
-   :file:`/etc/apache2/sites-available/default.conf`. Below is an example
-   how to use one with auto-generated config ```mod_dav_svn.conf```
-   from configured |RCE| instance.
-
-.. code-block:: apache
-
-    <VirtualHost *:8080>
-        ServerAdmin rhodecode-admin@localhost
-        DocumentRoot /var/www/html
-        ErrorLog ${'${APACHE_LOG_DIR}'}/error.log
-        CustomLog ${'${APACHE_LOG_DIR}'}/access.log combined
-        Include /home/user/.rccontrol/enterprise-1/mod_dav_svn.conf
-    </VirtualHost>
-
-
-2. Go to the :menuselection:`Admin --> Settings --> VCS` page, and
-   enable :guilabel:`Proxy Subversion HTTP requests`, and specify the
-   :guilabel:`Subversion HTTP Server URL`.
-
-3. Open the |RCE| configuration file,
-   :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
-
-4. Add the following configuration option in the ``[app:main]``
-   section if you don't have it yet.
-
-   This enables mapping of the created |RCE| repo groups into special |svn| paths.
-   Each time a new repository group is created, the system will update
-   the template file and create new mapping. Apache web server needs to be
-   reloaded to pick up the changes on this file.
-   It's recommended to add reload into a crontab so the changes can be picked
-   automatically once someone creates a repository group inside RhodeCode.
-
-
-.. code-block:: ini
-
-    ##############################################
-    ### Subversion proxy support (mod_dav_svn) ###
-    ##############################################
-    ## Enable or disable the config file generation.
-    svn.proxy.generate_config = true
-    ## Generate config file with `SVNListParentPath` set to `On`.
-    svn.proxy.list_parent_path = true
-    ## Set location and file name of generated config file.
-    svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
-    ## File system path to the directory containing the repositories served by
-    ## RhodeCode.
-    svn.proxy.parent_path_root = /path/to/repo_store
-    ## Used as a prefix to the <Location> block in the generated config file. In
-    ## most cases it should be set to `/`.
-    svn.proxy.location_root = /
-
-
-This would create a special template file called ```mod_dav_svn.conf```. We
-used that file path in the apache config above inside the Include statement.
-
-
-Using |svn|
-===========
-
-Once |svn| has been enabled on your instance, you can use it with the
-following examples. For more |svn| information, see the `Subversion Red Book`_
-
-.. code-block:: bash
-
-    # To clone a repository
-    svn checkout http://my-svn-server.example.com/my-svn-repo
-
-    # svn commit
-    svn commit
 
 .. _Subversion Red Book: http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn
 
-
-.. _Ask Ubuntu: http://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue
  No newline at end of file
+.. _Ask Ubuntu: http://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue

docs/api/methods/deprecated-methods.rst

@@ -1,7 +1,7 b''
 .. _deprecated-methods-ref:
 
 deprecated methods
-=================
+==================
 
 changeset_comment 
 -----------------

docs/api/methods/gist-methods.rst

@@ -1,7 +1,7 b''
 .. _gist-methods-ref:
 
 gist methods
-=================
+============
 
 create_gist 
 -----------

docs/api/methods/license-methods.rst

@@ -1,10 +1,10 b''
 .. _license-methods-ref:
 
 license methods
-=================
+===============
 
 get_license_info (EE only)
-----------------
+--------------------------
 
 .. py:function:: get_license_info(apiuser)
 
@@ -32,7 +32,7 b' get_license_info (EE only)'
 
 
 set_license_key (EE only)
----------------
+-------------------------
 
 .. py:function:: set_license_key(apiuser, key)
 

docs/api/methods/pull-request-methods.rst

@@ -1,7 +1,7 b''
 .. _pull-request-methods-ref:
 
 pull_request methods
-=================
+====================
 
 close_pull_request 
 ------------------
@@ -103,6 +103,10 b' create_pull_request'
    :type description: Optional(str)
    :param reviewers: Set the new pull request reviewers list.
    :type reviewers: Optional(list)
+       Accepts username strings or objects of the format:
+       {
+           'username': 'nick', 'reasons': ['original author']
+       }
 
 
 get_pull_request 
@@ -165,6 +169,15 b' get_pull_request'
                                         "commit_id": "<commit_id>",
                                     }
                                 },
+           "merge":             {
+                                    "clone_url":   "<clone_url>",
+                                    "reference":
+                                    {
+                                        "name":      "<name>",
+                                        "type":      "<type>",
+                                        "commit_id": "<commit_id>",
+                                    }
+                                },
           "author":             <user_obj>,
           "reviewers":          [
                                     ...
@@ -241,6 +254,15 b' get_pull_requests'
                                             "commit_id": "<commit_id>",
                                         }
                                     },
+               "merge":             {
+                                        "clone_url":   "<clone_url>",
+                                        "reference":
+                                        {
+                                            "name":      "<name>",
+                                            "type":      "<type>",
+                                            "commit_id": "<commit_id>",
+                                        }
+                                    },
               "author":             <user_obj>,
               "reviewers":          [
                                         ...
@@ -284,7 +306,12 b' merge_pull_request'
            "executed":         "<bool>",
            "failure_reason":   "<int>",
            "merge_commit_id":  "<merge_commit_id>",
-           "possible":         "<bool>"
+           "possible":         "<bool>",
+           "merge_ref":        {
+                                   "commit_id": "<commit_id>",
+                                   "type":      "<type>",
+                                   "name":      "<name>"
+                               }
        },
      "error": null
 

docs/api/methods/repo-group-methods.rst

@@ -1,24 +1,25 b''
 .. _repo-group-methods-ref:
 
 repo_group methods
-=================
+==================
 
 create_repo_group 
 -----------------
 
-.. py:function:: create_repo_group(apiuser, group_name, description=<Optional:''>, owner=<Optional:<OptionalAttr:apiuser>>, copy_permissions=<Optional:False>)
+.. py:function:: create_repo_group(apiuser, group_name, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, copy_permissions=<Optional:False>)
 
    Creates a repository group.
 
-   * If the repository group name contains "/", all the required repository
-     groups will be created.
+   * If the repository group name contains "/", repository group will be
+     created inside a repository group or nested repository groups
 
-     For example "foo/bar/baz" will create |repo| groups "foo" and "bar"
-     (with "foo" as parent). It will also create the "baz" repository
-     with "bar" as |repo| group.
+     For example "foo/bar/group1" will create repository group called "group1"
+     inside group "foo/bar". You have to have permissions to access and
+     write to the last repository group ("bar" in this example)
 
-   This command can only be run using an |authtoken| with admin
-   permissions.
+   This command can only be run using an |authtoken| with at least
+   permissions to create repository groups, or admin permissions to
+   parent repository groups.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
@@ -73,7 +74,7 b' delete_repo_group'
 
      id : <id_given_in_input>
      result : {
-       'msg': 'deleted repo group ID:<repogroupid> <repogroupname>
+       'msg': 'deleted repo group ID:<repogroupid> <repogroupname>'
        'repo_group': null
      }
      error :  null
@@ -325,13 +326,22 b' revoke_user_permission_from_repo_group'
 update_repo_group 
 -----------------
 
-.. py:function:: update_repo_group(apiuser, repogroupid, group_name=<Optional:''>, description=<Optional:''>, owner=<Optional:<OptionalAttr:apiuser>>, parent=<Optional:None>, enable_locking=<Optional:False>)
+.. py:function:: update_repo_group(apiuser, repogroupid, group_name=<Optional:''>, description=<Optional:''>, owner=<Optional:<OptionalAttr:apiuser>>, enable_locking=<Optional:False>)
 
    Updates repository group with the details given.
 
    This command can only be run using an |authtoken| with admin
    permissions.
 
+   * If the group_name name contains "/", repository group will be updated
+     accordingly with a repository group or nested repository groups
+
+     For example repogroupid=group-test group_name="foo/bar/group-test"
+     will update repository group called "group-test" and place it
+     inside group "foo/bar".
+     You have to have permissions to access and write to the last repository
+     group ("bar" in this example)
+
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repogroupid: Set the ID of repository group.
@@ -342,8 +352,6 b' update_repo_group'
    :type description: str
    :param owner: Set the |repo| group owner.
    :type owner: str
-   :param parent: Set the |repo| group parent.
-   :type parent: str or int
    :param enable_locking: Enable |repo| locking. The default is false.
    :type enable_locking: bool
 

docs/api/methods/repo-methods.rst

@@ -1,7 +1,7 b''
 .. _repo-methods-ref:
 
 repo methods
-=================
+============
 
 add_field_to_repo 
 -----------------
@@ -68,15 +68,16 b' create_repo'
 
    Creates a repository.
 
-   * If the repository name contains "/", all the required repository
-     groups will be created.
+   * If the repository name contains "/", repository will be created inside
+     a repository group or nested repository groups
 
-     For example "foo/bar/baz" will create |repo| groups "foo" and "bar"
-     (with "foo" as parent). It will also create the "baz" repository
-     with "bar" as |repo| group.
+     For example "foo/bar/repo1" will create |repo| called "repo1" inside
+     group "foo/bar". You have to have permissions to access and write to
+     the last repository group ("bar" in this example)
 
    This command can only be run using an |authtoken| with at least
-   write permissions to the |repo|.
+   permissions to create repositories, or write permissions to
+   parent repository groups.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
@@ -88,9 +89,9 b' create_repo'
    :type owner: Optional(str)
    :param description: Set the repository description.
    :type description: Optional(str)
-   :param private:
+   :param private: set repository as private
    :type private: bool
-   :param clone_uri:
+   :param clone_uri: set clone_uri
    :type clone_uri: str
    :param landing_rev: <rev_type>:<rev>
    :type landing_rev: str
@@ -125,7 +126,7 b' create_repo'
      id : <id_given_in_input>
      result : null
      error :  {
-        'failed to create repository `<repo_name>`
+        'failed to create repository `<repo_name>`'
      }
 
 
@@ -164,25 +165,29 b' delete_repo'
 fork_repo 
 ---------
 
-.. py:function:: fork_repo(apiuser, repoid, fork_name, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, copy_permissions=<Optional:False>, private=<Optional:False>, landing_rev=<Optional:'rev:tip'>)
+.. py:function:: fork_repo(apiuser, repoid, fork_name, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, copy_permissions=<Optional:False>)
 
    Creates a fork of the specified |repo|.
 
-   * If using |RCE| with Celery this will immediately return a success
-     message, even though the fork will be created asynchronously.
+   * If the fork_name contains "/", fork will be created inside
+     a repository group or nested repository groups
 
-   This command can only be run using an |authtoken| with fork
-   permissions on the |repo|.
+     For example "foo/bar/fork-repo" will create fork called "fork-repo"
+     inside group "foo/bar". You have to have permissions to access and
+     write to the last repository group ("bar" in this example)
+
+   This command can only be run using an |authtoken| with minimum
+   read permissions of the forked repo, create fork permissions for an user.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set repository name or repository ID.
    :type repoid: str or int
-   :param fork_name: Set the fork name.
+   :param fork_name: Set the fork name, including it's repository group membership.
    :type fork_name: str
    :param owner: Set the fork owner.
    :type owner: str
-   :param description: Set the fork descripton.
+   :param description: Set the fork description.
    :type description: str
    :param copy_permissions: Copy permissions from parent |repo|. The
        default is False.
@@ -729,7 +734,7 b' lock'
      id : <id_given_in_input>
      result : null
      error :  {
-       'Error occurred locking repository `<reponame>`
+       'Error occurred locking repository `<reponame>`'
      }
 
 
@@ -923,24 +928,31 b' strip'
 update_repo 
 -----------
 
-.. py:function:: update_repo(apiuser, repoid, name=<Optional:None>, owner=<Optional:<OptionalAttr:apiuser>>, group=<Optional:None>, fork_of=<Optional:None>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, enable_statistics=<Optional:False>, enable_locking=<Optional:False>, enable_downloads=<Optional:False>, fields=<Optional:''>)
+.. py:function:: update_repo(apiuser, repoid, repo_name=<Optional:None>, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, fork_of=<Optional:None>, enable_statistics=<Optional:False>, enable_locking=<Optional:False>, enable_downloads=<Optional:False>, fields=<Optional:''>)
 
    Updates a repository with the given information.
 
    This command can only be run using an |authtoken| with at least
-   write permissions to the |repo|.
+   admin permissions to the |repo|.
+
+   * If the repository name contains "/", repository will be updated
+     accordingly with a repository group or nested repository groups
+
+     For example repoid=repo-test name="foo/bar/repo-test" will update |repo|
+     called "repo-test" and place it inside group "foo/bar".
+     You have to have permissions to access and write to the last repository
+     group ("bar" in this example)
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: repository name or repository ID.
    :type repoid: str or int
-   :param name: Update the |repo| name.
-   :type name: str
+   :param repo_name: Update the |repo| name, including the
+       repository group it's in.
+   :type repo_name: str
    :param owner: Set the |repo| owner.
    :type owner: str
-   :param group: Set the |repo| group the |repo| belongs to.
-   :type group: str
-   :param fork_of: Set the master |repo| name.
+   :param fork_of: Set the |repo| as fork of another |repo|.
    :type fork_of: str
    :param description: Update the |repo| description.
    :type description: str
@@ -948,16 +960,13 b' update_repo'
    :type private: bool
    :param clone_uri: Update the |repo| clone URI.
    :type clone_uri: str
-   :param landing_rev: Set the |repo| landing revision. Default is
-       ``tip``.
+   :param landing_rev: Set the |repo| landing revision. Default is ``rev:tip``.
    :type landing_rev: str
-   :param enable_statistics: Enable statistics on the |repo|,
-       (True | False).
+   :param enable_statistics: Enable statistics on the |repo|, (True | False).
    :type enable_statistics: bool
    :param enable_locking: Enable |repo| locking.
    :type enable_locking: bool
-   :param enable_downloads: Enable downloads from the |repo|,
-       (True | False).
+   :param enable_downloads: Enable downloads from the |repo|, (True | False).
    :type enable_downloads: bool
    :param fields: Add extra fields to the |repo|. Use the following
        example format: ``field_key=field_val,field_key2=fieldval2``.

docs/api/methods/server-methods.rst

@@ -1,7 +1,7 b''
 .. _server-methods-ref:
 
 server methods
-=================
+==============
 
 get_ip 
 ------

docs/api/methods/user-group-methods.rst

@@ -1,7 +1,7 b''
 .. _user-group-methods-ref:
 
 user_group methods
-=================
+==================
 
 add_user_to_user_group 
 ----------------------

docs/api/methods/user-methods.rst

@@ -1,12 +1,12 b''
 .. _user-methods-ref:
 
 user methods
-=================
+============
 
 create_user 
 -----------
 
-.. py:function:: create_user(apiuser, username, email, password=<Optional:''>, firstname=<Optional:''>, lastname=<Optional:''>, active=<Optional:True>, admin=<Optional:False>, extern_name=<Optional:'rhodecode'>, extern_type=<Optional:'rhodecode'>, force_password_change=<Optional:False>)
+.. py:function:: create_user(apiuser, username, email, password=<Optional:''>, firstname=<Optional:''>, lastname=<Optional:''>, active=<Optional:True>, admin=<Optional:False>, extern_name=<Optional:'rhodecode'>, extern_type=<Optional:'rhodecode'>, force_password_change=<Optional:False>, create_personal_repo_group=<Optional:None>)
 
    Creates a new user and returns the new user object.
 
@@ -39,7 +39,8 b' create_user'
    :param force_password_change: Force the new user to change password
        on next login.
    :type force_password_change: Optional(``True`` | ``False``)
-
+   :param create_personal_repo_group: Create personal repo group for this user
+   :type create_personal_repo_group: Optional(``True`` | ``False``)
    Example output:
 
    .. code-block:: bash
@@ -163,6 +164,7 b' get_user'
                "usergroup.read",
                "hg.repogroup.create.false",
                "hg.create.none",
+               "hg.password_reset.enabled",
                "hg.extern_activate.manual",
                "hg.create.write_on_repogroup.false",
                "hg.usergroup.create.false",

docs/contributing/api.rst

@@ -1,7 +1,7 b''
 
-=====
- API
-=====
+===================
+CONTRIBUTING TO API
+===================
 
 
 

docs/contributing/checklist-tickets.rst

@@ -130,7 +130,7 b' is a very small pencil which has to be c'
 ticket.
 
 
-.. figure:: images/redmine-description.png
+.. figure:: ../images/redmine-description.png
    :alt: Example of pencil to change the ticket description
 
    Shows an example of the pencil which lets you change the description.

docs/contributing/testing/spec-by-example.rst

@@ -9,9 +9,6 b''
 .. Avoid duplicating the quickstart instructions by importing the README
    file.
 
-.. include:: ../../../acceptance_tests/README.rst
-
-
 
 Choices of technology and tools
 ===============================

docs/default.nix

@@ -88,10 +88,10 b' let'
   };
 
   Sphinx = buildPythonPackage (rec {
-    name = "Sphinx-1.4.4";
+    name = "Sphinx-1.4.8";
     src = fetchurl {
-      url = "https://pypi.python.org/packages/20/a2/72f44c84f6c4115e3fef58d36d657ec311d80196eab9fd5ec7bcde76143b/${name}.tar.gz";
-      md5 = "64ce2ec08d37ed56313a98232cbe2aee";
+      url = "https://pypi.python.org/packages/1f/f6/e54a7aad73e35232356103771ae76306dadd8546b024c646fbe75135571c/${name}.tar.gz";
+      md5 = "5ec718a4855917e149498bba91b74e67";
     };
     propagatedBuildInputs = [
       docutils

docs/index.rst

@@ -20,8 +20,10 b' and commit files and |repos| while manag'
 * Migration from existing databases.
 * |RCM| SDK.
 * Built-in analytics
+* Built in integrations including: Slack, Jenkins, Webhooks, Jira, Redmine, Hipchat
 * Pluggable authentication system.
-* Support for |LDAP|, Crowd, CAS, PAM.
+* Support for AD, |LDAP|, Crowd, CAS, PAM.
+* Support for external authentication via Oauth Google, Github, Bitbucket, Twitter.
 * Debug modes of operation.
 * Private and public gists.
 * Gists with limited lifetimes and within instance only sharing.

docs/integrations/integrations.rst

@@ -50,3 +50,4 b' See pages specific to each type of integ'
    redmine
    jira
    webhook
+   email

docs/integrations/webhook.rst

@@ -7,6 +7,16 b' The Webhook integration allows you to PO'
 or pull requests to a custom http endpoint as a json dict with details of the
 event.
 
+Starting from 4.5.0 release, webhook integration allows to use variables
+inside the URL. For example in URL `https://server-example.com/${repo_name}`
+${repo_name} will be replaced with the name of repository which events is
+triggered from. Some of the variables like
+`${branch}` will result in webhook be called multiple times when multiple
+branches are pushed.
+
+Some of the variables like `${pull_request_id}` will be replaced only in
+the pull request related events.
+
 To create a webhook integration, select "webhook" in the integration settings
-and use the url and key from your custom webhook. See
-:ref:`creating-integrations` for additional instructions.
  No newline at end of file
+and use the URL and key from your any previous custom webhook created. See
+:ref:`creating-integrations` for additional instructions.

docs/release-notes/release-notes-4.3.0.rst

@@ -7,38 +7,6 b' Release Date'
 - 2016-08-12
 
 
-General
-^^^^^^^
-
-- Subversion: detect requests also based on magic path.
-  This adds subversion 1.9 support for SVN backend.
-- Summary/changelog: unified how data is displayed for those pages.
-    * use consistent order of columns
-    * fix the link to commit status
-    * fix order of displaying comments
-- Live-chat: refactor live chat system for code review based on
-  latest channelstream changes.
-- SVN: Add template to generate the apache mod_dav_svn config for all
-  repository groups. Repository groups can now be automatically mapped to be
-  supported by SVN backend. Set `svn.proxy.generate_config = true` and similar
-  options found inside .ini config.
-- Readme/markup: improved order of generating readme files. Fixes #4050
-    * we now use order based on default system renderer
-    * having multiple readme files will pick correct one as set renderer
-- Api: add a max_file_bytes parameter to get_nodes so that large files
-  can be skipped.
-- Auth-ldap: added flag to set debug mode for LDAP connections.
-- Labs: moved rebase-merge option from labs settings into VCS settings.
-- System: send platform type and version to upgrade endpoint when checking
-  for new versions.
-- Packaging: update rhodecode-tools from 0.8.3 to 0.10.0
-- Packaging: update codemirror from 5.4.0 to 5.11.0
-- Packaging: updated pygments to 2.1.3
-- Packaging: bumped supervisor to 3.3.0
-- Packaging: bumped psycopg2 to 2.6.1
-- Packaging: bumped mercurial to 3.8.4
-
-
 New Features
 ^^^^^^^^^^^^
 
@@ -64,6 +32,38 b' New Features'
   onto comments you submitted while doing a code-review.
 
 
+General
+^^^^^^^
+
+- Subversion: detect requests also based on magic path.
+  This adds subversion 1.9 support for SVN backend.
+- Summary/changelog: unified how data is displayed for those pages.
+    * use consistent order of columns
+    * fix the link to commit status
+    * fix order of displaying comments
+- Live chat: refactor live chat system for code review based on
+  latest channelstream changes.
+- SVN: Add template to generate the apache mod_dav_svn config for all
+  repository groups. Repository groups can now be automatically mapped to be
+  supported by SVN backend. Set `svn.proxy.generate_config = true` and similar
+  options found inside .ini config.
+- Readme/markup: improved order of generating readme files. Fixes #4050
+    * we now use order based on default system renderer
+    * having multiple readme files will pick correct one as set renderer
+- Api: add a max_file_bytes parameter to get_nodes so that large files
+  can be skipped.
+- Auth-ldap: added flag to set debug mode for LDAP connections.
+- Labs: moved rebase-merge option from labs settings into VCS settings.
+- System: send platform type and version to upgrade endpoint when checking
+  for new versions.
+- Packaging: update rhodecode-tools from 0.8.3 to 0.10.0
+- Packaging: update codemirror from 5.4.0 to 5.11.0
+- Packaging: updated pygments to 2.1.3
+- Packaging: bumped supervisor to 3.3.0
+- Packaging: bumped psycopg2 to 2.6.1
+- Packaging: bumped mercurial to 3.8.4
+
+
 Security
 ^^^^^^^^
 
@@ -105,7 +105,7 b' Fixes'
   support to gevent compatible handling.
 - Diff2way: fixed unicode problem on non-ascii files.
 - Full text search: whoosh schema uses now bigger ints, fixes #4035
-- File-browser: optimized cached tree calculation, reduced load times by
+- File browser: optimized cached tree calculation, reduced load times by
   50% on complex file trees.
 - Styling: #4086 fixing bug where long commit messages did not wrap in file view.
 - SVN: Ignore the content length header from response, fixes #4112.

docs/release-notes/release-notes-4.3.1.rst

@@ -6,6 +6,27 b' Release Date'
 
 - 2016-08-23
 
+
+New Features
+^^^^^^^^^^^^
+
+
+
+General
+^^^^^^^
+
+
+
+Security
+^^^^^^^^
+
+
+
+Performance
+^^^^^^^^^^^
+
+
+
 Fixes
 ^^^^^
 

docs/release-notes/release-notes-4.4.0.rst

@@ -7,18 +7,6 b' Release Date'
 - 2016-09-16
 
 
-General
-^^^^^^^
-
-- UI: introduced Polymer webcomponents into core application. RhodeCode will
-  be now shipped together with Polymer framework webcomponents. Most of
-  dynamic UI components that require large amounts of interaction
-  will be done now with Polymer.
-- live-notifications: use rhodecode-toast for live notifications instead of
-  toastr jquery plugin.
-- Svn: moved svn http support out of labs settings. It's tested and stable now.
-
-
 New Features
 ^^^^^^^^^^^^
 
@@ -29,11 +17,11 b' New Features'
   It will allow to configure exactly which projects use which integrations.
 - Integrations: show branches/commits separately when posting push events
   to hipchat/slack, fixes #4192.
-- Pull-requests: summary page now shows update dates for pull request to
+- Pull requests: summary page now shows update dates for pull request to
   easier see which one were receantly updated.
 - UI: hidden inline comments will be shown in side view when browsing the diffs
 - Diffs: added inline comments toggle into pull requests diff view. #2884
-- Live-chat: added summon reviewers functionality. You can now request
+- Live chat: added summon reviewers functionality. You can now request
   presence from online users into a chat for collaborative code-review.
   This requires channelstream to be enabled.
 - UX: added a static 502 page for gateway error. Once configured via
@@ -41,6 +29,18 b' New Features'
   backend servers are offline. Fixes #4202.
 
 
+General
+^^^^^^^
+
+- UI: introduced Polymer webcomponents into core application. RhodeCode will
+  be now shipped together with Polymer framework webcomponents. Most of
+  dynamic UI components that require large amounts of interaction
+  will be done now with Polymer.
+- Live notifications: use rhodecode-toast for live notifications instead of
+  toastr jquery plugin.
+- Svn: moved svn http support out of labs settings. It's tested and stable now.
+
+
 Security
 ^^^^^^^^
 
@@ -67,12 +67,12 b' Fixes'
   match rest of ui, fixes: #4200.
 - UX: show multiple tags/branches in changelog/summary instead of
   truncating them.
-- My-account: fix test notifications for IE10+
+- My account: fix test notifications for IE10+
 - Vcs: change way refs are retrieved for git so same name branch/tags and
   remotes can be supported, fixes #298.
 - Lexers: added small extensions table to extend syntax highlighting for file
   sources. Fixes #4227.
 - Search: fix bug where file path link was wrong when the repository name was
   in the file path, fixes #4228
-- Fixed INT overflow bug
+- Pagination: fixed INT overflow bug.
 - Events: send pushed commits always in the correct in order.

docs/release-notes/release-notes-4.4.1.rst

@@ -7,19 +7,18 b' Release Date'
 - 2016-09-27
 
 
+New Features
+^^^^^^^^^^^^
+
+
 General
 ^^^^^^^
 
-- channelstream: auto-generate the url to channelstream server if it's not
+- Channelstream: auto-generate the url to channelstream server if it's not
   explicitly defined in the config. It allows to use a relative server
   without knowing its address upfront.
 
 
-New Features
-^^^^^^^^^^^^
-
-
-
 Security
 ^^^^^^^^
 
@@ -34,7 +33,7 b' Fixes'
 ^^^^^
 
 - GIT: properly extract branches on events and submit them to integrations.
-- Pullrequests: fix problems with unicode in auto-generated descriptions
+- Pull requests: fix problems with unicode in auto-generated descriptions
 - Gist: fixed bug in update functionality of Gists that auto changed them
   to private.
 - SVN: add proper escaping in the autogenerated svn mod_dav config

docs/release-notes/release-notes-4.4.2.rst

@@ -7,21 +7,21 b' Release Date'
 - 2016-10-17
 
 
-General
-^^^^^^^
-
-- packaging: pinned against rhodecode-tools 0.10.1
-
-
 New Features
 ^^^^^^^^^^^^
 
 
 
+General
+^^^^^^^
+
+- Packaging: pinned against rhodecode-tools 0.10.1
+
+
 Security
 ^^^^^^^^
 
-- integrations: fix 500 error on integrations page when delegated admin
+- Integrations: fix 500 error on integrations page when delegated admin
   tried to access integration page after adding some integrations.
   Permission checks were to strict for delegated admins.
 
@@ -34,8 +34,8 b' Performance'
 Fixes
 ^^^^^
 
-- vcsserver: make sure we correctly ping against bundled HG/GIT/SVN binaries.
+- Vcsserver: make sure we correctly ping against bundled HG/GIT/SVN binaries.
   This should fix a problem where system binaries could be used accidentally
   by the RhodeCode.
-- ldap: fixed email extraction issues. Empty email addresses from LDAP server
+- LDAP: fixed email extraction issues. Empty email addresses from LDAP server
   will no longer take precedence over those stored inside RhodeCode database.

docs/release-notes/release-notes.rst

@@ -9,6 +9,7 b' Release Notes'
 .. toctree::
    :maxdepth: 1
 
+   release-notes-4.5.0.rst
    release-notes-4.4.2.rst
    release-notes-4.4.1.rst
    release-notes-4.4.0.rst

docs/tutorials/scaling-best-practices.rst

@@ -4,7 +4,7 b''
 Scaling Best Practices
 ======================
 
-When deploying |RCE| at scale; 100s of users, multiple instances, CI servers,
+When deploying |RCE| at scale; 1000s of users, multiple instances, CI servers,
 there are a number of steps you can take to ensure you are getting the
 most out of your system.
 
@@ -15,20 +15,23 b' You can configure multiple |RCE| instanc'
 set of |repos|. This lets users work on an instance that has less traffic
 than those being hit by CI servers. To configure this, use |RCC| to install
 multiple instances and configure the database and |repos| connection. If you
-do need to reset the database connection, see the
+do need to reset/adjust the database connection, see the
 :ref:`config-database` section.
 
-Once configured, set your CI servers to use a particular instance and for
-user specific instances you can configure loads balancing. See the
-:ref:`nginx-ws-ref` section for examples.
+You can configure then a load-balancer to balance the traffic between the CI
+dedicated instance and instance that end users would use.
+See the :ref:`nginx-ws-ref` section for examples on how to do it in NGINX.
 
 Switch to Database Sessions
 ---------------------------
 
-To increase database performance switch to database-based user sessions. In a
-large scale deployment, we recommend switching from file-based
-sessions to database-based user sessions. For configuration details, see the
-:ref:`db-session-ref` section.
+To increase |RCE| performance switch from the default file based sessions to
+database-based. In such way all your distributed instances would not need to
+share the file storage to use file-based sessions.
+Database based session have an additional advantage of the file
+based ones that they don't need a periodic cleanup as the session library
+cleans them up for users. For configuration details,
+see the :ref:`db-session-ref` section.
 
 Tuning |RCE|
 ------------

grunt_config.json

@@ -6,7 +6,9 b''
     },
     "js": {
       "src": "rhodecode/public/js/src",
-      "dest": "rhodecode/public/js"
+      "dest": "rhodecode/public/js",
+      "bower": "bower_components",
+      "node_modules": "node_modules"
     }
   },
   "copy": {
@@ -34,7 +36,8 b''
         "<%= dirs.js.src %>/bootstrap.js",
         "<%= dirs.js.src %>/mousetrap.js",
         "<%= dirs.js.src %>/moment.js",
-        "<%= dirs.js.src %>/appenlight-client-0.4.1.min.js",
+        "<%= dirs.js.node_modules %>/appenlight-client/appenlight-client.min.js",
+        "<%= dirs.js.node_modules %>/favico.js/favico-0.3.10.min.js",
         "<%= dirs.js.src %>/i18n_utils.js",
         "<%= dirs.js.src %>/deform.js",
         "<%= dirs.js.src %>/plugins/jquery.pjax.js",
@@ -64,7 +67,6 b''
         "<%= dirs.js.src %>/rhodecode/utils/ie.js",
         "<%= dirs.js.src %>/rhodecode/utils/os.js",
         "<%= dirs.js.src %>/rhodecode/utils/topics.js",
-        "<%= dirs.js.src %>/rhodecode/widgets/multiselect.js",
         "<%= dirs.js.src %>/rhodecode/init.js",
         "<%= dirs.js.src %>/rhodecode/codemirror.js",
         "<%= dirs.js.src %>/rhodecode/comments.js",
@@ -144,7 +146,9 b''
         "less:development",
         "less:components",
         "concat:polymercss",
-        "vulcanize"
+        "vulcanize",
+        "crisper",
+        "concat:dist"
       ]
     },
     "js": {

package.json

@@ -13,6 +13,8 b''
     "grunt-crisper": "^1.0.1",
     "grunt-vulcanize": "^1.0.0",
     "jshint": "^2.9.1-rc3",
-    "bower": "^1.7.9"
+    "bower": "^1.7.9",
+    "favico.js": "^0.3.10",
+    "appenlight-client": "git+https://[e-mail]/AppEnlight/appenlight-client-js.git#0.5.0"
   }
 }

pkgs/bower-packages.nix

@@ -4,10 +4,12 b' buildEnv { name = "bower-env"; ignoreCol'
   (fetchbower "polymer" "Polymer/polymer#1.6.1" "Polymer/polymer#^1.6.1" "09mm0jgk457gvwqlc155swch7gjr6fs3g7spnvhi6vh5b6518540")
   (fetchbower "paper-button" "PolymerElements/paper-button#1.0.13" "PolymerElements/paper-button#^1.0.13" "0i3y153nqk06pn0gk282vyybnl3g1w3w41d5i9z659cgn27g3fvm")
   (fetchbower "paper-spinner" "PolymerElements/paper-spinner#1.2.0" "PolymerElements/paper-spinner#^1.2.0" "1av1m6y81jw3hjhz1yqy3rwcgxarjzl58ldfn4q6sn51pgzngfqb")
-  (fetchbower "paper-tooltip" "PolymerElements/paper-tooltip#1.1.2" "PolymerElements/paper-tooltip#^1.1.2" "1j64nprcyk2d2bbl3qwjyr0lbjngm4wclpyfwgai1c4y6g6bigd2")
+  (fetchbower "paper-tooltip" "PolymerElements/paper-tooltip#1.1.3" "PolymerElements/paper-tooltip#^1.1.2" "0vmrm1n8k9sk9nvqy03q177axy22pia6i3j1gxbk72j3pqiqvg6k")
   (fetchbower "paper-toast" "PolymerElements/paper-toast#1.3.0" "PolymerElements/paper-toast#^1.3.0" "0x9rqxsks5455s8pk4aikpp99ijdn6kxr9gvhwh99nbcqdzcxq1m")
   (fetchbower "paper-toggle-button" "PolymerElements/paper-toggle-button#1.2.0" "PolymerElements/paper-toggle-button#^1.2.0" "0mphcng3ngspbpg4jjn0mb91nvr4xc1phq3qswib15h6sfww1b2w")
   (fetchbower "iron-ajax" "PolymerElements/iron-ajax#1.4.3" "PolymerElements/iron-ajax#^1.4.3" "0m3dx27arwmlcp00b7n516sc5a51f40p9vapr1nvd57l3i3z0pzm")
+  (fetchbower "iron-autogrow-textarea" "PolymerElements/iron-autogrow-textarea#1.0.13" "PolymerElements/iron-autogrow-textarea#^1.0.13" "0zwhpl97vii1s8k0lgain8i9dnw29b0mxc5ixdscx9las13n2lqq")
+  (fetchbower "iron-a11y-keys" "PolymerElements/iron-a11y-keys#1.0.6" "PolymerElements/iron-a11y-keys#^1.0.6" "1xz3mgghfcxixq28sdb654iaxj4nyi1bzcwf77ydkms6fviqs9mv")
   (fetchbower "iron-flex-layout" "PolymerElements/iron-flex-layout#1.3.1" "PolymerElements/iron-flex-layout#^1.0.0" "0nswv3ih3bhflgcd2wjfmddqswzgqxb2xbq65jk9w3rkj26hplbl")
   (fetchbower "paper-behaviors" "PolymerElements/paper-behaviors#1.0.12" "PolymerElements/paper-behaviors#^1.0.0" "012bqk97awgz55cn7rm9g7cckrdhkqhls3zvp8l6nd4rdwcrdzq8")
   (fetchbower "paper-material" "PolymerElements/paper-material#1.0.6" "PolymerElements/paper-material#^1.0.0" "0rljmknfdbm5aabvx9pk77754zckj3l127c3mvnmwkpkkr353xnh")
@@ -19,13 +21,13 b' buildEnv { name = "bower-env"; ignoreCol'
   (fetchbower "iron-checked-element-behavior" "PolymerElements/iron-checked-element-behavior#1.0.5" "PolymerElements/iron-checked-element-behavior#^1.0.0" "0l0yy4ah454s8bzfv076s8by7h67zy9ni6xb932qwyhx8br6c1m7")
   (fetchbower "promise-polyfill" "polymerlabs/promise-polyfill#1.0.1" "polymerlabs/promise-polyfill#^1.0.0" "045bj2caav3famr5hhxgs1dx7n08r4s46mlzwb313vdy17is38xb")
   (fetchbower "iron-behaviors" "PolymerElements/iron-behaviors#1.0.17" "PolymerElements/iron-behaviors#^1.0.0" "021qvkmbk32jrrmmphpmwgby4bzi5jyf47rh1bxmq2ip07ly4bpr")
+  (fetchbower "iron-validatable-behavior" "PolymerElements/iron-validatable-behavior#1.1.1" "PolymerElements/iron-validatable-behavior#^1.0.0" "1yhxlvywhw2klbbgm3f3cmanxfxggagph4ii635zv0c13707wslv")
+  (fetchbower "iron-form-element-behavior" "PolymerElements/iron-form-element-behavior#1.0.6" "PolymerElements/iron-form-element-behavior#^1.0.0" "0rdhxivgkdhhz2yadgdbjfc70l555p3y83vjh8rfj5hr0asyn6q1")
+  (fetchbower "iron-a11y-keys-behavior" "polymerelements/iron-a11y-keys-behavior#1.1.9" "polymerelements/iron-a11y-keys-behavior#^1.0.0" "1imm4gc84qizihhbyhfa8lwjh3myhj837f79i5m04xjgwrjmkaf6")
   (fetchbower "paper-ripple" "PolymerElements/paper-ripple#1.0.8" "PolymerElements/paper-ripple#^1.0.0" "0r9sq8ik7wwrw0qb82c3rw0c030ljwd3s466c9y4qbcrsbvfjnns")
   (fetchbower "font-roboto" "PolymerElements/font-roboto#1.0.1" "PolymerElements/font-roboto#^1.0.1" "02jz43r0wkyr3yp7rq2rc08l5cwnsgca9fr54sr4rhsnl7cjpxrj")
   (fetchbower "iron-meta" "PolymerElements/iron-meta#1.1.2" "PolymerElements/iron-meta#^1.0.0" "1wl4dx8fnsknw9z9xi8bpc4cy9x70c11x4zxwxnj73hf3smifppl")
   (fetchbower "iron-resizable-behavior" "PolymerElements/iron-resizable-behavior#1.0.5" "PolymerElements/iron-resizable-behavior#^1.0.0" "1fd5zmbr2hax42vmcasncvk7lzi38fmb1kyii26nn8pnnjak7zkn")
   (fetchbower "iron-selector" "PolymerElements/iron-selector#1.5.2" "PolymerElements/iron-selector#^1.0.0" "1ajv46llqzvahm5g6g75w7nfyjcslp53ji0wm96l2k94j87spv3r")
   (fetchbower "web-animations-js" "web-animations/web-animations-js#2.2.2" "web-animations/web-animations-js#^2.2.0" "1izfvm3l67vwys0bqbhidi9rqziw2f8wv289386sc6jsxzgkzhga")
-  (fetchbower "iron-a11y-keys-behavior" "PolymerElements/iron-a11y-keys-behavior#1.1.7" "PolymerElements/iron-a11y-keys-behavior#^1.0.0" "070z46dbbz242002gmqrgy28x0y1fcqp9hnvbi05r3zphiqfx3l7")
-  (fetchbower "iron-validatable-behavior" "PolymerElements/iron-validatable-behavior#1.1.1" "PolymerElements/iron-validatable-behavior#^1.0.0" "1yhxlvywhw2klbbgm3f3cmanxfxggagph4ii635zv0c13707wslv")
-  (fetchbower "iron-form-element-behavior" "PolymerElements/iron-form-element-behavior#1.0.6" "PolymerElements/iron-form-element-behavior#^1.0.0" "0rdhxivgkdhhz2yadgdbjfc70l555p3y83vjh8rfj5hr0asyn6q1")
 ]; }

pkgs/node-packages.nix

@@ -103,6 +103,34 b' let'
         sha1 = "a2e14ff85c2d6bf8c8080e5aa55129ebc6a2d320";
       };
     };
+    "bower-1.7.9" = {
+      name = "bower";
+      packageName = "bower";
+      version = "1.7.9";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/bower/-/bower-1.7.9.tgz";
+        sha1 = "b7296c2393e0d75edaa6ca39648132dd255812b0";
+      };
+    };
+    "favico.js-0.3.10" = {
+      name = "favico.js";
+      packageName = "favico.js";
+      version = "0.3.10";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/favico.js/-/favico.js-0.3.10.tgz";
+        sha1 = "80586e27a117f24a8d51c18a99bdc714d4339301";
+      };
+    };
+    "appenlight-client-git+https://[e-mail]/AppEnlight/appenlight-client-js.git#0.5.0" = {
+      name = "appenlight-client";
+      packageName = "appenlight-client";
+      version = "0.5.0";
+      src = fetchgit {
+        url = "https://[e-mail]/AppEnlight/appenlight-client-js.git";
+        rev = "b1d6853345dc3e96468b34537810b3eb77e0764f";
+        sha256 = "2ef00aef7dafdecdc1666d2e83fc190a796849985d04a8f0fad148d64aa4f8db";
+      };
+    };
     "async-0.1.22" = {
       name = "async";
       packageName = "async";
@@ -301,13 +329,13 b' let'
         sha1 = "fadd834b9683073da179b3eae6d9c0d15053f73e";
       };
     };
-    "inherits-2.0.1" = {
+    "inherits-2.0.3" = {
       name = "inherits";
       packageName = "inherits";
-      version = "2.0.1";
+      version = "2.0.3";
       src = fetchurl {
-        url = "https://registry.npmjs.org/inherits/-/inherits-2.0.1.tgz";
-        sha1 = "b17d08d326b4423e568eff719f91b0b1cbdf69f1";
+        url = "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz";
+        sha1 = "633c2c83e3da42a502f52466022480f4208261de";
       };
     };
     "minimatch-0.3.0" = {
@@ -580,13 +608,13 b' let'
         sha1 = "6cbfea22b3b830304e9a5fb371d54fa480c9d7cf";
       };
     };
-    "lodash-4.15.0" = {
+    "lodash-4.16.2" = {
       name = "lodash";
       packageName = "lodash";
-      version = "4.15.0";
+      version = "4.16.2";
       src = fetchurl {
-        url = "https://registry.npmjs.org/lodash/-/lodash-4.15.0.tgz";
-        sha1 = "3162391d8f0140aa22cf8f6b3c34d6b7f63d3aa9";
+        url = "https://registry.npmjs.org/lodash/-/lodash-4.16.2.tgz";
+        sha1 = "3e626db827048a699281a8a125226326cfc0e652";
       };
     };
     "errno-0.1.4" = {
@@ -598,13 +626,13 b' let'
         sha1 = "b896e23a9e5e8ba33871fc996abd3635fc9a1c7d";
       };
     };
-    "graceful-fs-4.1.6" = {
+    "graceful-fs-4.1.8" = {
       name = "graceful-fs";
       packageName = "graceful-fs";
-      version = "4.1.6";
+      version = "4.1.8";
       src = fetchurl {
-        url = "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.1.6.tgz";
-        sha1 = "514c38772b31bee2e08bedc21a0aeb3abf54c19e";
+        url = "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.1.8.tgz";
+        sha1 = "da3e11135eb2168bdd374532c4e2649751672890";
       };
     };
     "image-size-0.5.0" = {
@@ -670,13 +698,13 b' let'
         sha1 = "857fcabfc3397d2625b8228262e86aa7a011b05d";
       };
     };
-    "asap-2.0.4" = {
+    "asap-2.0.5" = {
       name = "asap";
       packageName = "asap";
-      version = "2.0.4";
+      version = "2.0.5";
       src = fetchurl {
-        url = "https://registry.npmjs.org/asap/-/asap-2.0.4.tgz";
-        sha1 = "b391bf7f6bfbc65706022fec8f49c4b07fecf589";
+        url = "https://registry.npmjs.org/asap/-/asap-2.0.5.tgz";
+        sha1 = "522765b50c3510490e52d7dcfe085ef9ba96958f";
       };
     };
     "gaze-0.5.2" = {
@@ -778,13 +806,13 b' let'
         sha1 = "f197d6eaff34c9085577484b2864375b294f5697";
       };
     };
-    "dom5-1.3.3" = {
+    "dom5-1.3.6" = {
       name = "dom5";
       packageName = "dom5";
-      version = "1.3.3";
+      version = "1.3.6";
       src = fetchurl {
-        url = "https://registry.npmjs.org/dom5/-/dom5-1.3.3.tgz";
-        sha1 = "07e514522c245c7aa8512aa3f9118e8bcab9f909";
+        url = "https://registry.npmjs.org/dom5/-/dom5-1.3.6.tgz";
+        sha1 = "a7088a9fc5f3b08dc9f6eda4c7abaeb241945e0d";
       };
     };
     "array-back-1.0.3" = {
@@ -832,13 +860,13 b' let'
         sha1 = "a2d6ce740d15f0d92b1b26763e2ce9c0e361fd98";
       };
     };
-    "typical-2.5.0" = {
+    "typical-2.6.0" = {
       name = "typical";
       packageName = "typical";
-      version = "2.5.0";
+      version = "2.6.0";
       src = fetchurl {
-        url = "https://registry.npmjs.org/typical/-/typical-2.5.0.tgz";
-        sha1 = "81244918aa28180c9e602aa457173404be0604f1";
+        url = "https://registry.npmjs.org/typical/-/typical-2.6.0.tgz";
+        sha1 = "89d51554ab139848a65bcc2c8772f8fb450c40ed";
       };
     };
     "ansi-escape-sequences-2.2.2" = {
@@ -958,22 +986,22 b' let'
         sha1 = "a09136f72ec043d27c893707c2b159bfad7de93f";
       };
     };
-    "test-value-2.0.0" = {
+    "test-value-2.1.0" = {
       name = "test-value";
       packageName = "test-value";
-      version = "2.0.0";
+      version = "2.1.0";
       src = fetchurl {
-        url = "https://registry.npmjs.org/test-value/-/test-value-2.0.0.tgz";
-        sha1 = "0d65c45ee0b48a757c4507a5e98ec2680a9db137";
+        url = "https://registry.npmjs.org/test-value/-/test-value-2.1.0.tgz";
+        sha1 = "11da6ff670f3471a73b625ca4f3fdcf7bb748291";
       };
     };
-    "@types/clone-0.1.29" = {
+    "@types/clone-0.1.30" = {
       name = "@types/clone";
       packageName = "@types/clone";
-      version = "0.1.29";
+      version = "0.1.30";
       src = fetchurl {
-        url = "https://registry.npmjs.org/@types/clone/-/clone-0.1.29.tgz";
-        sha1 = "65a0be88189ffddcd373e450aa6b68c9c83218b7";
+        url = "https://registry.npmjs.org/@types/clone/-/clone-0.1.30.tgz";
+        sha1 = "e7365648c1b42136a59c7d5040637b3b5c83b614";
       };
     };
     "@types/node-4.0.30" = {
@@ -985,13 +1013,13 b' let'
         sha1 = "553f490ed3030311620f88003e7abfc0edcb301e";
       };
     };
-    "@types/parse5-0.0.28" = {
+    "@types/parse5-0.0.31" = {
       name = "@types/parse5";
       packageName = "@types/parse5";
-      version = "0.0.28";
+      version = "0.0.31";
       src = fetchurl {
-        url = "https://registry.npmjs.org/@types/parse5/-/parse5-0.0.28.tgz";
-        sha1 = "2a38cb7145bb157688d4ad2c46944c6dffae3cc6";
+        url = "https://registry.npmjs.org/@types/parse5/-/parse5-0.0.31.tgz";
+        sha1 = "e827a493a443b156e1b582a2e4c3bdc0040f2ee7";
       };
     };
     "clone-1.0.2" = {
@@ -1012,13 +1040,13 b' let'
         sha1 = "9b7f3b0de32be78dc2401b17573ccaf0f6f59d94";
       };
     };
-    "@types/node-6.0.37" = {
+    "@types/node-6.0.41" = {
       name = "@types/node";
       packageName = "@types/node";
-      version = "6.0.37";
+      version = "6.0.41";
       src = fetchurl {
-        url = "https://registry.npmjs.org/@types/node/-/node-6.0.37.tgz";
-        sha1 = "a1e081f2ec60074113d3a1fbf11f35d304f30e39";
+        url = "https://registry.npmjs.org/@types/node/-/node-6.0.41.tgz";
+        sha1 = "578cf53aaec65887bcaf16792f8722932e8ff8ea";
       };
     };
     "es6-promise-2.3.0" = {
@@ -1093,13 +1121,13 b' let'
         sha1 = "5a5b53af4693110bebb0867aa3430dd3b70a1018";
       };
     };
-    "espree-3.1.7" = {
+    "espree-3.3.1" = {
       name = "espree";
       packageName = "espree";
-      version = "3.1.7";
+      version = "3.3.1";
       src = fetchurl {
-        url = "https://registry.npmjs.org/espree/-/espree-3.1.7.tgz";
-        sha1 = "fd5deec76a97a5120a9cd3a7cb1177a0923b11d2";
+        url = "https://registry.npmjs.org/espree/-/espree-3.3.1.tgz";
+        sha1 = "42107376856738a65ff3b5877f3a58bd52497643";
       };
     };
     "estraverse-3.1.0" = {
@@ -1183,13 +1211,13 b' let'
         sha1 = "96e3b70d5779f6ad49cd032673d1c312767ba581";
       };
     };
-    "optionator-0.8.1" = {
+    "optionator-0.8.2" = {
       name = "optionator";
       packageName = "optionator";
-      version = "0.8.1";
+      version = "0.8.2";
       src = fetchurl {
-        url = "https://registry.npmjs.org/optionator/-/optionator-0.8.1.tgz";
-        sha1 = "e31b4932cdd5fb862a8b0d10bc63d3ee1ec7d78b";
+        url = "https://registry.npmjs.org/optionator/-/optionator-0.8.2.tgz";
+        sha1 = "364c5e409d3f4d6301d6c0b4c05bba50180aeb64";
       };
     };
     "source-map-0.2.0" = {
@@ -1246,13 +1274,31 b' let'
         sha1 = "3b09924edf9f083c0490fdd4c0bc4421e04764ee";
       };
     };
-    "fast-levenshtein-1.1.4" = {
+    "fast-levenshtein-2.0.4" = {
       name = "fast-levenshtein";
       packageName = "fast-levenshtein";
-      version = "1.1.4";
+      version = "2.0.4";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/fast-levenshtein/-/fast-levenshtein-2.0.4.tgz";
+        sha1 = "e31e729eea62233c60a7bc9dce2bdcc88b4fffe3";
+      };
+    };
+    "acorn-4.0.3" = {
+      name = "acorn";
+      packageName = "acorn";
+      version = "4.0.3";
       src = fetchurl {
-        url = "https://registry.npmjs.org/fast-levenshtein/-/fast-levenshtein-1.1.4.tgz";
-        sha1 = "e6a754cc8f15e58987aa9cbd27af66fd6f4e5af9";
+        url = "https://registry.npmjs.org/acorn/-/acorn-4.0.3.tgz";
+        sha1 = "1a3e850b428e73ba6b09d1cc527f5aaad4d03ef1";
+      };
+    };
+    "acorn-jsx-3.0.1" = {
+      name = "acorn-jsx";
+      packageName = "acorn-jsx";
+      version = "3.0.1";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/acorn-jsx/-/acorn-jsx-3.0.1.tgz";
+        sha1 = "afdf9488fb1ecefc8348f6fb22f464e32a58b36b";
       };
     };
     "acorn-3.3.0" = {
@@ -1264,15 +1310,6 b' let'
         sha1 = "45e37fb39e8da3f25baee3ff5369e2bb5f22017a";
       };
     };
-    "acorn-jsx-3.0.1" = {
-      name = "acorn-jsx";
-      packageName = "acorn-jsx";
-      version = "3.0.1";
-      src = fetchurl {
-        url = "https://registry.npmjs.org/acorn-jsx/-/acorn-jsx-3.0.1.tgz";
-        sha1 = "afdf9488fb1ecefc8348f6fb22f464e32a58b36b";
-      };
-    };
     "boxen-0.3.1" = {
       name = "boxen";
       packageName = "boxen";
@@ -1282,13 +1319,13 b' let'
         sha1 = "a7d898243ae622f7abb6bb604d740a76c6a5461b";
       };
     };
-    "configstore-2.0.0" = {
+    "configstore-2.1.0" = {
       name = "configstore";
       packageName = "configstore";
-      version = "2.0.0";
+      version = "2.1.0";
       src = fetchurl {
-        url = "https://registry.npmjs.org/configstore/-/configstore-2.0.0.tgz";
-        sha1 = "8d81e9cdfa73ebd0e06bc985147856b2f1c4e764";
+        url = "https://registry.npmjs.org/configstore/-/configstore-2.1.0.tgz";
+        sha1 = "737a3a7036e9886102aa6099e47bb33ab1aba1a1";
       };
     };
     "is-npm-1.0.0" = {
@@ -1399,13 +1436,13 b' let'
         sha1 = "ef9e31386f031a7f0d643af82fde50c457ef00cb";
       };
     };
-    "dot-prop-2.4.0" = {
+    "dot-prop-3.0.0" = {
       name = "dot-prop";
       packageName = "dot-prop";
-      version = "2.4.0";
+      version = "3.0.0";
       src = fetchurl {
-        url = "https://registry.npmjs.org/dot-prop/-/dot-prop-2.4.0.tgz";
-        sha1 = "848e28f7f1d50740c6747ab3cb07670462b6f89c";
+        url = "https://registry.npmjs.org/dot-prop/-/dot-prop-3.0.0.tgz";
+        sha1 = "1b708af094a49c9a0e7dbcad790aba539dac1177";
       };
     };
     "os-tmpdir-1.0.1" = {
@@ -1426,13 +1463,13 b' let'
         sha1 = "83cf05c6d6458fc4d5ac6362ea325d92f2754217";
       };
     };
-    "uuid-2.0.2" = {
+    "uuid-2.0.3" = {
       name = "uuid";
       packageName = "uuid";
-      version = "2.0.2";
+      version = "2.0.3";
       src = fetchurl {
-        url = "https://registry.npmjs.org/uuid/-/uuid-2.0.2.tgz";
-        sha1 = "48bd5698f0677e3c7901a1c46ef15b1643794726";
+        url = "https://registry.npmjs.org/uuid/-/uuid-2.0.3.tgz";
+        sha1 = "67e2e863797215530dff318e5bf9dcebfd47b21a";
       };
     };
     "write-file-atomic-1.2.0" = {
@@ -1489,13 +1526,13 b' let'
         sha1 = "56eb027d65b4d2dce6cb2e2d32c4d4afc9e1d707";
       };
     };
-    "package-json-2.3.3" = {
+    "package-json-2.4.0" = {
       name = "package-json";
       packageName = "package-json";
-      version = "2.3.3";
+      version = "2.4.0";
       src = fetchurl {
-        url = "https://registry.npmjs.org/package-json/-/package-json-2.3.3.tgz";
-        sha1 = "14895311a963d18edf8801e06b67ea87795d15b9";
+        url = "https://registry.npmjs.org/package-json/-/package-json-2.4.0.tgz";
+        sha1 = "0d15bd67d1cbbddbb2ca222ff2edb86bcb31a8bb";
       };
     };
     "got-5.6.0" = {
@@ -1507,13 +1544,13 b' let'
         sha1 = "bb1d7ee163b78082bbc8eb836f3f395004ea6fbf";
       };
     };
-    "rc-1.1.6" = {
-      name = "rc";
-      packageName = "rc";
-      version = "1.1.6";
+    "registry-auth-token-3.0.1" = {
+      name = "registry-auth-token";
+      packageName = "registry-auth-token";
+      version = "3.0.1";
       src = fetchurl {
-        url = "https://registry.npmjs.org/rc/-/rc-1.1.6.tgz";
-        sha1 = "43651b76b6ae53b5c802f1151fa3fc3b059969c9";
+        url = "https://registry.npmjs.org/registry-auth-token/-/registry-auth-token-3.0.1.tgz";
+        sha1 = "c3ee5ec585bce29f88bf41629a3944c71ed53e25";
       };
     };
     "registry-url-3.1.0" = {
@@ -1651,13 +1688,13 b' let'
         sha1 = "f38b0ae81d3747d628001f41dafc652ace671c0a";
       };
     };
-    "unzip-response-1.0.0" = {
+    "unzip-response-1.0.1" = {
       name = "unzip-response";
       packageName = "unzip-response";
-      version = "1.0.0";
+      version = "1.0.1";
       src = fetchurl {
-        url = "https://registry.npmjs.org/unzip-response/-/unzip-response-1.0.0.tgz";
-        sha1 = "bfda54eeec658f00c2df4d4494b9dca0ca00f3e4";
+        url = "https://registry.npmjs.org/unzip-response/-/unzip-response-1.0.1.tgz";
+        sha1 = "4a73959f2989470fa503791cefb54e1dbbc68412";
       };
     };
     "url-parse-lax-1.0.0" = {
@@ -1768,6 +1805,15 b' let'
         sha1 = "d4f4562b0ce3696e41ac52d0e002e57a635dc6dc";
       };
     };
+    "rc-1.1.6" = {
+      name = "rc";
+      packageName = "rc";
+      version = "1.1.6";
+      src = fetchurl {
+        url = "https://registry.npmjs.org/rc/-/rc-1.1.6.tgz";
+        sha1 = "43651b76b6ae53b5c802f1151fa3fc3b059969c9";
+      };
+    };
     "ini-1.3.4" = {
       name = "ini";
       packageName = "ini";
@@ -1858,13 +1904,13 b' let'
         sha1 = "3678bd8ab995057c07ade836ed2ef087da811d45";
       };
     };
-    "glob-7.0.6" = {
+    "glob-7.1.0" = {
       name = "glob";
       packageName = "glob";
-      version = "7.0.6";
+      version = "7.1.0";
       src = fetchurl {
-        url = "https://registry.npmjs.org/glob/-/glob-7.0.6.tgz";
-        sha1 = "211bafaf49e525b8cd93260d14ab136152b3f57a";
+        url = "https://registry.npmjs.org/glob/-/glob-7.1.0.tgz";
+        sha1 = "36add856d746d0d99e4cc2797bba1ae2c67272fd";
       };
     };
     "fs.realpath-1.0.0" = {
@@ -1885,13 +1931,13 b' let'
         sha1 = "db3204cd5a9de2e6cd890b85c6e2f66bcf4f620a";
       };
     };
-    "once-1.3.3" = {
+    "once-1.4.0" = {
       name = "once";
       packageName = "once";
-      version = "1.3.3";
+      version = "1.4.0";
       src = fetchurl {
-        url = "https://registry.npmjs.org/once/-/once-1.3.3.tgz";
-        sha1 = "b2e261557ce4c314ec8304f3fa82663e4297ca20";
+        url = "https://registry.npmjs.org/once/-/once-1.4.0.tgz";
+        sha1 = "583b1aa775961d4b113ac17d9c50baef9dd76bd1";
       };
     };
     "wrappy-1.0.2" = {
@@ -2034,7 +2080,7 b' let'
       (sources."grunt-contrib-less-1.4.0" // {
         dependencies = [
           sources."async-2.0.1"
-          sources."lodash-4.15.0"
+          sources."lodash-4.16.2"
         ];
       })
       (sources."grunt-contrib-watch-0.6.1" // {
@@ -2062,6 +2108,9 b' let'
           sources."lodash-3.7.0"
         ];
       })
+      sources."bower-1.7.9"
+      sources."favico.js-0.3.10"
+      sources."appenlight-client-git+https://[e-mail]/AppEnlight/appenlight-client-js.git#0.5.0"
       sources."async-0.1.22"
       sources."coffee-script-1.3.3"
       sources."colors-0.6.2"
@@ -2097,7 +2146,7 b' let'
           sources."underscore.string-2.3.3"
         ];
       })
-      sources."inherits-2.0.1"
+      sources."inherits-2.0.3"
       sources."lru-cache-2.7.3"
       sources."sigmund-1.0.1"
       sources."graceful-fs-1.2.3"
@@ -2127,7 +2176,7 b' let'
       sources."amdefine-1.0.0"
       (sources."less-2.7.1" // {
         dependencies = [
-          sources."graceful-fs-4.1.6"
+          sources."graceful-fs-4.1.8"
           sources."source-map-0.5.6"
         ];
       })
@@ -2138,7 +2187,7 b' let'
       sources."promise-7.1.1"
       sources."prr-0.0.0"
       sources."minimist-0.0.8"
-      sources."asap-2.0.4"
+      sources."asap-2.0.5"
       sources."gaze-0.5.2"
       sources."tiny-lr-fork-0.0.5"
       (sources."globule-0.1.0" // {
@@ -2155,17 +2204,17 b' let'
       })
       sources."debug-0.7.4"
       sources."command-line-args-2.1.6"
-      sources."dom5-1.3.3"
+      sources."dom5-1.3.6"
       sources."array-back-1.0.3"
       sources."command-line-usage-2.0.5"
       sources."core-js-2.4.1"
       sources."feature-detect-es6-1.3.1"
       (sources."find-replace-1.0.2" // {
         dependencies = [
-          sources."test-value-2.0.0"
+          sources."test-value-2.1.0"
         ];
       })
-      sources."typical-2.5.0"
+      sources."typical-2.6.0"
       sources."ansi-escape-sequences-2.2.2"
       sources."column-layout-2.1.4"
       sources."wordwrapjs-1.2.1"
@@ -2182,11 +2231,11 b' let'
       sources."object-tools-2.0.6"
       sources."object-get-2.1.0"
       sources."test-value-1.1.0"
-      sources."@types/clone-0.1.29"
+      sources."@types/clone-0.1.30"
       sources."@types/node-4.0.30"
-      (sources."@types/parse5-0.0.28" // {
+      (sources."@types/parse5-0.0.31" // {
         dependencies = [
-          sources."@types/node-6.0.37"
+          sources."@types/node-6.0.41"
         ];
       })
       sources."clone-1.0.2"
@@ -2205,26 +2254,30 b' let'
           sources."source-map-0.2.0"
         ];
       })
-      sources."espree-3.1.7"
+      sources."espree-3.3.1"
       sources."estraverse-3.1.0"
       sources."path-is-absolute-1.0.0"
       sources."babel-runtime-6.11.6"
       sources."regenerator-runtime-0.9.5"
       sources."esutils-1.1.6"
       sources."isarray-0.0.1"
-      sources."optionator-0.8.1"
+      sources."optionator-0.8.2"
       sources."prelude-ls-1.1.2"
       sources."deep-is-0.1.3"
       sources."wordwrap-1.0.0"
       sources."type-check-0.3.2"
       sources."levn-0.3.0"
-      sources."fast-levenshtein-1.1.4"
-      sources."acorn-3.3.0"
-      sources."acorn-jsx-3.0.1"
+      sources."fast-levenshtein-2.0.4"
+      sources."acorn-4.0.3"
+      (sources."acorn-jsx-3.0.1" // {
+        dependencies = [
+          sources."acorn-3.3.0"
+        ];
+      })
       sources."boxen-0.3.1"
-      (sources."configstore-2.0.0" // {
+      (sources."configstore-2.1.0" // {
         dependencies = [
-          sources."graceful-fs-4.1.6"
+          sources."graceful-fs-4.1.8"
         ];
       })
       sources."is-npm-1.0.0"
@@ -2239,13 +2292,13 b' let'
       sources."number-is-nan-1.0.0"
       sources."code-point-at-1.0.0"
       sources."is-fullwidth-code-point-1.0.0"
-      sources."dot-prop-2.4.0"
+      sources."dot-prop-3.0.0"
       sources."os-tmpdir-1.0.1"
       sources."osenv-0.1.3"
-      sources."uuid-2.0.2"
+      sources."uuid-2.0.3"
       (sources."write-file-atomic-1.2.0" // {
         dependencies = [
-          sources."graceful-fs-4.1.6"
+          sources."graceful-fs-4.1.8"
         ];
       })
       sources."xdg-basedir-2.0.0"
@@ -2253,13 +2306,9 b' let'
       sources."os-homedir-1.0.1"
       sources."imurmurhash-0.1.4"
       sources."slide-1.1.6"
-      sources."package-json-2.3.3"
+      sources."package-json-2.4.0"
       sources."got-5.6.0"
-      (sources."rc-1.1.6" // {
-        dependencies = [
-          sources."minimist-1.2.0"
-        ];
-      })
+      sources."registry-auth-token-3.0.1"
       sources."registry-url-3.1.0"
       sources."semver-5.3.0"
       sources."create-error-class-3.0.2"
@@ -2279,7 +2328,7 b' let'
         ];
       })
       sources."timed-out-2.0.0"
-      sources."unzip-response-1.0.0"
+      sources."unzip-response-1.0.1"
       sources."url-parse-lax-1.0.0"
       sources."capture-stack-trace-1.0.0"
       sources."error-ex-1.3.0"
@@ -2291,11 +2340,16 b' let'
       sources."string_decoder-0.10.31"
       sources."util-deprecate-1.0.2"
       sources."prepend-http-1.0.4"
+      (sources."rc-1.1.6" // {
+        dependencies = [
+          sources."minimist-1.2.0"
+        ];
+      })
       sources."ini-1.3.4"
       sources."strip-json-comments-1.0.4"
       (sources."cli-1.0.0" // {
         dependencies = [
-          sources."glob-7.0.6"
+          sources."glob-7.1.0"
           sources."minimatch-3.0.3"
         ];
       })
@@ -2308,7 +2362,7 b' let'
       sources."shelljs-0.3.0"
       sources."fs.realpath-1.0.0"
       sources."inflight-1.0.5"
-      sources."once-1.3.3"
+      sources."once-1.4.0"
       sources."wrappy-1.0.2"
       sources."brace-expansion-1.1.6"
       sources."balanced-match-0.4.2"

pkgs/python-packages-overrides.nix

@@ -15,19 +15,6 b' let'
     };
   };
 
-  # johbo: Interim bridge which allows us to build with the upcoming
-  # nixos.16.09 branch (unstable at the moment of writing this note) and the
-  # current stable nixos-16.03.
-  backwardsCompatibleFetchgit = { ... }@args:
-    let
-      origSources = pkgs.fetchgit args;
-    in
-    pkgs.lib.overrideDerivation origSources (oldAttrs: {
-      NIX_PREFETCH_GIT_CHECKOUT_HOOK = ''
-        find $out -name '.git*' -print0 | xargs -0 rm -rf
-      '';
-    });
-
 in
 
 self: super: {
@@ -77,6 +64,9 b' self: super: {'
   });
 
   lxml = super.lxml.override (attrs: {
+    # johbo: On 16.09 we need this to compile on darwin, otherwise compilation
+    # fails on Darwin.
+    hardeningDisable = if pkgs.stdenv.isDarwin then [ "format" ] else null;
     buildInputs = with self; [
       pkgs.libxml2
       pkgs.libxslt
@@ -110,7 +100,7 b' self: super: {'
   });
 
   py-gfm = super.py-gfm.override  {
-    src = backwardsCompatibleFetchgit {
+    src = pkgs.fetchgit {
       url = "https://code.rhodecode.com/upstream/py-gfm";
       rev = "0d66a19bc16e3d49de273c0f797d4e4781e8c0f2";
       sha256 = "0ryp74jyihd3ckszq31bml5jr3bciimhfp7va7kw6ld92930ksv3";
@@ -134,7 +124,7 b' self: super: {'
 
   Pylons = super.Pylons.override (attrs: {
     name = "Pylons-1.0.1-patch1";
-    src = backwardsCompatibleFetchgit {
+    src = pkgs.fetchgit {
       url = "https://code.rhodecode.com/upstream/pylons";
       rev = "707354ee4261b9c10450404fc9852ccea4fd667d";
       sha256 = "b2763274c2780523a335f83a1df65be22ebe4ff413a7bc9e9288d23c1f62032e";
@@ -190,7 +180,8 b' self: super: {'
       pkgs.openldap
       pkgs.openssl
     ];
-    NIX_CFLAGS_COMPILE = "-I${pkgs.cyrus_sasl}/include/sasl";
+    # TODO: johbo: Remove the "or" once we drop 16.03 support.
+    NIX_CFLAGS_COMPILE = "-I${pkgs.cyrus_sasl.dev or pkgs.cyrus_sasl}/include/sasl";
   });
 
   python-pam = super.python-pam.override (attrs:

pkgs/python-packages.nix

@@ -431,6 +431,19 b''
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
+  backports.shutil-get-terminal-size = super.buildPythonPackage {
+    name = "backports.shutil-get-terminal-size-1.0.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/ec/9c/368086faa9c016efce5da3e0e13ba392c9db79e3ab740b763fe28620b18b/backports.shutil_get_terminal_size-1.0.0.tar.gz";
+      md5 = "03267762480bd86b50580dc19dff3c66";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
   bottle = super.buildPythonPackage {
     name = "bottle-0.12.8";
     buildInputs = with self; [];
@@ -678,17 +691,17 b''
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
-  flake8 = super.buildPythonPackage {
-    name = "flake8-2.4.1";
+  enum34 = super.buildPythonPackage {
+    name = "enum34-1.1.6";
     buildInputs = with self; [];
     doCheck = false;
-    propagatedBuildInputs = with self; [pyflakes pep8 mccabe];
+    propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/8f/b5/9a73c66c7dba273bac8758398f060c008a25f3e84531063b42503b5d0a95/flake8-2.4.1.tar.gz";
-      md5 = "ed45d3db81a3b7c88bd63c6e37ca1d65";
+      url = "https://pypi.python.org/packages/bf/3e/31d502c25302814a7c2f1d3959d2a3b3f78e509002ba91aea64993936876/enum34-1.1.6.tar.gz";
+      md5 = "5f13a0841a61f7fc295c514490d120d0";
     };
     meta = {
-      license = [ pkgs.lib.licenses.mit ];
+      license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   future = super.buildPythonPackage {
@@ -809,26 +822,39 b''
     };
   };
   ipdb = super.buildPythonPackage {
-    name = "ipdb-0.8";
+    name = "ipdb-0.10.1";
     buildInputs = with self; [];
     doCheck = false;
-    propagatedBuildInputs = with self; [ipython];
+    propagatedBuildInputs = with self; [ipython setuptools];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/f0/25/d7dd430ced6cd8dc242a933c8682b5dbf32eb4011d82f87e34209e5ec845/ipdb-0.8.zip";
-      md5 = "96dca0712efa01aa5eaf6b22071dd3ed";
+      url = "https://pypi.python.org/packages/eb/0a/0a37dc19572580336ad3813792c0d18c8d7117c2d66fc63c501f13a7a8f8/ipdb-0.10.1.tar.gz";
+      md5 = "4aeab65f633ddc98ebdb5eebf08dc713";
     };
     meta = {
-      license = [ pkgs.lib.licenses.gpl1 ];
+      license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipython = super.buildPythonPackage {
-    name = "ipython-3.1.0";
+    name = "ipython-5.1.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [setuptools decorator pickleshare simplegeneric traitlets prompt-toolkit Pygments pexpect backports.shutil-get-terminal-size pathlib2 pexpect];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/89/63/a9292f7cd9d0090a0f995e1167f3f17d5889dcbc9a175261719c513b9848/ipython-5.1.0.tar.gz";
+      md5 = "47c8122420f65b58784cb4b9b4af35e3";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
+  ipython-genutils = super.buildPythonPackage {
+    name = "ipython-genutils-0.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/06/91/120c0835254c120af89f066afaabf81289bc2726c1fc3ca0555df6882f58/ipython-3.1.0.tar.gz";
-      md5 = "a749d90c16068687b0ec45a27e72ef8f";
+      url = "https://pypi.python.org/packages/71/b7/a64c71578521606edbbce15151358598f3dfb72a3431763edc2baf19e71f/ipython_genutils-0.1.0.tar.gz";
+      md5 = "9a8afbe0978adbcbfcb3b35b2d015a56";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -886,19 +912,6 b''
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
-  mccabe = super.buildPythonPackage {
-    name = "mccabe-0.3";
-    buildInputs = with self; [];
-    doCheck = false;
-    propagatedBuildInputs = with self; [];
-    src = fetchurl {
-      url = "https://pypi.python.org/packages/c9/2e/75231479e11a906b64ac43bad9d0bb534d00080b18bdca8db9da46e1faf7/mccabe-0.3.tar.gz";
-      md5 = "81640948ff226f8c12b3277059489157";
-    };
-    meta = {
-      license = [ { fullName = "Expat license"; } pkgs.lib.licenses.mit ];
-    };
-  };
   meld3 = super.buildPythonPackage {
     name = "meld3-1.0.2";
     buildInputs = with self; [];
@@ -990,17 +1003,17 b''
       license = [ { fullName = "LGPL"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
     };
   };
-  pep8 = super.buildPythonPackage {
-    name = "pep8-1.5.7";
+  pathlib2 = super.buildPythonPackage {
+    name = "pathlib2-2.1.0";
     buildInputs = with self; [];
     doCheck = false;
-    propagatedBuildInputs = with self; [];
+    propagatedBuildInputs = with self; [six];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/8b/de/259f5e735897ada1683489dd514b2a1c91aaa74e5e6b68f80acf128a6368/pep8-1.5.7.tar.gz";
-      md5 = "f6adbdd69365ecca20513c709f9b7c93";
+      url = "https://pypi.python.org/packages/c9/27/8448b10d8440c08efeff0794adf7d0ed27adb98372c70c7b38f3947d4749/pathlib2-2.1.0.tar.gz";
+      md5 = "38e4f58b4d69dfcb9edb49a54a8b28d2";
     };
     meta = {
-      license = [ { fullName = "Expat license"; } pkgs.lib.licenses.mit ];
+      license = [ pkgs.lib.licenses.mit ];
     };
   };
   peppercorn = super.buildPythonPackage {
@@ -1016,14 +1029,53 b''
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
+  pexpect = super.buildPythonPackage {
+    name = "pexpect-4.2.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [ptyprocess];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/e8/13/d0b0599099d6cd23663043a2a0bb7c61e58c6ba359b2656e6fb000ef5b98/pexpect-4.2.1.tar.gz";
+      md5 = "3694410001a99dff83f0b500a1ca1c95";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.isc { fullName = "ISC License (ISCL)"; } ];
+    };
+  };
+  pickleshare = super.buildPythonPackage {
+    name = "pickleshare-0.7.4";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [pathlib2];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/69/fe/dd137d84daa0fd13a709e448138e310d9ea93070620c9db5454e234af525/pickleshare-0.7.4.tar.gz";
+      md5 = "6a9e5dd8dfc023031f6b7b3f824cab12";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
+  prompt-toolkit = super.buildPythonPackage {
+    name = "prompt-toolkit-1.0.9";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [six wcwidth];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/83/14/5ac258da6c530eca02852ee25c7a9ff3ca78287bb4c198d0d0055845d856/prompt_toolkit-1.0.9.tar.gz";
+      md5 = "a39f91a54308fb7446b1a421c11f227c";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
   psutil = super.buildPythonPackage {
-    name = "psutil-2.2.1";
+    name = "psutil-4.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/df/47/ee54ef14dd40f8ce831a7581001a5096494dc99fe71586260ca6b531fe86/psutil-2.2.1.tar.gz";
-      md5 = "1a2b58cd9e3a53528bb6148f0c4d5244";
+      url = "https://pypi.python.org/packages/78/cc/f267a1371f229bf16db6a4e604428c3b032b823b83155bd33cef45e49a53/psutil-4.3.1.tar.gz";
+      md5 = "199a366dba829c88bddaf5b41d19ddc0";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
@@ -1042,6 +1094,19 b''
       license = [ pkgs.lib.licenses.zpt21 { fullName = "GNU Library or Lesser General Public License (LGPL)"; } { fullName = "LGPL with exceptions or ZPL"; } ];
     };
   };
+  ptyprocess = super.buildPythonPackage {
+    name = "ptyprocess-0.5.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/db/d7/b465161910f3d1cef593c5e002bff67e0384898f597f1a7fdc8db4c02bf6/ptyprocess-0.5.1.tar.gz";
+      md5 = "94e537122914cc9ec9c1eadcd36e73a1";
+    };
+    meta = {
+      license = [  ];
+    };
+  };
   py = super.buildPythonPackage {
     name = "py-1.4.29";
     buildInputs = with self; [];
@@ -1120,6 +1185,19 b''
       license = [ pkgs.lib.licenses.mit ];
     };
   };
+  pygments-markdown-lexer = super.buildPythonPackage {
+    name = "pygments-markdown-lexer-0.1.0.dev39";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [Pygments];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/c3/12/674cdee66635d638cedb2c5d9c85ce507b7b2f91bdba29e482f1b1160ff6/pygments-markdown-lexer-0.1.0.dev39.zip";
+      md5 = "6360fe0f6d1f896e35b7a0142ce6459c";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
+  };
   pyparsing = super.buildPythonPackage {
     name = "pyparsing-1.5.7";
     buildInputs = with self; [];
@@ -1420,10 +1498,10 b''
     };
   };
   rhodecode-enterprise-ce = super.buildPythonPackage {
-    name = "rhodecode-enterprise-ce-4.4.2";
-    buildInputs = with self; [WebTest configobj cssselect flake8 lxml mock pytest pytest-cov pytest-runner];
+    name = "rhodecode-enterprise-ce-4.5.0";
+    buildInputs = with self; [WebTest configobj cssselect lxml mock pytest pytest-cov pytest-runner pytest-sugar];
     doCheck = true;
-    propagatedBuildInputs = with self; [Babel Beaker FormEncode Mako Markdown MarkupSafe MySQL-python Paste PasteDeploy PasteScript Pygments Pylons Pyro4 Routes SQLAlchemy Tempita URLObject WebError WebHelpers WebHelpers2 WebOb WebTest Whoosh alembic amqplib anyjson appenlight-client authomatic backport-ipaddress celery channelstream colander decorator deform docutils gevent gunicorn infrae.cache ipython iso8601 kombu msgpack-python packaging psycopg2 py-gfm pycrypto pycurl pyparsing pyramid pyramid-debugtoolbar pyramid-mako pyramid-beaker pysqlite python-dateutil python-ldap python-memcached python-pam recaptcha-client repoze.lru requests simplejson waitress zope.cachedescriptors dogpile.cache dogpile.core psutil py-bcrypt];
+    propagatedBuildInputs = with self; [Babel Beaker FormEncode Mako Markdown MarkupSafe MySQL-python Paste PasteDeploy PasteScript Pygments pygments-markdown-lexer Pylons Pyro4 Routes SQLAlchemy Tempita URLObject WebError WebHelpers WebHelpers2 WebOb WebTest Whoosh alembic amqplib anyjson appenlight-client authomatic backport-ipaddress celery channelstream colander decorator deform docutils gevent gunicorn infrae.cache ipython iso8601 kombu msgpack-python packaging psycopg2 py-gfm pycrypto pycurl pyparsing pyramid pyramid-debugtoolbar pyramid-mako pyramid-beaker pysqlite python-dateutil python-ldap python-memcached python-pam recaptcha-client repoze.lru requests simplejson subprocess32 waitress zope.cachedescriptors dogpile.cache dogpile.core psutil py-bcrypt];
     src = ./.;
     meta = {
       license = [ { fullName = "AGPLv3, and Commercial License"; } ];
@@ -1494,6 +1572,19 b''
       license = [ pkgs.lib.licenses.mit ];
     };
   };
+  simplegeneric = super.buildPythonPackage {
+    name = "simplegeneric-0.8.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip";
+      md5 = "f9c1fab00fd981be588fc32759f474e3";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 ];
+    };
+  };
   simplejson = super.buildPythonPackage {
     name = "simplejson-3.7.2";
     buildInputs = with self; [];
@@ -1546,6 +1637,19 b''
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
+  traitlets = super.buildPythonPackage {
+    name = "traitlets-4.3.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [ipython-genutils six decorator enum34];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/b1/d6/5b5aa6d5c474691909b91493da1e8972e309c9f01ecfe4aeafd272eb3234/traitlets-4.3.1.tar.gz";
+      md5 = "dd0b1b6e5d31ce446d55a4b5e5083c98";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
   transifex-client = super.buildPythonPackage {
     name = "transifex-client-0.10";
     buildInputs = with self; [];
@@ -1637,6 +1741,19 b''
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
+  wcwidth = super.buildPythonPackage {
+    name = "wcwidth-0.1.7";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/55/11/e4a2bb08bb450fdbd42cc709dd40de4ed2c472cf0ccb9e64af22279c5495/wcwidth-0.1.7.tar.gz";
+      md5 = "b3b6a0a08f0c8a34d1de8cf44150a4ad";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
   ws4py = super.buildPythonPackage {
     name = "ws4py-0.3.5";
     buildInputs = with self; [];
@@ -1718,5 +1835,30 b''
 
 ### Test requirements
 
-  
+  pytest-sugar = super.buildPythonPackage {
+    name = "pytest-sugar-0.7.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [pytest termcolor];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/03/97/05d988b4fa870e7373e8ee4582408543b9ca2bd35c3c67b569369c6f9c49/pytest-sugar-0.7.1.tar.gz";
+      md5 = "7400f7c11f3d572b2c2a3b60352d35fe";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  };
+  termcolor = super.buildPythonPackage {
+    name = "termcolor-1.1.0";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz";
+      md5 = "043e89644f8909d462fbbfa511c768df";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
 }

pytest.ini

@@ -1,9 +1,9 b''
 [pytest]
 testpaths = ./rhodecode
-pylons_config = test.ini
-vcsserver_protocol = pyro4
-vcsserver_config = rhodecode/tests/vcsserver.ini
-vcsserver_config_http = rhodecode/tests/vcsserver_pyramid.ini
+pylons_config = rhodecode/tests/rhodecode.ini
+vcsserver_protocol = http
+vcsserver_config_pyro4 = rhodecode/tests/vcsserver_pyro4.ini
+vcsserver_config_http = rhodecode/tests/vcsserver_http.ini
 norecursedirs = tests/scripts
 addopts = -k "not _BaseTest"
 markers =

requirements.txt

@@ -1,5 +1,6 b''
 Babel==1.3
 Beaker==1.7.0
+Chameleon==2.24
 CProfileV==1.0.6
 FormEncode==1.2.4
 Jinja2==2.7.3
@@ -11,6 +12,7 b' Paste==2.0.2'
 PasteDeploy==1.5.2
 PasteScript==1.7.5
 Pygments==2.1.3
+pygments-markdown-lexer==0.1.0.dev39
 
 # TODO: This version is not available on PyPI
 # Pylons==1.0.2.dev20160108
@@ -62,7 +64,6 b' dogpile.cache==0.6.1'
 dogpile.core==0.4.1
 dulwich==0.12.0
 ecdsa==0.11
-flake8==2.4.1
 future==0.14.3
 futures==3.0.2
 gevent==1.1.1
@@ -77,13 +78,12 b' gunicorn==19.6.0'
 gnureadline==6.3.3
 infrae.cache==1.0.1
 invoke==0.13.0
-ipdb==0.8
-ipython==3.1.0
+ipdb==0.10.1
+ipython==5.1.0
 iso8601==0.1.11
 itsdangerous==0.24
 kombu==1.5.1
 lxml==3.4.4
-mccabe==0.3
 meld3==1.0.2
 mock==1.0.1
 msgpack-python==0.4.6
@@ -91,8 +91,7 b' nose==1.3.6'
 objgraph==2.0.0
 packaging==15.2
 paramiko==1.15.1
-pep8==1.5.7
-psutil==2.2.1
+psutil==4.3.1
 psycopg2==2.6.1
 py==1.4.29
 py-bcrypt==0.4
@@ -141,6 +140,7 b' transifex-client==0.10'
 translationstring==1.3
 trollius==1.0.4
 uWSGI==2.0.11.2
+urllib3==1.16
 venusian==1.0
 waitress==0.8.9
 wsgiref==0.1.2

rhodecode/VERSION

@@ -1,1 +1,1 b''
-4.4.2
  No newline at end of file
+4.5.0
  No newline at end of file

rhodecode/__init__.py

@@ -51,7 +51,7 b' PYRAMID_SETTINGS = {}'
 EXTENSIONS = {}
 
 __version__ = ('.'.join((str(each) for each in VERSION[:3])))
-__dbversion__ = 58  # defines current db version for migrations
+__dbversion__ = 63  # defines current db version for migrations
 __platform__ = platform.system()
 __license__ = 'AGPLv3, and Commercial License'
 __author__ = 'RhodeCode GmbH'

rhodecode/admin/__init__.py

@@ -35,6 +35,9 b' def includeme(config):'
     config.add_route(
         name='admin_settings_open_source',
         pattern=ADMIN_PREFIX + '/settings/open_source')
+    config.add_route(
+        name='admin_settings_vcs_svn_generate_cfg',
+        pattern=ADMIN_PREFIX + '/settings/vcs/svn_generate_cfg')
 
     # Scan module for configuration decorators.
     config.scan()

rhodecode/admin/views.py

@@ -24,8 +24,11 b' import logging'
 from pylons import tmpl_context as c
 from pyramid.view import view_config
 
-from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator
+from rhodecode.lib.auth import (
+    LoginRequired, HasPermissionAllDecorator, CSRFRequired)
 from rhodecode.lib.utils import read_opensource_licenses
+from rhodecode.svn_support.utils import generate_mod_dav_svn_config
+from rhodecode.translation import _
 
 from .navigation import navigation_list
 
@@ -53,3 +56,27 b' class AdminSettingsView(object):'
             sorted(read_opensource_licenses().items(), key=lambda t: t[0]))
 
         return {}
+
+    @LoginRequired()
+    @CSRFRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        route_name='admin_settings_vcs_svn_generate_cfg',
+        request_method='POST', renderer='json')
+    def vcs_svn_generate_config(self):
+        try:
+            generate_mod_dav_svn_config(self.request.registry)
+            msg = {
+                'message': _('Apache configuration for Subversion generated.'),
+                'level': 'success',
+            }
+        except Exception:
+            log.exception(
+                'Exception while generating the Apache configuration for Subversion.')
+            msg = {
+                'message': _('Failed to generate the Apache configuration for Subversion.'),
+                'level': 'error',
+            }
+
+        data = {'message': msg}
+        return data

rhodecode/api/tests/test_create_repo.py

@@ -23,8 +23,11 b' import json'
 import mock
 import pytest
 
+from rhodecode.lib.utils2 import safe_unicode
 from rhodecode.lib.vcs import settings
+from rhodecode.model.meta import Session
 from rhodecode.model.repo import RepoModel
+from rhodecode.model.user import UserModel
 from rhodecode.tests import TEST_USER_ADMIN_LOGIN
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_ok, assert_error, crash)
@@ -36,29 +39,37 b' fixture = Fixture()'
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestCreateRepo(object):
-    def test_api_create_repo(self, backend):
-        repo_name = 'api-repo-1'
+
+    @pytest.mark.parametrize('given, expected_name, expected_exc', [
+        ('api repo-1', 'api-repo-1', False),
+        ('api-repo 1-ąć', 'api-repo-1-ąć', False),
+        (u'unicode-ąć', u'unicode-ąć', False),
+        ('some repo v1.2', 'some-repo-v1.2', False),
+        ('v2.0', 'v2.0', False),
+    ])
+    def test_api_create_repo(self, backend, given, expected_name, expected_exc):
+
         id_, params = build_data(
             self.apikey,
             'create_repo',
-            repo_name=repo_name,
+            repo_name=given,
             owner=TEST_USER_ADMIN_LOGIN,
             repo_type=backend.alias,
         )
         response = api_call(self.app, params)
 
-        repo = RepoModel().get_by_repo_name(repo_name)
-
-        assert repo is not None
         ret = {
-            'msg': 'Created new repository `%s`' % (repo_name,),
+            'msg': 'Created new repository `%s`' % (expected_name,),
             'success': True,
             'task': None,
         }
         expected = ret
         assert_ok(id_, expected, given=response.body)
 
-        id_, params = build_data(self.apikey, 'get_repo', repoid=repo_name)
+        repo = RepoModel().get_by_repo_name(safe_unicode(expected_name))
+        assert repo is not None
+
+        id_, params = build_data(self.apikey, 'get_repo', repoid=expected_name)
         response = api_call(self.app, params)
         body = json.loads(response.body)
 
@@ -66,7 +77,7 b' class TestCreateRepo(object):'
         assert body['result']['enable_locking'] is False
         assert body['result']['enable_statistics'] is False
 
-        fixture.destroy_repo(repo_name)
+        fixture.destroy_repo(safe_unicode(expected_name))
 
     def test_api_create_restricted_repo_type(self, backend):
         repo_name = 'api-repo-type-{0}'.format(backend.alias)
@@ -158,6 +169,21 b' class TestCreateRepo(object):'
         fixture.destroy_repo(repo_name)
         fixture.destroy_repo_group(repo_group_name)
 
+    def test_create_repo_in_group_that_doesnt_exist(self, backend, user_util):
+        repo_group_name = 'fake_group'
+
+        repo_name = '%s/api-repo-gr' % (repo_group_name,)
+        id_, params = build_data(
+            self.apikey, 'create_repo',
+            repo_name=repo_name,
+            owner=TEST_USER_ADMIN_LOGIN,
+            repo_type=backend.alias,)
+        response = api_call(self.app, params)
+
+        expected = {'repo_group': 'Repository group `{}` does not exist'.format(
+            repo_group_name)}
+        assert_error(id_, expected, given=response.body)
+
     def test_api_create_repo_unknown_owner(self, backend):
         repo_name = 'api-repo-2'
         owner = 'i-dont-exist'
@@ -218,10 +244,48 b' class TestCreateRepo(object):'
             owner=owner)
         response = api_call(self.app, params)
 
-        expected = 'Only RhodeCode admin can specify `owner` param'
+        expected = 'Only RhodeCode super-admin can specify `owner` param'
         assert_error(id_, expected, given=response.body)
         fixture.destroy_repo(repo_name)
 
+    def test_api_create_repo_by_non_admin_no_parent_group_perms(self, backend):
+        repo_group_name = 'no-access'
+        fixture.create_repo_group(repo_group_name)
+        repo_name = 'no-access/api-repo'
+
+        id_, params = build_data(
+            self.apikey_regular, 'create_repo',
+            repo_name=repo_name,
+            repo_type=backend.alias)
+        response = api_call(self.app, params)
+
+        expected = {'repo_group': 'Repository group `{}` does not exist'.format(
+            repo_group_name)}
+        assert_error(id_, expected, given=response.body)
+        fixture.destroy_repo_group(repo_group_name)
+        fixture.destroy_repo(repo_name)
+
+    def test_api_create_repo_non_admin_no_permission_to_create_to_root_level(
+            self, backend, user_util):
+
+        regular_user = user_util.create_user()
+        regular_user_api_key = regular_user.api_key
+
+        usr = UserModel().get_by_username(regular_user.username)
+        usr.inherit_default_permissions = False
+        Session().add(usr)
+
+        repo_name = backend.new_repo_name()
+        id_, params = build_data(
+            regular_user_api_key, 'create_repo',
+            repo_name=repo_name,
+            repo_type=backend.alias)
+        response = api_call(self.app, params)
+        expected = {
+            "repo_name": "You do not have the permission to "
+                         "store repositories in the root location."}
+        assert_error(id_, expected, given=response.body)
+
     def test_api_create_repo_exists(self, backend):
         repo_name = backend.repo_name
         id_, params = build_data(
@@ -230,7 +294,9 b' class TestCreateRepo(object):'
             owner=TEST_USER_ADMIN_LOGIN,
             repo_type=backend.alias,)
         response = api_call(self.app, params)
-        expected = "repo `%s` already exist" % (repo_name,)
+        expected = {
+            'unique_repo_name': 'Repository with name `{}` already exists'.format(
+                repo_name)}
         assert_error(id_, expected, given=response.body)
 
     @mock.patch.object(RepoModel, 'create', crash)
@@ -245,26 +311,40 b' class TestCreateRepo(object):'
         expected = 'failed to create repository `%s`' % (repo_name,)
         assert_error(id_, expected, given=response.body)
 
-    def test_create_repo_with_extra_slashes_in_name(self, backend, user_util):
-        existing_repo_group = user_util.create_repo_group()
-        dirty_repo_name = '//{}/repo_name//'.format(
-            existing_repo_group.group_name)
-        cleaned_repo_name = '{}/repo_name'.format(
-            existing_repo_group.group_name)
+    @pytest.mark.parametrize('parent_group, dirty_name, expected_name', [
+        (None, 'foo bar x', 'foo-bar-x'),
+        ('foo', '/foo//bar x', 'foo/bar-x'),
+        ('foo-bar', 'foo-bar //bar x', 'foo-bar/bar-x'),
+    ])
+    def test_create_repo_with_extra_slashes_in_name(
+            self, backend, parent_group, dirty_name, expected_name):
+
+        if parent_group:
+            gr = fixture.create_repo_group(parent_group)
+            assert gr.group_name == parent_group
 
         id_, params = build_data(
             self.apikey, 'create_repo',
-            repo_name=dirty_repo_name,
+            repo_name=dirty_name,
             repo_type=backend.alias,
             owner=TEST_USER_ADMIN_LOGIN,)
         response = api_call(self.app, params)
-        repo = RepoModel().get_by_repo_name(cleaned_repo_name)
+        expected ={
+           "msg": "Created new repository `{}`".format(expected_name),
+           "task": None,
+           "success": True
+        }
+        assert_ok(id_, expected, response.body)
+
+        repo = RepoModel().get_by_repo_name(expected_name)
         assert repo is not None
 
         expected = {
-            'msg': 'Created new repository `%s`' % (cleaned_repo_name,),
+            'msg': 'Created new repository `%s`' % (expected_name,),
             'success': True,
             'task': None,
         }
         assert_ok(id_, expected, given=response.body)
-        fixture.destroy_repo(cleaned_repo_name)
+        fixture.destroy_repo(expected_name)
+        if parent_group:
+            fixture.destroy_repo_group(parent_group)

rhodecode/api/tests/test_create_repo_group.py

@@ -54,55 +54,10 b' class TestCreateRepoGroup(object):'
             'repo_group': repo_group.get_api_data()
         }
         expected = ret
-        assert_ok(id_, expected, given=response.body)
-        fixture.destroy_repo_group(repo_group_name)
-
-    def test_api_create_repo_group_regular_user(self):
-        repo_group_name = 'api-repo-group'
-
-        usr = UserModel().get_by_username(self.TEST_USER_LOGIN)
-        usr.inherit_default_permissions = False
-        Session().add(usr)
-        UserModel().grant_perm(
-            self.TEST_USER_LOGIN, 'hg.repogroup.create.true')
-        Session().commit()
-
-        repo_group = RepoGroupModel.cls.get_by_group_name(repo_group_name)
-        assert repo_group is None
-
-        id_, params = build_data(
-            self.apikey_regular, 'create_repo_group',
-            group_name=repo_group_name,
-            owner=TEST_USER_ADMIN_LOGIN,)
-        response = api_call(self.app, params)
-
-        repo_group = RepoGroupModel.cls.get_by_group_name(repo_group_name)
-        assert repo_group is not None
-        ret = {
-            'msg': 'Created new repo group `%s`' % (repo_group_name,),
-            'repo_group': repo_group.get_api_data()
-        }
-        expected = ret
-        assert_ok(id_, expected, given=response.body)
-        fixture.destroy_repo_group(repo_group_name)
-        UserModel().revoke_perm(
-            self.TEST_USER_LOGIN, 'hg.repogroup.create.true')
-        usr = UserModel().get_by_username(self.TEST_USER_LOGIN)
-        usr.inherit_default_permissions = True
-        Session().add(usr)
-        Session().commit()
-
-    def test_api_create_repo_group_regular_user_no_permission(self):
-        repo_group_name = 'api-repo-group'
-
-        id_, params = build_data(
-            self.apikey_regular, 'create_repo_group',
-            group_name=repo_group_name,
-            owner=TEST_USER_ADMIN_LOGIN,)
-        response = api_call(self.app, params)
-
-        expected = "Access was denied to this resource."
-        assert_error(id_, expected, given=response.body)
+        try:
+            assert_ok(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo_group(repo_group_name)
 
     def test_api_create_repo_group_in_another_group(self):
         repo_group_name = 'api-repo-group'
@@ -127,9 +82,11 b' class TestCreateRepoGroup(object):'
             'repo_group': repo_group.get_api_data()
         }
         expected = ret
-        assert_ok(id_, expected, given=response.body)
-        fixture.destroy_repo_group(full_repo_group_name)
-        fixture.destroy_repo_group(repo_group_name)
+        try:
+            assert_ok(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo_group(full_repo_group_name)
+            fixture.destroy_repo_group(repo_group_name)
 
     def test_api_create_repo_group_in_another_group_not_existing(self):
         repo_group_name = 'api-repo-group-no'
@@ -144,7 +101,10 b' class TestCreateRepoGroup(object):'
             owner=TEST_USER_ADMIN_LOGIN,
             copy_permissions=True)
         response = api_call(self.app, params)
-        expected = 'repository group `%s` does not exist' % (repo_group_name,)
+        expected = {
+            'repo_group':
+                'Parent repository group `{}` does not exist'.format(
+                    repo_group_name)}
         assert_error(id_, expected, given=response.body)
 
     def test_api_create_repo_group_that_exists(self):
@@ -159,9 +119,139 b' class TestCreateRepoGroup(object):'
             group_name=repo_group_name,
             owner=TEST_USER_ADMIN_LOGIN,)
         response = api_call(self.app, params)
-        expected = 'repo group `%s` already exist' % (repo_group_name,)
+        expected = {
+            'unique_repo_group_name':
+                'Repository group with name `{}` already exists'.format(
+                    repo_group_name)}
+        try:
+            assert_error(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo_group(repo_group_name)
+
+    def test_api_create_repo_group_regular_user_wit_root_location_perms(
+            self, user_util):
+        regular_user = user_util.create_user()
+        regular_user_api_key = regular_user.api_key
+
+        repo_group_name = 'api-repo-group-by-regular-user'
+
+        usr = UserModel().get_by_username(regular_user.username)
+        usr.inherit_default_permissions = False
+        Session().add(usr)
+
+        UserModel().grant_perm(
+            regular_user.username, 'hg.repogroup.create.true')
+        Session().commit()
+
+        repo_group = RepoGroupModel.cls.get_by_group_name(repo_group_name)
+        assert repo_group is None
+
+        id_, params = build_data(
+            regular_user_api_key, 'create_repo_group',
+            group_name=repo_group_name)
+        response = api_call(self.app, params)
+
+        repo_group = RepoGroupModel.cls.get_by_group_name(repo_group_name)
+        assert repo_group is not None
+        expected = {
+            'msg': 'Created new repo group `%s`' % (repo_group_name,),
+            'repo_group': repo_group.get_api_data()
+        }
+        try:
+            assert_ok(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo_group(repo_group_name)
+
+    def test_api_create_repo_group_regular_user_with_admin_perms_to_parent(
+            self, user_util):
+
+        repo_group_name = 'api-repo-group-parent'
+
+        repo_group = RepoGroupModel.cls.get_by_group_name(repo_group_name)
+        assert repo_group is None
+        # create the parent
+        fixture.create_repo_group(repo_group_name)
+
+        # user perms
+        regular_user = user_util.create_user()
+        regular_user_api_key = regular_user.api_key
+
+        usr = UserModel().get_by_username(regular_user.username)
+        usr.inherit_default_permissions = False
+        Session().add(usr)
+
+        RepoGroupModel().grant_user_permission(
+            repo_group_name, regular_user.username, 'group.admin')
+        Session().commit()
+
+        full_repo_group_name = repo_group_name + '/' + repo_group_name
+        id_, params = build_data(
+            regular_user_api_key, 'create_repo_group',
+            group_name=full_repo_group_name)
+        response = api_call(self.app, params)
+
+        repo_group = RepoGroupModel.cls.get_by_group_name(full_repo_group_name)
+        assert repo_group is not None
+        expected = {
+            'msg': 'Created new repo group `{}`'.format(full_repo_group_name),
+            'repo_group': repo_group.get_api_data()
+        }
+        try:
+            assert_ok(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo_group(full_repo_group_name)
+            fixture.destroy_repo_group(repo_group_name)
+
+    def test_api_create_repo_group_regular_user_no_permission_to_create_to_root_level(self):
+        repo_group_name = 'api-repo-group'
+
+        id_, params = build_data(
+            self.apikey_regular, 'create_repo_group',
+            group_name=repo_group_name)
+        response = api_call(self.app, params)
+
+        expected = {
+            'repo_group':
+                u'You do not have the permission to store '
+                u'repository groups in the root location.'}
         assert_error(id_, expected, given=response.body)
-        fixture.destroy_repo_group(repo_group_name)
+
+    def test_api_create_repo_group_regular_user_no_parent_group_perms(self):
+        repo_group_name = 'api-repo-group-regular-user'
+
+        repo_group = RepoGroupModel.cls.get_by_group_name(repo_group_name)
+        assert repo_group is None
+        # create the parent
+        fixture.create_repo_group(repo_group_name)
+
+        full_repo_group_name = repo_group_name+'/'+repo_group_name
+
+        id_, params = build_data(
+            self.apikey_regular, 'create_repo_group',
+            group_name=full_repo_group_name)
+        response = api_call(self.app, params)
+
+        expected = {
+            'repo_group':
+                'Parent repository group `{}` does not exist'.format(
+                    repo_group_name)}
+        try:
+            assert_error(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo_group(repo_group_name)
+
+    def test_api_create_repo_group_regular_user_no_permission_to_specify_owner(
+            self):
+        repo_group_name = 'api-repo-group'
+
+        id_, params = build_data(
+            self.apikey_regular, 'create_repo_group',
+            group_name=repo_group_name,
+            owner=TEST_USER_ADMIN_LOGIN,)
+        response = api_call(self.app, params)
+
+        expected = "Only RhodeCode super-admin can specify `owner` param"
+        assert_error(id_, expected, given=response.body)
 
     @mock.patch.object(RepoGroupModel, 'create', crash)
     def test_api_create_repo_group_exception_occurred(self):

rhodecode/api/tests/test_create_user.py

@@ -28,6 +28,7 b' from rhodecode.tests import ('
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_ok, assert_error, jsonify, crash)
 from rhodecode.tests.fixture import Fixture
+from rhodecode.model.db import RepoGroup
 
 
 # TODO: mikhail: remove fixture from here
@@ -145,6 +146,36 b' class TestCreateUser(object):'
         finally:
             fixture.destroy_user(usr.user_id)
 
+    def test_api_create_user_with_personal_repo_group(self):
+        username = 'test_new_api_user_personal_group'
+        email = username + "@foo.com"
+
+        id_, params = build_data(
+            self.apikey, 'create_user',
+            username=username,
+            email=email, extern_name='rhodecode',
+            create_personal_repo_group=True)
+        response = api_call(self.app, params)
+
+        usr = UserModel().get_by_username(username)
+        ret = {
+            'msg': 'created new user `%s`' % (username,),
+            'user': jsonify(usr.get_api_data(include_secrets=True)),
+        }
+
+        personal_group = RepoGroup.get_by_group_name(username)
+        assert personal_group
+        assert personal_group.personal == True
+        assert personal_group.user.username == username
+
+        try:
+            expected = ret
+            assert_ok(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo_group(username)
+            fixture.destroy_user(usr.user_id)
+
+
     @mock.patch.object(UserModel, 'create_or_update', crash)
     def test_api_create_user_when_exception_happened(self):
 

rhodecode/api/tests/test_fork_repo.py

@@ -24,6 +24,7 b' import pytest'
 
 from rhodecode.model.meta import Session
 from rhodecode.model.repo import RepoModel
+from rhodecode.model.repo_group import RepoGroupModel
 from rhodecode.model.user import UserModel
 from rhodecode.tests import TEST_USER_ADMIN_LOGIN
 from rhodecode.api.tests.utils import (
@@ -99,11 +100,35 b' class TestApiForkRepo(object):'
         finally:
             fixture.destroy_repo(fork_name)
 
+    def test_api_fork_repo_non_admin_into_group_no_permission(self, backend, user_util):
+        source_name = backend['minimal'].repo_name
+        repo_group = user_util.create_repo_group()
+        repo_group_name = repo_group.group_name
+        fork_name = '%s/api-repo-fork' % repo_group_name
+
+        id_, params = build_data(
+            self.apikey_regular, 'fork_repo',
+            repoid=source_name,
+            fork_name=fork_name)
+        response = api_call(self.app, params)
+
+        expected = {
+            'repo_group': 'Repository group `{}` does not exist'.format(
+                repo_group_name)}
+        try:
+            assert_error(id_, expected, given=response.body)
+        finally:
+            fixture.destroy_repo(fork_name)
+
     def test_api_fork_repo_non_admin_into_group(self, backend, user_util):
         source_name = backend['minimal'].repo_name
         repo_group = user_util.create_repo_group()
         fork_name = '%s/api-repo-fork' % repo_group.group_name
 
+        RepoGroupModel().grant_user_permission(
+            repo_group, self.TEST_USER_LOGIN, 'group.admin')
+        Session().commit()
+
         id_, params = build_data(
             self.apikey_regular, 'fork_repo',
             repoid=source_name,
@@ -129,10 +154,11 b' class TestApiForkRepo(object):'
             fork_name=fork_name,
             owner=TEST_USER_ADMIN_LOGIN)
         response = api_call(self.app, params)
-        expected = 'Only RhodeCode admin can specify `owner` param'
+        expected = 'Only RhodeCode super-admin can specify `owner` param'
         assert_error(id_, expected, given=response.body)
 
-    def test_api_fork_repo_non_admin_no_permission_to_fork(self, backend):
+    def test_api_fork_repo_non_admin_no_permission_of_source_repo(
+            self, backend):
         source_name = backend['minimal'].repo_name
         RepoModel().grant_user_permission(repo=source_name,
                                           user=self.TEST_USER_LOGIN,
@@ -147,19 +173,44 b' class TestApiForkRepo(object):'
         assert_error(id_, expected, given=response.body)
 
     def test_api_fork_repo_non_admin_no_permission_to_fork_to_root_level(
-            self, backend):
+            self, backend, user_util):
+
+        regular_user = user_util.create_user()
+        regular_user_api_key = regular_user.api_key
+        usr = UserModel().get_by_username(regular_user.username)
+        usr.inherit_default_permissions = False
+        Session().add(usr)
+        UserModel().grant_perm(regular_user.username, 'hg.fork.repository')
+
         source_name = backend['minimal'].repo_name
+        fork_name = backend.new_repo_name()
+        id_, params = build_data(
+            regular_user_api_key, 'fork_repo',
+            repoid=source_name,
+            fork_name=fork_name)
+        response = api_call(self.app, params)
+        expected = {
+            "repo_name": "You do not have the permission to "
+                         "store repositories in the root location."}
+        assert_error(id_, expected, given=response.body)
 
-        usr = UserModel().get_by_username(self.TEST_USER_LOGIN)
+    def test_api_fork_repo_non_admin_no_permission_to_fork(
+            self, backend, user_util):
+
+        regular_user = user_util.create_user()
+        regular_user_api_key = regular_user.api_key
+        usr = UserModel().get_by_username(regular_user.username)
         usr.inherit_default_permissions = False
         Session().add(usr)
 
+        source_name = backend['minimal'].repo_name
         fork_name = backend.new_repo_name()
         id_, params = build_data(
-            self.apikey_regular, 'fork_repo',
+            regular_user_api_key, 'fork_repo',
             repoid=source_name,
             fork_name=fork_name)
         response = api_call(self.app, params)
+
         expected = "Access was denied to this resource."
         assert_error(id_, expected, given=response.body)
 
@@ -189,7 +240,9 b' class TestApiForkRepo(object):'
         response = api_call(self.app, params)
 
         try:
-            expected = "fork `%s` already exist" % (fork_name,)
+            expected = {
+                'unique_repo_name': 'Repository with name `{}` already exists'.format(
+                    fork_name)}
             assert_error(id_, expected, given=response.body)
         finally:
             fixture.destroy_repo(fork_repo.repo_name)
@@ -205,7 +258,9 b' class TestApiForkRepo(object):'
             owner=TEST_USER_ADMIN_LOGIN)
         response = api_call(self.app, params)
 
-        expected = "repo `%s` already exist" % (fork_name,)
+        expected = {
+            'unique_repo_name': 'Repository with name `{}` already exists'.format(
+                fork_name)}
         assert_error(id_, expected, given=response.body)
 
     @mock.patch.object(RepoModel, 'create_fork', crash)

rhodecode/api/tests/test_get_pull_request.py

@@ -34,6 +34,7 b' pytestmark = pytest.mark.backends("git",'
 class TestGetPullRequest(object):
 
     def test_api_get_pull_request(self, pr_util):
+        from rhodecode.model.pull_request import PullRequestModel
         pull_request = pr_util.create_pull_request(mergeable=True)
         id_, params = build_data(
             self.apikey, 'get_pull_request',
@@ -57,6 +58,8 b' class TestGetPullRequest(object):'
         target_url = unicode(
             pull_request.target_repo.clone_url()
                 .with_netloc('test.example.com:80'))
+        shadow_url = unicode(
+            PullRequestModel().get_shadow_clone_url(pull_request))
         expected = {
             'pull_request_id': pull_request.pull_request_id,
             'url': pr_url,
@@ -89,15 +92,24 b' class TestGetPullRequest(object):'
                     'commit_id': pull_request.target_ref_parts.commit_id,
                 },
             },
+            'merge': {
+                'clone_url': shadow_url,
+                'reference': {
+                    'name': pull_request.shadow_merge_ref.name,
+                    'type': pull_request.shadow_merge_ref.type,
+                    'commit_id': pull_request.shadow_merge_ref.commit_id,
+                },
+            },
             'author': pull_request.author.get_api_data(include_secrets=False,
                                                        details='basic'),
             'reviewers': [
                 {
                     'user': reviewer.get_api_data(include_secrets=False,
                                                   details='basic'),
+                    'reasons': reasons,
                     'review_status': st[0][1].status if st else 'not_reviewed',
                 }
-                for reviewer, st in pull_request.reviewers_statuses()
+                for reviewer, reasons, st in pull_request.reviewers_statuses()
             ]
         }
         assert_ok(id_, expected, response.body)

rhodecode/api/tests/test_get_server_info.py

@@ -45,8 +45,13 b' class TestGetServerInfo(object):'
         expected['uptime'] = resp['result']['uptime']
         expected['load'] = resp['result']['load']
         expected['cpu'] = resp['result']['cpu']
-        expected['disk'] = resp['result']['disk']
-        expected['server_ip'] = '127.0.0.1:80'
+        expected['storage'] = resp['result']['storage']
+        expected['storage_temp'] = resp['result']['storage_temp']
+        expected['storage_inodes'] = resp['result']['storage_inodes']
+        expected['server'] = resp['result']['server']
+
+        expected['index_storage'] = resp['result']['index_storage']
+        expected['storage'] = resp['result']['storage']
 
         assert_ok(id_, expected, given=response.body)
 
@@ -59,7 +64,21 b' class TestGetServerInfo(object):'
         expected['uptime'] = resp['result']['uptime']
         expected['load'] = resp['result']['load']
         expected['cpu'] = resp['result']['cpu']
-        expected['disk'] = resp['result']['disk']
-        expected['server_ip'] = '127.0.0.1:80'
+        expected['storage'] = resp['result']['storage']
+        expected['storage_temp'] = resp['result']['storage_temp']
+        expected['storage_inodes'] = resp['result']['storage_inodes']
+        expected['server'] = resp['result']['server']
+
+        expected['index_storage'] = resp['result']['index_storage']
+        expected['storage'] = resp['result']['storage']
 
         assert_ok(id_, expected, given=response.body)
+
+    def test_api_get_server_info_data_for_search_index_build(self):
+        id_, params = build_data(self.apikey, 'get_server_info')
+        response = api_call(self.app, params)
+        resp = response.json
+
+        # required by indexer
+        assert resp['result']['index_storage']
+        assert resp['result']['storage']

rhodecode/api/tests/test_merge_pull_request.py

@@ -32,42 +32,37 b' from rhodecode.api.tests.utils import ('
 class TestMergePullRequest(object):
     @pytest.mark.backends("git", "hg")
     def test_api_merge_pull_request(self, pr_util, no_notifications):
-        pull_request = pr_util.create_pull_request()
-        pull_request_2 = PullRequestModel().create(
-            created_by=pull_request.author,
-            source_repo=pull_request.source_repo,
-            source_ref=pull_request.source_ref,
-            target_repo=pull_request.target_repo,
-            target_ref=pull_request.target_ref,
-            revisions=pull_request.revisions,
-            reviewers=(),
-            title=pull_request.title,
-            description=pull_request.description,
-        )
+        pull_request = pr_util.create_pull_request(mergeable=True)
         author = pull_request.user_id
-        repo = pull_request_2.target_repo.repo_id
-        pull_request_2_id = pull_request_2.pull_request_id
-        pull_request_2_repo = pull_request_2.target_repo.repo_name
-        Session().commit()
+        repo = pull_request.target_repo.repo_id
+        pull_request_id = pull_request.pull_request_id
+        pull_request_repo = pull_request.target_repo.repo_name
 
         id_, params = build_data(
             self.apikey, 'merge_pull_request',
-            repoid=pull_request_2_repo,
-            pullrequestid=pull_request_2_id)
+            repoid=pull_request_repo,
+            pullrequestid=pull_request_id)
+
         response = api_call(self.app, params)
 
+        # The above api call detaches the pull request DB object from the
+        # session because of an unconditional transaction rollback in our
+        # middleware. Therefore we need to add it back here if we want to use
+        # it.
+        Session().add(pull_request)
+
         expected = {
             'executed': True,
             'failure_reason': 0,
-            'possible': True
+            'possible': True,
+            'merge_commit_id': pull_request.shadow_merge_ref.commit_id,
+            'merge_ref': pull_request.shadow_merge_ref._asdict()
         }
 
         response_json = response.json['result']
-        assert response_json['merge_commit_id']
-        response_json.pop('merge_commit_id')
         assert response_json == expected
 
-        action = 'user_merged_pull_request:%d' % (pull_request_2_id, )
+        action = 'user_merged_pull_request:%d' % (pull_request_id, )
         journal = UserLog.query()\
             .filter(UserLog.user_id == author)\
             .filter(UserLog.repository_id == repo)\
@@ -77,11 +72,11 b' class TestMergePullRequest(object):'
 
         id_, params = build_data(
             self.apikey, 'merge_pull_request',
-            repoid=pull_request_2_repo, pullrequestid=pull_request_2_id)
+            repoid=pull_request_repo, pullrequestid=pull_request_id)
         response = api_call(self.app, params)
 
         expected = 'pull request `%s` merge failed, pull request is closed' % (
-            pull_request_2_id)
+            pull_request_id)
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.backends("git", "hg")

rhodecode/api/tests/test_update_repo.py

@@ -32,35 +32,60 b' fixture = Fixture()'
 
 UPDATE_REPO_NAME = 'api_update_me'
 
-class SAME_AS_UPDATES(object): """ Constant used for tests below """
+
+class SAME_AS_UPDATES(object):
+    """ Constant used for tests below """
+
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestApiUpdateRepo(object):
 
     @pytest.mark.parametrize("updates, expected", [
-        ({'owner': TEST_USER_REGULAR_LOGIN}, SAME_AS_UPDATES),
-        ({'description': 'new description'}, SAME_AS_UPDATES),
-        ({'clone_uri': 'http://foo.com/repo'}, SAME_AS_UPDATES),
-        ({'clone_uri': None}, {'clone_uri': ''}),
-        ({'clone_uri': ''}, {'clone_uri': ''}),
-        ({'landing_rev': 'branch:master'}, {'landing_rev': ['branch','master']}),
-        ({'enable_statistics': True}, SAME_AS_UPDATES),
-        ({'enable_locking': True}, SAME_AS_UPDATES),
-        ({'enable_downloads': True}, SAME_AS_UPDATES),
-        ({'name': 'new_repo_name'}, {
+        ({'owner': TEST_USER_REGULAR_LOGIN},
+         SAME_AS_UPDATES),
+
+        ({'description': 'new description'},
+         SAME_AS_UPDATES),
+
+        ({'clone_uri': 'http://foo.com/repo'},
+         SAME_AS_UPDATES),
+
+        ({'clone_uri': None},
+         {'clone_uri': ''}),
+
+        ({'clone_uri': ''},
+         {'clone_uri': ''}),
+
+        ({'landing_rev': 'rev:tip'},
+         {'landing_rev': ['rev', 'tip']}),
+
+        ({'enable_statistics': True},
+         SAME_AS_UPDATES),
+
+        ({'enable_locking': True},
+         SAME_AS_UPDATES),
+
+        ({'enable_downloads': True},
+         SAME_AS_UPDATES),
+
+        ({'repo_name': 'new_repo_name'},
+         {
             'repo_name': 'new_repo_name',
-            'url': 'http://test.example.com:80/new_repo_name',
-        }),
-        ({'group': 'test_group_for_update'}, {
-            'repo_name': 'test_group_for_update/%s' % UPDATE_REPO_NAME,
-            'url': 'http://test.example.com:80/test_group_for_update/%s' % UPDATE_REPO_NAME
-        }),
+            'url': 'http://test.example.com:80/new_repo_name'
+         }),
+
+        ({'repo_name': 'test_group_for_update/{}'.format(UPDATE_REPO_NAME),
+          '_group': 'test_group_for_update'},
+         {
+            'repo_name': 'test_group_for_update/{}'.format(UPDATE_REPO_NAME),
+            'url': 'http://test.example.com:80/test_group_for_update/{}'.format(UPDATE_REPO_NAME)
+         }),
     ])
     def test_api_update_repo(self, updates, expected, backend):
         repo_name = UPDATE_REPO_NAME
         repo = fixture.create_repo(repo_name, repo_type=backend.alias)
-        if updates.get('group'):
-            fixture.create_repo_group(updates['group'])
+        if updates.get('_group'):
+            fixture.create_repo_group(updates['_group'])
 
         expected_api_data = repo.get_api_data(include_secrets=True)
         if expected is SAME_AS_UPDATES:
@@ -68,15 +93,12 b' class TestApiUpdateRepo(object):'
         else:
             expected_api_data.update(expected)
 
-
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo_name, **updates)
         response = api_call(self.app, params)
 
-        if updates.get('name'):
-            repo_name = updates['name']
-        if updates.get('group'):
-            repo_name = '/'.join([updates['group'], repo_name])
+        if updates.get('repo_name'):
+            repo_name = updates['repo_name']
 
         try:
             expected = {
@@ -86,8 +108,8 b' class TestApiUpdateRepo(object):'
             assert_ok(id_, expected, given=response.body)
         finally:
             fixture.destroy_repo(repo_name)
-            if updates.get('group'):
-                fixture.destroy_repo_group(updates['group'])
+            if updates.get('_group'):
+                fixture.destroy_repo_group(updates['_group'])
 
     def test_api_update_repo_fork_of_field(self, backend):
         master_repo = backend.create_repo()
@@ -118,19 +140,23 b' class TestApiUpdateRepo(object):'
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo.repo_name, **updates)
         response = api_call(self.app, params)
-        expected = 'repository `{}` does not exist'.format(master_repo_name)
+        expected = {
+            'repo_fork_of': 'Fork with id `{}` does not exists'.format(
+                master_repo_name)}
         assert_error(id_, expected, given=response.body)
 
     def test_api_update_repo_with_repo_group_not_existing(self):
         repo_name = 'admin_owned'
+        fake_repo_group = 'test_group_for_update'
         fixture.create_repo(repo_name)
-        updates = {'group': 'test_group_for_update'}
+        updates = {'repo_name': '{}/{}'.format(fake_repo_group, repo_name)}
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo_name, **updates)
         response = api_call(self.app, params)
         try:
-            expected = 'repository group `%s` does not exist' % (
-                updates['group'],)
+            expected = {
+                'repo_group': 'Repository group `{}` does not exist'.format(fake_repo_group)
+            }
             assert_error(id_, expected, given=response.body)
         finally:
             fixture.destroy_repo(repo_name)

rhodecode/api/tests/test_update_repo_group.py

@@ -30,22 +30,30 b' from rhodecode.api.tests.utils import ('
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestApiUpdateRepoGroup(object):
+
     def test_update_group_name(self, user_util):
         new_group_name = 'new-group'
         initial_name = self._update(user_util, group_name=new_group_name)
         assert RepoGroupModel()._get_repo_group(initial_name) is None
-        assert RepoGroupModel()._get_repo_group(new_group_name) is not None
+        new_group = RepoGroupModel()._get_repo_group(new_group_name)
+        assert new_group is not None
+        assert new_group.full_path == new_group_name
 
-    def test_update_parent(self, user_util):
+    def test_update_group_name_change_parent(self, user_util):
+
         parent_group = user_util.create_repo_group()
-        initial_name = self._update(user_util, parent=parent_group.name)
+        parent_group_name = parent_group.name
 
-        expected_group_name = '{}/{}'.format(parent_group.name, initial_name)
+        expected_group_name = '{}/{}'.format(parent_group_name, 'new-group')
+        initial_name = self._update(user_util, group_name=expected_group_name)
+
         repo_group = RepoGroupModel()._get_repo_group(expected_group_name)
+
         assert repo_group is not None
         assert repo_group.group_name == expected_group_name
-        assert repo_group.name == initial_name
+        assert repo_group.full_path == expected_group_name
         assert RepoGroupModel()._get_repo_group(initial_name) is None
+
         new_path = os.path.join(
             RepoGroupModel().repos_path, *repo_group.full_path_splitted)
         assert os.path.exists(new_path)
@@ -67,15 +75,47 b' class TestApiUpdateRepoGroup(object):'
         repo_group = RepoGroupModel()._get_repo_group(initial_name)
         assert repo_group.user.username == owner
 
-    def test_api_update_repo_group_by_regular_user_no_permission(
-            self, backend):
-        repo = backend.create_repo()
-        repo_name = repo.repo_name
+    def test_update_group_name_conflict_with_existing(self, user_util):
+        group_1 = user_util.create_repo_group()
+        group_2 = user_util.create_repo_group()
+        repo_group_name_1 = group_1.group_name
+        repo_group_name_2 = group_2.group_name
 
         id_, params = build_data(
-            self.apikey_regular, 'update_repo_group', repogroupid=repo_name)
+            self.apikey, 'update_repo_group', repogroupid=repo_group_name_1,
+            group_name=repo_group_name_2)
+        response = api_call(self.app, params)
+        expected = {
+            'unique_repo_group_name':
+                'Repository group with name `{}` already exists'.format(
+                    repo_group_name_2)}
+        assert_error(id_, expected, given=response.body)
+
+    def test_api_update_repo_group_by_regular_user_no_permission(self, user_util):
+        temp_user = user_util.create_user()
+        temp_user_api_key = temp_user.api_key
+        parent_group = user_util.create_repo_group()
+        repo_group_name = parent_group.group_name
+        id_, params = build_data(
+            temp_user_api_key, 'update_repo_group', repogroupid=repo_group_name)
         response = api_call(self.app, params)
-        expected = 'repository group `%s` does not exist' % (repo_name,)
+        expected = 'repository group `%s` does not exist' % (repo_group_name,)
+        assert_error(id_, expected, given=response.body)
+
+    def test_api_update_repo_group_regular_user_no_root_write_permissions(
+            self, user_util):
+        temp_user = user_util.create_user()
+        temp_user_api_key = temp_user.api_key
+        parent_group = user_util.create_repo_group(owner=temp_user.username)
+        repo_group_name = parent_group.group_name
+
+        id_, params = build_data(
+            temp_user_api_key, 'update_repo_group', repogroupid=repo_group_name,
+            group_name='at-root-level')
+        response = api_call(self.app, params)
+        expected = {
+            'repo_group': 'You do not have the permission to store '
+                          'repository groups in the root location.'}
         assert_error(id_, expected, given=response.body)
 
     def _update(self, user_util, **kwargs):
@@ -89,7 +129,10 b' class TestApiUpdateRepoGroup(object):'
             self.apikey, 'update_repo_group', repogroupid=initial_name,
             **kwargs)
         response = api_call(self.app, params)
-        ret = {
+
+        repo_group = RepoGroupModel.cls.get(repo_group.group_id)
+
+        expected = {
             'msg': 'updated repository group ID:{} {}'.format(
                 repo_group.group_id, repo_group.group_name),
             'repo_group': {
@@ -103,5 +146,5 b' class TestApiUpdateRepoGroup(object):'
                     if repo_group.parent_group else None)
             }
         }
-        assert_ok(id_, ret, given=response.body)
+        assert_ok(id_, expected, given=response.body)
         return initial_name

rhodecode/api/tests/test_utils.py

@@ -249,7 +249,7 b' class TestRepoAccess(object):'
         fake_repo = Mock()
         with self.repo_perm_patch as rmock:
             rmock.return_value = repo_mock
-            assert utils.has_repo_permissions(
+            assert utils.validate_repo_permissions(
                 'fake_user', 'fake_repo_id', fake_repo,
                 ['perm1', 'perm2'])
             rmock.assert_called_once_with(*['perm1', 'perm2'])
@@ -263,6 +263,6 b' class TestRepoAccess(object):'
         with self.repo_perm_patch as rmock:
             rmock.return_value = repo_mock
             with pytest.raises(JSONRPCError) as excinfo:
-                utils.has_repo_permissions(
+                utils.validate_repo_permissions(
                     'fake_user', 'fake_repo_id', fake_repo, 'perms')
                 assert 'fake_repo_id' in excinfo

rhodecode/api/utils.py

@@ -26,7 +26,8 b' import collections'
 import logging
 
 from rhodecode.api.exc import JSONRPCError
-from rhodecode.lib.auth import HasPermissionAnyApi, HasRepoPermissionAnyApi
+from rhodecode.lib.auth import HasPermissionAnyApi, HasRepoPermissionAnyApi, \
+    HasRepoGroupPermissionAnyApi
 from rhodecode.lib.utils import safe_unicode
 from rhodecode.controllers.utils import get_commit_from_ref_name
 from rhodecode.lib.vcs.exceptions import RepositoryError
@@ -153,7 +154,7 b' def has_superadmin_permission(apiuser):'
     return False
 
 
-def has_repo_permissions(apiuser, repoid, repo, perms):
+def validate_repo_permissions(apiuser, repoid, repo, perms):
     """
     Raise JsonRPCError if apiuser is not authorized or return True
 
@@ -170,6 +171,36 b' def has_repo_permissions(apiuser, repoid'
     return True
 
 
+def validate_repo_group_permissions(apiuser, repogroupid, repo_group, perms):
+    """
+    Raise JsonRPCError if apiuser is not authorized or return True
+
+    :param apiuser:
+    :param repogroupid: just the id of repository group
+    :param repo_group: instance of repo_group
+    :param perms:
+    """
+    if not HasRepoGroupPermissionAnyApi(*perms)(
+            user=apiuser, group_name=repo_group.group_name):
+        raise JSONRPCError(
+            'repository group `%s` does not exist' % repogroupid)
+
+    return True
+
+
+def validate_set_owner_permissions(apiuser, owner):
+    if isinstance(owner, Optional):
+        owner = get_user_or_error(apiuser.user_id)
+    else:
+        if has_superadmin_permission(apiuser):
+            owner = get_user_or_error(owner)
+        else:
+            # forbid setting owner for non-admins
+            raise JSONRPCError(
+                'Only RhodeCode super-admin can specify `owner` param')
+    return owner
+
+
 def get_user_or_error(userid):
     """
     Get user by id or name or return JsonRPCError if not found

rhodecode/api/views/pull_request_api.py

@@ -25,7 +25,7 b' from rhodecode.api import jsonrpc_method'
 from rhodecode.api.utils import (
     has_superadmin_permission, Optional, OAttr, get_repo_or_error,
     get_pull_request_or_error, get_commit_or_error, get_user_or_error,
-    has_repo_permissions, resolve_ref_or_error)
+    validate_repo_permissions, resolve_ref_or_error)
 from rhodecode.lib.auth import (HasRepoPermissionAnyApi)
 from rhodecode.lib.base import vcs_operation_context
 from rhodecode.lib.utils2 import str2bool
@@ -96,6 +96,15 b' def get_pull_request(request, apiuser, r'
                                          "commit_id": "<commit_id>",
                                      }
                                  },
+            "merge":             {
+                                     "clone_url":   "<clone_url>",
+                                     "reference":
+                                     {
+                                         "name":      "<name>",
+                                         "type":      "<type>",
+                                         "commit_id": "<commit_id>",
+                                     }
+                                 },
            "author":             <user_obj>,
            "reviewers":          [
                                      ...
@@ -178,6 +187,15 b' def get_pull_requests(request, apiuser, '
                                              "commit_id": "<commit_id>",
                                          }
                                      },
+                "merge":             {
+                                         "clone_url":   "<clone_url>",
+                                         "reference":
+                                         {
+                                             "name":      "<name>",
+                                             "type":      "<type>",
+                                             "commit_id": "<commit_id>",
+                                         }
+                                     },
                "author":             <user_obj>,
                "reviewers":          [
                                          ...
@@ -197,7 +215,7 b' def get_pull_requests(request, apiuser, '
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     status = Optional.extract(status)
     pull_requests = PullRequestModel().get_all(repo, statuses=[status])
@@ -232,7 +250,12 b' def merge_pull_request(request, apiuser,'
             "executed":         "<bool>",
             "failure_reason":   "<int>",
             "merge_commit_id":  "<merge_commit_id>",
-            "possible":         "<bool>"
+            "possible":         "<bool>",
+            "merge_ref":        {
+                                    "commit_id": "<commit_id>",
+                                    "type":      "<type>",
+                                    "name":      "<name>"
+                                }
         },
       "error": null
 
@@ -260,13 +283,21 b' def merge_pull_request(request, apiuser,'
         request.environ, repo_name=target_repo.repo_name,
         username=apiuser.username, action='push',
         scm=target_repo.repo_type)
-    data = PullRequestModel().merge(pull_request, apiuser, extras=extras)
-    if data.executed:
+    merge_response = PullRequestModel().merge(
+        pull_request, apiuser, extras=extras)
+    if merge_response.executed:
         PullRequestModel().close_pull_request(
             pull_request.pull_request_id, apiuser)
 
         Session().commit()
-    return data
+
+    # In previous versions the merge response directly contained the merge
+    # commit id. It is now contained in the merge reference object. To be
+    # backwards compatible we have to extract it again.
+    merge_response = merge_response._asdict()
+    merge_response['merge_commit_id'] = merge_response['merge_ref'].commit_id
+
+    return merge_response
 
 
 @jsonrpc_method()
@@ -463,12 +494,17 b' def create_pull_request('
     :type description: Optional(str)
     :param reviewers: Set the new pull request reviewers list.
     :type reviewers: Optional(list)
+        Accepts username strings or objects of the format:
+        {
+            'username': 'nick', 'reasons': ['original author']
+        }
     """
+
     source = get_repo_or_error(source_repo)
     target = get_repo_or_error(target_repo)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin', 'repository.write', 'repository.read',)
-        has_repo_permissions(apiuser, source_repo, source, _perms)
+        validate_repo_permissions(apiuser, source_repo, source, _perms)
 
     full_source_ref = resolve_ref_or_error(source_ref, source)
     full_target_ref = resolve_ref_or_error(target_ref, target)
@@ -490,12 +526,21 b' def create_pull_request('
     if not ancestor:
         raise JSONRPCError('no common ancestor found')
 
-    reviewer_names = Optional.extract(reviewers) or []
-    if not isinstance(reviewer_names, list):
+    reviewer_objects = Optional.extract(reviewers) or []
+    if not isinstance(reviewer_objects, list):
         raise JSONRPCError('reviewers should be specified as a list')
 
-    reviewer_users = [get_user_or_error(n) for n in reviewer_names]
-    reviewer_ids = [u.user_id for u in reviewer_users]
+    reviewers_reasons = []
+    for reviewer_object in reviewer_objects:
+        reviewer_reasons = []
+        if isinstance(reviewer_object, (basestring, int)):
+            reviewer_username = reviewer_object
+        else:
+            reviewer_username = reviewer_object['username']
+            reviewer_reasons = reviewer_object.get('reasons', [])
+
+        user = get_user_or_error(reviewer_username)
+        reviewers_reasons.append((user.user_id, reviewer_reasons))
 
     pull_request_model = PullRequestModel()
     pull_request = pull_request_model.create(
@@ -506,7 +551,7 b' def create_pull_request('
         target_ref=full_target_ref,
         revisions=reversed(
             [commit.raw_id for commit in reversed(commit_ranges)]),
-        reviewers=reviewer_ids,
+        reviewers=reviewers_reasons,
         title=title,
         description=Optional.extract(description)
     )
@@ -585,12 +630,23 b' def update_pull_request('
             'pull request `%s` update failed, pull request is closed' % (
                 pullrequestid,))
 
-    reviewer_names = Optional.extract(reviewers) or []
-    if not isinstance(reviewer_names, list):
+    reviewer_objects = Optional.extract(reviewers) or []
+    if not isinstance(reviewer_objects, list):
         raise JSONRPCError('reviewers should be specified as a list')
 
-    reviewer_users = [get_user_or_error(n) for n in reviewer_names]
-    reviewer_ids = [u.user_id for u in reviewer_users]
+    reviewers_reasons = []
+    reviewer_ids = set()
+    for reviewer_object in reviewer_objects:
+        reviewer_reasons = []
+        if isinstance(reviewer_object, (int, basestring)):
+            reviewer_username = reviewer_object
+        else:
+            reviewer_username = reviewer_object['username']
+            reviewer_reasons = reviewer_object.get('reasons', [])
+
+        user = get_user_or_error(reviewer_username)
+        reviewer_ids.add(user.user_id)
+        reviewers_reasons.append((user.user_id, reviewer_reasons))
 
     title = Optional.extract(title)
     description = Optional.extract(description)
@@ -603,15 +659,15 b' def update_pull_request('
     commit_changes = {"added": [], "common": [], "removed": []}
     if str2bool(Optional.extract(update_commits)):
         if PullRequestModel().has_valid_update_type(pull_request):
-            _version, _commit_changes = PullRequestModel().update_commits(
+            update_response = PullRequestModel().update_commits(
                 pull_request)
-            commit_changes = _commit_changes or commit_changes
+            commit_changes = update_response.changes or commit_changes
         Session().commit()
 
     reviewers_changes = {"added": [], "removed": []}
     if reviewer_ids:
         added_reviewers, removed_reviewers = \
-            PullRequestModel().update_reviewers(pull_request, reviewer_ids)
+            PullRequestModel().update_reviewers(pull_request, reviewers_reasons)
 
         reviewers_changes['added'] = sorted(
             [get_user_or_error(n).username for n in added_reviewers])
@@ -631,5 +687,5 b' def update_pull_request('
         'updated_commits': commit_changes,
         'updated_reviewers': reviewers_changes
     }
+
     return data
-

rhodecode/api/views/repo_api.py

@@ -21,29 +21,26 b''
 import logging
 import time
 
-import colander
-
-from rhodecode import BACKENDS
-from rhodecode.api import jsonrpc_method, JSONRPCError, JSONRPCForbidden, json
+import rhodecode
+from rhodecode.api import (
+    jsonrpc_method, JSONRPCError, JSONRPCForbidden, JSONRPCValidationError)
 from rhodecode.api.utils import (
     has_superadmin_permission, Optional, OAttr, get_repo_or_error,
-    get_user_group_or_error, get_user_or_error, has_repo_permissions,
-    get_perm_or_error, store_update, get_repo_group_or_error, parse_args,
-    get_origin, build_commit_data)
-from rhodecode.lib.auth import (
-    HasPermissionAnyApi, HasRepoGroupPermissionAnyApi,
-    HasUserGroupPermissionAnyApi)
+    get_user_group_or_error, get_user_or_error, validate_repo_permissions,
+    get_perm_or_error, parse_args, get_origin, build_commit_data,
+    validate_set_owner_permissions)
+from rhodecode.lib.auth import HasPermissionAnyApi, HasUserGroupPermissionAnyApi
 from rhodecode.lib.exceptions import StatusChangeOnClosedPullRequestError
-from rhodecode.lib.utils import map_groups
 from rhodecode.lib.utils2 import str2bool, time_to_datetime
+from rhodecode.lib.ext_json import json
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import ChangesetCommentsModel
 from rhodecode.model.db import (
     Session, ChangesetStatus, RepositoryField, Repository)
 from rhodecode.model.repo import RepoModel
-from rhodecode.model.repo_group import RepoGroupModel
 from rhodecode.model.scm import ScmModel, RepoList
 from rhodecode.model.settings import SettingsModel, VcsSettingsModel
+from rhodecode.model import validation_schema
 from rhodecode.model.validation_schema.schemas import repo_schema
 
 log = logging.getLogger(__name__)
@@ -177,6 +174,7 b' def get_repo(request, apiuser, repoid, c'
 
     repo = get_repo_or_error(repoid)
     cache = Optional.extract(cache)
+
     include_secrets = False
     if has_superadmin_permission(apiuser):
         include_secrets = True
@@ -184,7 +182,7 b' def get_repo(request, apiuser, repoid, c'
         # check if we have at least read permission for this repo !
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     permissions = []
     for _user in repo.permissions():
@@ -292,7 +290,7 b' def get_repo_changeset(request, apiuser,'
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     changes_details = Optional.extract(details)
     _changes_details_types = ['basic', 'extended', 'full']
@@ -355,7 +353,7 b' def get_repo_changesets(request, apiuser'
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     changes_details = Optional.extract(details)
     _changes_details_types = ['basic', 'extended', 'full']
@@ -450,7 +448,7 b' def get_repo_nodes(request, apiuser, rep'
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     ret_type = Optional.extract(ret_type)
     details = Optional.extract(details)
@@ -523,7 +521,7 b' def get_repo_refs(request, apiuser, repo'
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin', 'repository.write', 'repository.read',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         # check if repo is not empty by any chance, skip quicker if it is.
@@ -538,26 +536,30 b' def get_repo_refs(request, apiuser, repo'
 
 
 @jsonrpc_method()
-def create_repo(request, apiuser, repo_name, repo_type,
-                owner=Optional(OAttr('apiuser')), description=Optional(''),
-                private=Optional(False), clone_uri=Optional(None),
-                landing_rev=Optional('rev:tip'),
-                enable_statistics=Optional(False),
-                enable_locking=Optional(False),
-                enable_downloads=Optional(False),
-                copy_permissions=Optional(False)):
+def create_repo(
+        request, apiuser, repo_name, repo_type,
+        owner=Optional(OAttr('apiuser')),
+        description=Optional(''),
+        private=Optional(False),
+        clone_uri=Optional(None),
+        landing_rev=Optional('rev:tip'),
+        enable_statistics=Optional(False),
+        enable_locking=Optional(False),
+        enable_downloads=Optional(False),
+        copy_permissions=Optional(False)):
     """
     Creates a repository.
 
-    * If the repository name contains "/", all the required repository
-      groups will be created.
+    * If the repository name contains "/", repository will be created inside
+      a repository group or nested repository groups
 
-      For example "foo/bar/baz" will create |repo| groups "foo" and "bar"
-      (with "foo" as parent). It will also create the "baz" repository
-      with "bar" as |repo| group.
+      For example "foo/bar/repo1" will create |repo| called "repo1" inside
+      group "foo/bar". You have to have permissions to access and write to
+      the last repository group ("bar" in this example)
 
     This command can only be run using an |authtoken| with at least
-    write permissions to the |repo|.
+    permissions to create repositories, or write permissions to
+    parent repository groups.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
@@ -569,9 +571,9 b' def create_repo(request, apiuser, repo_n'
     :type owner: Optional(str)
     :param description: Set the repository description.
     :type description: Optional(str)
-    :param private:
+    :param private: set repository as private
     :type private: bool
-    :param clone_uri:
+    :param clone_uri: set clone_uri
     :type clone_uri: str
     :param landing_rev: <rev_type>:<rev>
     :type landing_rev: str
@@ -606,53 +608,17 b' def create_repo(request, apiuser, repo_n'
       id : <id_given_in_input>
       result : null
       error :  {
-         'failed to create repository `<repo_name>`
+         'failed to create repository `<repo_name>`'
       }
 
     """
-    schema = repo_schema.RepoSchema()
-    try:
-        data = schema.deserialize({
-            'repo_name': repo_name
-        })
-    except colander.Invalid as e:
-        raise JSONRPCError("Validation failed: %s" % (e.asdict(),))
-    repo_name = data['repo_name']
 
-    (repo_name_cleaned,
-     parent_group_name) = RepoGroupModel()._get_group_name_and_parent(
-        repo_name)
-
-    if not HasPermissionAnyApi(
-            'hg.admin', 'hg.create.repository')(user=apiuser):
-        # check if we have admin permission for this repo group if given !
-
-        if parent_group_name:
-            repogroupid = parent_group_name
-            repo_group = get_repo_group_or_error(parent_group_name)
+    owner = validate_set_owner_permissions(apiuser, owner)
 
-            _perms = ('group.admin',)
-            if not HasRepoGroupPermissionAnyApi(*_perms)(
-                    user=apiuser, group_name=repo_group.group_name):
-                raise JSONRPCError(
-                    'repository group `%s` does not exist' % (
-                        repogroupid,))
-        else:
-            raise JSONRPCForbidden()
-
-    if not has_superadmin_permission(apiuser):
-        if not isinstance(owner, Optional):
-            # forbid setting owner for non-admins
-            raise JSONRPCError(
-                'Only RhodeCode admin can specify `owner` param')
-
-    if isinstance(owner, Optional):
-        owner = apiuser.user_id
-
-    owner = get_user_or_error(owner)
-
-    if RepoModel().get_by_repo_name(repo_name):
-        raise JSONRPCError("repo `%s` already exist" % repo_name)
+    description = Optional.extract(description)
+    copy_permissions = Optional.extract(copy_permissions)
+    clone_uri = Optional.extract(clone_uri)
+    landing_commit_ref = Optional.extract(landing_rev)
 
     defs = SettingsModel().get_default_repo_settings(strip_prefix=True)
     if isinstance(private, Optional):
@@ -666,32 +632,44 b' def create_repo(request, apiuser, repo_n'
     if isinstance(enable_downloads, Optional):
         enable_downloads = defs.get('repo_enable_downloads')
 
-    clone_uri = Optional.extract(clone_uri)
-    description = Optional.extract(description)
-    landing_rev = Optional.extract(landing_rev)
-    copy_permissions = Optional.extract(copy_permissions)
+    schema = repo_schema.RepoSchema().bind(
+        repo_type_options=rhodecode.BACKENDS.keys(),
+        # user caller
+        user=apiuser)
 
     try:
-        # create structure of groups and return the last group
-        repo_group = map_groups(repo_name)
+        schema_data = schema.deserialize(dict(
+            repo_name=repo_name,
+            repo_type=repo_type,
+            repo_owner=owner.username,
+            repo_description=description,
+            repo_landing_commit_ref=landing_commit_ref,
+            repo_clone_uri=clone_uri,
+            repo_private=private,
+            repo_copy_permissions=copy_permissions,
+            repo_enable_statistics=enable_statistics,
+            repo_enable_downloads=enable_downloads,
+            repo_enable_locking=enable_locking))
+    except validation_schema.Invalid as err:
+        raise JSONRPCValidationError(colander_exc=err)
+
+    try:
         data = {
-            'repo_name': repo_name_cleaned,
-            'repo_name_full': repo_name,
-            'repo_type': repo_type,
-            'repo_description': description,
             'owner': owner,
-            'repo_private': private,
-            'clone_uri': clone_uri,
-            'repo_group': repo_group.group_id if repo_group else None,
-            'repo_landing_rev': landing_rev,
-            'enable_statistics': enable_statistics,
-            'enable_locking': enable_locking,
-            'enable_downloads': enable_downloads,
-            'repo_copy_permissions': copy_permissions,
+            'repo_name': schema_data['repo_group']['repo_name_without_group'],
+            'repo_name_full': schema_data['repo_name'],
+            'repo_group': schema_data['repo_group']['repo_group_id'],
+            'repo_type': schema_data['repo_type'],
+            'repo_description': schema_data['repo_description'],
+            'repo_private': schema_data['repo_private'],
+            'clone_uri': schema_data['repo_clone_uri'],
+            'repo_landing_rev': schema_data['repo_landing_commit_ref'],
+            'enable_statistics': schema_data['repo_enable_statistics'],
+            'enable_locking': schema_data['repo_enable_locking'],
+            'enable_downloads': schema_data['repo_enable_downloads'],
+            'repo_copy_permissions': schema_data['repo_copy_permissions'],
         }
 
-        if repo_type not in BACKENDS.keys():
-            raise Exception("Invalid backend type %s" % repo_type)
         task = RepoModel().create(form_data=data, cur_user=owner)
         from celery.result import BaseAsyncResult
         task_id = None
@@ -699,17 +677,17 b' def create_repo(request, apiuser, repo_n'
             task_id = task.task_id
         # no commit, it's done in RepoModel, or async via celery
         return {
-            'msg': "Created new repository `%s`" % (repo_name,),
+            'msg': "Created new repository `%s`" % (schema_data['repo_name'],),
             'success': True,  # cannot return the repo data here since fork
-            # cann be done async
+            # can be done async
             'task': task_id
         }
     except Exception:
         log.exception(
             u"Exception while trying to create the repository %s",
-            repo_name)
+            schema_data['repo_name'])
         raise JSONRPCError(
-            'failed to create repository `%s`' % (repo_name,))
+            'failed to create repository `%s`' % (schema_data['repo_name'],))
 
 
 @jsonrpc_method()
@@ -735,7 +713,7 b' def add_field_to_repo(request, apiuser, '
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     label = Optional.extract(label) or key
     description = Optional.extract(description)
@@ -778,7 +756,7 b' def remove_field_from_repo(request, apiu'
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     field = RepositoryField.get_by_key_name(key, repo)
     if not field:
@@ -800,33 +778,38 b' def remove_field_from_repo(request, apiu'
 
 
 @jsonrpc_method()
-def update_repo(request, apiuser, repoid, name=Optional(None),
-                owner=Optional(OAttr('apiuser')),
-                group=Optional(None),
-                fork_of=Optional(None),
-                description=Optional(''), private=Optional(False),
-                clone_uri=Optional(None), landing_rev=Optional('rev:tip'),
-                enable_statistics=Optional(False),
-                enable_locking=Optional(False),
-                enable_downloads=Optional(False),
-                fields=Optional('')):
+def update_repo(
+        request, apiuser, repoid, repo_name=Optional(None),
+        owner=Optional(OAttr('apiuser')), description=Optional(''),
+        private=Optional(False), clone_uri=Optional(None),
+        landing_rev=Optional('rev:tip'), fork_of=Optional(None),
+        enable_statistics=Optional(False),
+        enable_locking=Optional(False),
+        enable_downloads=Optional(False), fields=Optional('')):
     """
     Updates a repository with the given information.
 
     This command can only be run using an |authtoken| with at least
-    write permissions to the |repo|.
+    admin permissions to the |repo|.
+
+    * If the repository name contains "/", repository will be updated
+      accordingly with a repository group or nested repository groups
+
+      For example repoid=repo-test name="foo/bar/repo-test" will update |repo|
+      called "repo-test" and place it inside group "foo/bar".
+      You have to have permissions to access and write to the last repository
+      group ("bar" in this example)
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: repository name or repository ID.
     :type repoid: str or int
-    :param name: Update the |repo| name.
-    :type name: str
+    :param repo_name: Update the |repo| name, including the
+        repository group it's in.
+    :type repo_name: str
     :param owner: Set the |repo| owner.
     :type owner: str
-    :param group: Set the |repo| group the |repo| belongs to.
-    :type group: str
-    :param fork_of: Set the master |repo| name.
+    :param fork_of: Set the |repo| as fork of another |repo|.
     :type fork_of: str
     :param description: Update the |repo| description.
     :type description: str
@@ -834,69 +817,115 b' def update_repo(request, apiuser, repoid'
     :type private: bool
     :param clone_uri: Update the |repo| clone URI.
     :type clone_uri: str
-    :param landing_rev: Set the |repo| landing revision. Default is
-        ``tip``.
+    :param landing_rev: Set the |repo| landing revision. Default is ``rev:tip``.
     :type landing_rev: str
-    :param enable_statistics: Enable statistics on the |repo|,
-        (True | False).
+    :param enable_statistics: Enable statistics on the |repo|, (True | False).
     :type enable_statistics: bool
     :param enable_locking: Enable |repo| locking.
     :type enable_locking: bool
-    :param enable_downloads: Enable downloads from the |repo|,
-        (True | False).
+    :param enable_downloads: Enable downloads from the |repo|, (True | False).
     :type enable_downloads: bool
     :param fields: Add extra fields to the |repo|. Use the following
         example format: ``field_key=field_val,field_key2=fieldval2``.
         Escape ', ' with \,
     :type fields: str
     """
+
     repo = get_repo_or_error(repoid)
+
     include_secrets = False
-    if has_superadmin_permission(apiuser):
+    if not has_superadmin_permission(apiuser):
+        validate_repo_permissions(apiuser, repoid, repo, ('repository.admin',))
+    else:
         include_secrets = True
-    else:
-        _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+
+    updates = dict(
+        repo_name=repo_name
+        if not isinstance(repo_name, Optional) else repo.repo_name,
+
+        fork_id=fork_of
+        if not isinstance(fork_of, Optional) else repo.fork.repo_name if repo.fork else None,
+
+        user=owner
+        if not isinstance(owner, Optional) else repo.user.username,
+
+        repo_description=description
+        if not isinstance(description, Optional) else repo.description,
+
+        repo_private=private
+        if not isinstance(private, Optional) else repo.private,
+
+        clone_uri=clone_uri
+        if not isinstance(clone_uri, Optional) else repo.clone_uri,
+
+        repo_landing_rev=landing_rev
+        if not isinstance(landing_rev, Optional) else repo._landing_revision,
+
+        repo_enable_statistics=enable_statistics
+        if not isinstance(enable_statistics, Optional) else repo.enable_statistics,
+
+        repo_enable_locking=enable_locking
+        if not isinstance(enable_locking, Optional) else repo.enable_locking,
+
+        repo_enable_downloads=enable_downloads
+        if not isinstance(enable_downloads, Optional) else repo.enable_downloads)
+
+    ref_choices, _labels = ScmModel().get_repo_landing_revs(repo=repo)
 
-    updates = {
-        # update function requires this.
-        'repo_name': repo.just_name
-    }
-    repo_group = group
-    if not isinstance(repo_group, Optional):
-        repo_group = get_repo_group_or_error(repo_group)
-        repo_group = repo_group.group_id
+    schema = repo_schema.RepoSchema().bind(
+        repo_type_options=rhodecode.BACKENDS.keys(),
+        repo_ref_options=ref_choices,
+        # user caller
+        user=apiuser,
+        old_values=repo.get_api_data())
+    try:
+        schema_data = schema.deserialize(dict(
+            # we save old value, users cannot change type
+            repo_type=repo.repo_type,
+
+            repo_name=updates['repo_name'],
+            repo_owner=updates['user'],
+            repo_description=updates['repo_description'],
+            repo_clone_uri=updates['clone_uri'],
+            repo_fork_of=updates['fork_id'],
+            repo_private=updates['repo_private'],
+            repo_landing_commit_ref=updates['repo_landing_rev'],
+            repo_enable_statistics=updates['repo_enable_statistics'],
+            repo_enable_downloads=updates['repo_enable_downloads'],
+            repo_enable_locking=updates['repo_enable_locking']))
+    except validation_schema.Invalid as err:
+        raise JSONRPCValidationError(colander_exc=err)
 
-    repo_fork_of = fork_of
-    if not isinstance(repo_fork_of, Optional):
-        repo_fork_of = get_repo_or_error(repo_fork_of)
-        repo_fork_of = repo_fork_of.repo_id
+    # save validated data back into the updates dict
+    validated_updates = dict(
+        repo_name=schema_data['repo_group']['repo_name_without_group'],
+        repo_group=schema_data['repo_group']['repo_group_id'],
+
+        user=schema_data['repo_owner'],
+        repo_description=schema_data['repo_description'],
+        repo_private=schema_data['repo_private'],
+        clone_uri=schema_data['repo_clone_uri'],
+        repo_landing_rev=schema_data['repo_landing_commit_ref'],
+        repo_enable_statistics=schema_data['repo_enable_statistics'],
+        repo_enable_locking=schema_data['repo_enable_locking'],
+        repo_enable_downloads=schema_data['repo_enable_downloads'],
+    )
+
+    if schema_data['repo_fork_of']:
+        fork_repo = get_repo_or_error(schema_data['repo_fork_of'])
+        validated_updates['fork_id'] = fork_repo.repo_id
+
+    # extra fields
+    fields = parse_args(Optional.extract(fields), key_prefix='ex_')
+    if fields:
+        validated_updates.update(fields)
 
     try:
-        store_update(updates, name, 'repo_name')
-        store_update(updates, repo_group, 'repo_group')
-        store_update(updates, repo_fork_of, 'fork_id')
-        store_update(updates, owner, 'user')
-        store_update(updates, description, 'repo_description')
-        store_update(updates, private, 'repo_private')
-        store_update(updates, clone_uri, 'clone_uri')
-        store_update(updates, landing_rev, 'repo_landing_rev')
-        store_update(updates, enable_statistics, 'repo_enable_statistics')
-        store_update(updates, enable_locking, 'repo_enable_locking')
-        store_update(updates, enable_downloads, 'repo_enable_downloads')
-
-        # extra fields
-        fields = parse_args(Optional.extract(fields), key_prefix='ex_')
-        if fields:
-            updates.update(fields)
-
-        RepoModel().update(repo, **updates)
+        RepoModel().update(repo, **validated_updates)
         Session().commit()
         return {
-            'msg': 'updated repo ID:%s %s' % (
-                repo.repo_id, repo.repo_name),
-            'repository': repo.get_api_data(
-                include_secrets=include_secrets)
+            'msg': 'updated repo ID:%s %s' % (repo.repo_id, repo.repo_name),
+            'repository': repo.get_api_data(include_secrets=include_secrets)
         }
     except Exception:
         log.exception(
@@ -908,26 +937,33 b' def update_repo(request, apiuser, repoid'
 @jsonrpc_method()
 def fork_repo(request, apiuser, repoid, fork_name,
               owner=Optional(OAttr('apiuser')),
-              description=Optional(''), copy_permissions=Optional(False),
-              private=Optional(False), landing_rev=Optional('rev:tip')):
+              description=Optional(''),
+              private=Optional(False),
+              clone_uri=Optional(None),
+              landing_rev=Optional('rev:tip'),
+              copy_permissions=Optional(False)):
     """
     Creates a fork of the specified |repo|.
 
-    * If using |RCE| with Celery this will immediately return a success
-      message, even though the fork will be created asynchronously.
+    * If the fork_name contains "/", fork will be created inside
+      a repository group or nested repository groups
 
-    This command can only be run using an |authtoken| with fork
-    permissions on the |repo|.
+      For example "foo/bar/fork-repo" will create fork called "fork-repo"
+      inside group "foo/bar". You have to have permissions to access and
+      write to the last repository group ("bar" in this example)
+
+    This command can only be run using an |authtoken| with minimum
+    read permissions of the forked repo, create fork permissions for an user.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set repository name or repository ID.
     :type repoid: str or int
-    :param fork_name: Set the fork name.
+    :param fork_name: Set the fork name, including it's repository group membership.
     :type fork_name: str
     :param owner: Set the fork owner.
     :type owner: str
-    :param description: Set the fork descripton.
+    :param description: Set the fork description.
     :type description: str
     :param copy_permissions: Copy permissions from parent |repo|. The
         default is False.
@@ -965,71 +1001,63 b' def fork_repo(request, apiuser, repoid, '
         error:  null
 
     """
-    if not has_superadmin_permission(apiuser):
-        if not HasPermissionAnyApi('hg.fork.repository')(user=apiuser):
-            raise JSONRPCForbidden()
 
     repo = get_repo_or_error(repoid)
     repo_name = repo.repo_name
 
-    (fork_name_cleaned,
-     parent_group_name) = RepoGroupModel()._get_group_name_and_parent(
-        fork_name)
-
     if not has_superadmin_permission(apiuser):
         # check if we have at least read permission for
         # this repo that we fork !
         _perms = (
             'repository.admin', 'repository.write', 'repository.read')
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
-        if not isinstance(owner, Optional):
-            # forbid setting owner for non super admins
-            raise JSONRPCError(
-                'Only RhodeCode admin can specify `owner` param'
-            )
-        # check if we have a create.repo permission if not maybe the parent
-        # group permission
-        if not HasPermissionAnyApi('hg.create.repository')(user=apiuser):
-            if parent_group_name:
-                repogroupid = parent_group_name
-                repo_group = get_repo_group_or_error(parent_group_name)
+        # check if the regular user has at least fork permissions as well
+        if not HasPermissionAnyApi('hg.fork.repository')(user=apiuser):
+            raise JSONRPCForbidden()
+
+    # check if user can set owner parameter
+    owner = validate_set_owner_permissions(apiuser, owner)
 
-                _perms = ('group.admin',)
-                if not HasRepoGroupPermissionAnyApi(*_perms)(
-                        user=apiuser, group_name=repo_group.group_name):
-                    raise JSONRPCError(
-                        'repository group `%s` does not exist' % (
-                            repogroupid,))
-            else:
-                raise JSONRPCForbidden()
+    description = Optional.extract(description)
+    copy_permissions = Optional.extract(copy_permissions)
+    clone_uri = Optional.extract(clone_uri)
+    landing_commit_ref = Optional.extract(landing_rev)
+    private = Optional.extract(private)
 
-    _repo = RepoModel().get_by_repo_name(fork_name)
-    if _repo:
-        type_ = 'fork' if _repo.fork else 'repo'
-        raise JSONRPCError("%s `%s` already exist" % (type_, fork_name))
-
-    if isinstance(owner, Optional):
-        owner = apiuser.user_id
-
-    owner = get_user_or_error(owner)
+    schema = repo_schema.RepoSchema().bind(
+        repo_type_options=rhodecode.BACKENDS.keys(),
+        # user caller
+        user=apiuser)
 
     try:
-        # create structure of groups and return the last group
-        repo_group = map_groups(fork_name)
-        form_data = {
-            'repo_name': fork_name_cleaned,
-            'repo_name_full': fork_name,
-            'repo_group': repo_group.group_id if repo_group else None,
-            'repo_type': repo.repo_type,
-            'description': Optional.extract(description),
-            'private': Optional.extract(private),
-            'copy_permissions': Optional.extract(copy_permissions),
-            'landing_rev': Optional.extract(landing_rev),
+        schema_data = schema.deserialize(dict(
+            repo_name=fork_name,
+            repo_type=repo.repo_type,
+            repo_owner=owner.username,
+            repo_description=description,
+            repo_landing_commit_ref=landing_commit_ref,
+            repo_clone_uri=clone_uri,
+            repo_private=private,
+            repo_copy_permissions=copy_permissions))
+    except validation_schema.Invalid as err:
+        raise JSONRPCValidationError(colander_exc=err)
+
+    try:
+        data = {
             'fork_parent_id': repo.repo_id,
+
+            'repo_name': schema_data['repo_group']['repo_name_without_group'],
+            'repo_name_full': schema_data['repo_name'],
+            'repo_group': schema_data['repo_group']['repo_group_id'],
+            'repo_type': schema_data['repo_type'],
+            'description': schema_data['repo_description'],
+            'private': schema_data['repo_private'],
+            'copy_permissions': schema_data['repo_copy_permissions'],
+            'landing_rev': schema_data['repo_landing_commit_ref'],
         }
 
-        task = RepoModel().create_fork(form_data, cur_user=owner)
+        task = RepoModel().create_fork(data, cur_user=owner)
         # no commit, it's done in RepoModel, or async via celery
         from celery.result import BaseAsyncResult
         task_id = None
@@ -1037,16 +1065,18 b' def fork_repo(request, apiuser, repoid, '
             task_id = task.task_id
         return {
             'msg': 'Created fork of `%s` as `%s`' % (
-                repo.repo_name, fork_name),
+                repo.repo_name, schema_data['repo_name']),
             'success': True,  # cannot return the repo data here since fork
             # can be done async
             'task': task_id
         }
     except Exception:
-        log.exception("Exception occurred while trying to fork a repo")
+        log.exception(
+            u"Exception while trying to create fork %s",
+            schema_data['repo_name'])
         raise JSONRPCError(
             'failed to fork repository `%s` as `%s`' % (
-                repo_name, fork_name))
+                repo_name, schema_data['repo_name']))
 
 
 @jsonrpc_method()
@@ -1082,7 +1112,7 b' def delete_repo(request, apiuser, repoid'
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         handle_forks = Optional.extract(forks)
@@ -1157,7 +1187,7 b' def invalidate_cache(request, apiuser, r'
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin', 'repository.write',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     delete = Optional.extract(delete_keys)
     try:
@@ -1228,7 +1258,7 b' def lock(request, apiuser, repoid, locke'
       id : <id_given_in_input>
       result : null
       error :  {
-        'Error occurred locking repository `<reponame>`
+        'Error occurred locking repository `<reponame>`'
       }
     """
 
@@ -1236,7 +1266,7 b' def lock(request, apiuser, repoid, locke'
     if not has_superadmin_permission(apiuser):
         # check if we have at least write permission for this repo !
         _perms = ('repository.admin', 'repository.write',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
         # make sure normal user does not pass someone else userid,
         # he is not allowed to do that
@@ -1347,7 +1377,7 b' def comment_commit('
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.read', 'repository.write', 'repository.admin')
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     if isinstance(userid, Optional):
         userid = apiuser.user_id
@@ -1438,7 +1468,7 b' def grant_user_permission(request, apius'
     perm = get_perm_or_error(perm)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
 
@@ -1492,7 +1522,7 b' def revoke_user_permission(request, apiu'
     user = get_user_or_error(userid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         RepoModel().revoke_user_permission(repo=repo, user=user)
@@ -1560,7 +1590,7 b' def grant_user_group_permission(request,'
     perm = get_perm_or_error(perm)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
@@ -1625,7 +1655,7 b' def revoke_user_group_permission(request'
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
@@ -1701,7 +1731,7 b' def pull(request, apiuser, repoid):'
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         ScmModel().pull_changes(repo.repo_name, apiuser.username)
@@ -1764,7 +1794,7 b' def strip(request, apiuser, repoid, revi'
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
-        has_repo_permissions(apiuser, repoid, repo, _perms)
+        validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         ScmModel().strip(repo, revision, branch)

rhodecode/api/views/repo_group_api.py

@@ -21,19 +21,18 b''
 
 import logging
 
-import colander
-
-from rhodecode.api import jsonrpc_method, JSONRPCError, JSONRPCForbidden
+from rhodecode.api import JSONRPCValidationError
+from rhodecode.api import jsonrpc_method, JSONRPCError
 from rhodecode.api.utils import (
     has_superadmin_permission, Optional, OAttr, get_user_or_error,
-    store_update, get_repo_group_or_error,
-    get_perm_or_error, get_user_group_or_error, get_origin)
+    get_repo_group_or_error, get_perm_or_error, get_user_group_or_error,
+    get_origin, validate_repo_group_permissions, validate_set_owner_permissions)
 from rhodecode.lib.auth import (
-    HasPermissionAnyApi, HasRepoGroupPermissionAnyApi,
-    HasUserGroupPermissionAnyApi)
-from rhodecode.model.db import Session, RepoGroup
+    HasRepoGroupPermissionAnyApi, HasUserGroupPermissionAnyApi)
+from rhodecode.model.db import Session
 from rhodecode.model.repo_group import RepoGroupModel
 from rhodecode.model.scm import RepoGroupList
+from rhodecode.model import validation_schema
 from rhodecode.model.validation_schema.schemas import repo_group_schema
 
 
@@ -142,21 +141,24 b' def get_repo_groups(request, apiuser):'
 
 
 @jsonrpc_method()
-def create_repo_group(request, apiuser, group_name, description=Optional(''),
-                      owner=Optional(OAttr('apiuser')),
-                      copy_permissions=Optional(False)):
+def create_repo_group(
+        request, apiuser, group_name,
+        owner=Optional(OAttr('apiuser')),
+        description=Optional(''),
+        copy_permissions=Optional(False)):
     """
     Creates a repository group.
 
-    * If the repository group name contains "/", all the required repository
-      groups will be created.
+    * If the repository group name contains "/", repository group will be
+      created inside a repository group or nested repository groups
 
-      For example "foo/bar/baz" will create |repo| groups "foo" and "bar"
-      (with "foo" as parent). It will also create the "baz" repository
-      with "bar" as |repo| group.
+      For example "foo/bar/group1" will create repository group called "group1"
+      inside group "foo/bar". You have to have permissions to access and
+      write to the last repository group ("bar" in this example)
 
-    This command can only be run using an |authtoken| with admin
-    permissions.
+    This command can only be run using an |authtoken| with at least
+    permissions to create repository groups, or admin permissions to
+    parent repository groups.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
@@ -193,72 +195,64 b' def create_repo_group(request, apiuser, '
 
     """
 
-    schema = repo_group_schema.RepoGroupSchema()
-    try:
-        data = schema.deserialize({
-            'group_name': group_name
-        })
-    except colander.Invalid as e:
-        raise JSONRPCError("Validation failed: %s" % (e.asdict(),))
-    group_name = data['group_name']
+    owner = validate_set_owner_permissions(apiuser, owner)
 
-    if isinstance(owner, Optional):
-        owner = apiuser.user_id
-
-    group_description = Optional.extract(description)
+    description = Optional.extract(description)
     copy_permissions = Optional.extract(copy_permissions)
 
-    # get by full name with parents, check if it already exist
-    if RepoGroup.get_by_group_name(group_name):
-        raise JSONRPCError("repo group `%s` already exist" % (group_name,))
-
-    (group_name_cleaned,
-     parent_group_name) = RepoGroupModel()._get_group_name_and_parent(
-        group_name)
+    schema = repo_group_schema.RepoGroupSchema().bind(
+        # user caller
+        user=apiuser)
 
-    parent_group = None
-    if parent_group_name:
-        parent_group = get_repo_group_or_error(parent_group_name)
+    try:
+        schema_data = schema.deserialize(dict(
+            repo_group_name=group_name,
+            repo_group_owner=owner.username,
+            repo_group_description=description,
+            repo_group_copy_permissions=copy_permissions,
+        ))
+    except validation_schema.Invalid as err:
+        raise JSONRPCValidationError(colander_exc=err)
 
-    if not HasPermissionAnyApi(
-            'hg.admin', 'hg.repogroup.create.true')(user=apiuser):
-        # check if we have admin permission for this parent repo group !
-        # users without admin or hg.repogroup.create can only create other
-        # groups in groups they own so this is a required, but can be empty
-        parent_group = getattr(parent_group, 'group_name', '')
-        _perms = ('group.admin',)
-        if not HasRepoGroupPermissionAnyApi(*_perms)(
-                user=apiuser, group_name=parent_group):
-            raise JSONRPCForbidden()
+    validated_group_name = schema_data['repo_group_name']
 
     try:
         repo_group = RepoGroupModel().create(
-            group_name=group_name,
-            group_description=group_description,
             owner=owner,
-            copy_permissions=copy_permissions)
+            group_name=validated_group_name,
+            group_description=schema_data['repo_group_name'],
+            copy_permissions=schema_data['repo_group_copy_permissions'])
         Session().commit()
         return {
-            'msg': 'Created new repo group `%s`' % group_name,
+            'msg': 'Created new repo group `%s`' % validated_group_name,
             'repo_group': repo_group.get_api_data()
         }
     except Exception:
         log.exception("Exception occurred while trying create repo group")
         raise JSONRPCError(
-            'failed to create repo group `%s`' % (group_name,))
+            'failed to create repo group `%s`' % (validated_group_name,))
 
 
 @jsonrpc_method()
 def update_repo_group(
         request, apiuser, repogroupid, group_name=Optional(''),
         description=Optional(''), owner=Optional(OAttr('apiuser')),
-        parent=Optional(None), enable_locking=Optional(False)):
+        enable_locking=Optional(False)):
     """
     Updates repository group with the details given.
 
     This command can only be run using an |authtoken| with admin
     permissions.
 
+    * If the group_name name contains "/", repository group will be updated
+      accordingly with a repository group or nested repository groups
+
+      For example repogroupid=group-test group_name="foo/bar/group-test"
+      will update repository group called "group-test" and place it
+      inside group "foo/bar".
+      You have to have permissions to access and write to the last repository
+      group ("bar" in this example)
+
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repogroupid: Set the ID of repository group.
@@ -269,29 +263,55 b' def update_repo_group('
     :type description: str
     :param owner: Set the |repo| group owner.
     :type owner: str
-    :param parent: Set the |repo| group parent.
-    :type parent: str or int
     :param enable_locking: Enable |repo| locking. The default is false.
     :type enable_locking: bool
     """
 
     repo_group = get_repo_group_or_error(repogroupid)
+
     if not has_superadmin_permission(apiuser):
-        # check if we have admin permission for this repo group !
-        _perms = ('group.admin',)
-        if not HasRepoGroupPermissionAnyApi(*_perms)(
-                user=apiuser, group_name=repo_group.group_name):
-            raise JSONRPCError(
-                'repository group `%s` does not exist' % (repogroupid,))
+        validate_repo_group_permissions(
+            apiuser, repogroupid, repo_group, ('group.admin',))
+
+    updates = dict(
+        group_name=group_name
+        if not isinstance(group_name, Optional) else repo_group.group_name,
+
+        group_description=description
+        if not isinstance(description, Optional) else repo_group.group_description,
+
+        user=owner
+        if not isinstance(owner, Optional) else repo_group.user.username,
+
+        enable_locking=enable_locking
+        if not isinstance(enable_locking, Optional) else repo_group.enable_locking
+    )
 
-    updates = {}
+    schema = repo_group_schema.RepoGroupSchema().bind(
+        # user caller
+        user=apiuser,
+        old_values=repo_group.get_api_data())
+
     try:
-        store_update(updates, group_name, 'group_name')
-        store_update(updates, description, 'group_description')
-        store_update(updates, owner, 'user')
-        store_update(updates, parent, 'group_parent_id')
-        store_update(updates, enable_locking, 'enable_locking')
-        repo_group = RepoGroupModel().update(repo_group, updates)
+        schema_data = schema.deserialize(dict(
+            repo_group_name=updates['group_name'],
+            repo_group_owner=updates['user'],
+            repo_group_description=updates['group_description'],
+            repo_group_enable_locking=updates['enable_locking'],
+        ))
+    except validation_schema.Invalid as err:
+        raise JSONRPCValidationError(colander_exc=err)
+
+    validated_updates = dict(
+        group_name=schema_data['repo_group']['repo_group_name_without_group'],
+        group_parent_id=schema_data['repo_group']['repo_group_id'],
+        user=schema_data['repo_group_owner'],
+        group_description=schema_data['repo_group_description'],
+        enable_locking=schema_data['repo_group_enable_locking'],
+    )
+
+    try:
+        RepoGroupModel().update(repo_group, validated_updates)
         Session().commit()
         return {
             'msg': 'updated repository group ID:%s %s' % (
@@ -299,7 +319,9 b' def update_repo_group('
             'repo_group': repo_group.get_api_data()
         }
     except Exception:
-        log.exception("Exception occurred while trying update repo group")
+        log.exception(
+            u"Exception occurred while trying update repo group %s",
+            repogroupid)
         raise JSONRPCError('failed to update repository group `%s`'
                            % (repogroupid,))
 
@@ -321,7 +343,7 b' def delete_repo_group(request, apiuser, '
 
       id : <id_given_in_input>
       result : {
-        'msg': 'deleted repo group ID:<repogroupid> <repogroupname>
+        'msg': 'deleted repo group ID:<repogroupid> <repogroupname>'
         'repo_group': null
       }
       error :  null
@@ -340,12 +362,9 b' def delete_repo_group(request, apiuser, '
 
     repo_group = get_repo_group_or_error(repogroupid)
     if not has_superadmin_permission(apiuser):
-        # check if we have admin permission for this repo group !
-        _perms = ('group.admin',)
-        if not HasRepoGroupPermissionAnyApi(*_perms)(
-                user=apiuser, group_name=repo_group.group_name):
-            raise JSONRPCError(
-                'repository group `%s` does not exist' % (repogroupid,))
+        validate_repo_group_permissions(
+            apiuser, repogroupid, repo_group, ('group.admin',))
+
     try:
         RepoGroupModel().delete(repo_group)
         Session().commit()
@@ -408,12 +427,8 b' def grant_user_permission_to_repo_group('
     repo_group = get_repo_group_or_error(repogroupid)
 
     if not has_superadmin_permission(apiuser):
-        # check if we have admin permission for this repo group !
-        _perms = ('group.admin',)
-        if not HasRepoGroupPermissionAnyApi(*_perms)(
-                user=apiuser, group_name=repo_group.group_name):
-            raise JSONRPCError(
-                'repository group `%s` does not exist' % (repogroupid,))
+        validate_repo_group_permissions(
+            apiuser, repogroupid, repo_group, ('group.admin',))
 
     user = get_user_or_error(userid)
     perm = get_perm_or_error(perm, prefix='group.')
@@ -487,12 +502,8 b' def revoke_user_permission_from_repo_gro'
     repo_group = get_repo_group_or_error(repogroupid)
 
     if not has_superadmin_permission(apiuser):
-        # check if we have admin permission for this repo group !
-        _perms = ('group.admin',)
-        if not HasRepoGroupPermissionAnyApi(*_perms)(
-                user=apiuser, group_name=repo_group.group_name):
-            raise JSONRPCError(
-                'repository group `%s` does not exist' % (repogroupid,))
+        validate_repo_group_permissions(
+            apiuser, repogroupid, repo_group, ('group.admin',))
 
     user = get_user_or_error(userid)
     apply_to_children = Optional.extract(apply_to_children)
@@ -569,12 +580,8 b' def grant_user_group_permission_to_repo_'
     perm = get_perm_or_error(perm, prefix='group.')
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
-        # check if we have admin permission for this repo group !
-        _perms = ('group.admin',)
-        if not HasRepoGroupPermissionAnyApi(*_perms)(
-                user=apiuser, group_name=repo_group.group_name):
-            raise JSONRPCError(
-                'repository group `%s` does not exist' % (repogroupid,))
+        validate_repo_group_permissions(
+            apiuser, repogroupid, repo_group, ('group.admin',))
 
         # check if we have at least read permission for this user group !
         _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)
@@ -656,12 +663,8 b' def revoke_user_group_permission_from_re'
     repo_group = get_repo_group_or_error(repogroupid)
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
-        # check if we have admin permission for this repo group !
-        _perms = ('group.admin',)
-        if not HasRepoGroupPermissionAnyApi(*_perms)(
-                user=apiuser, group_name=repo_group.group_name):
-            raise JSONRPCError(
-                'repository group `%s` does not exist' % (repogroupid,))
+        validate_repo_group_permissions(
+            apiuser, repogroupid, repo_group, ('group.admin',))
 
         # check if we have at least read permission for this user group !
         _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)

rhodecode/api/views/server_api.py

@@ -60,7 +60,13 b' def get_server_info(request, apiuser):'
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
-    return ScmModel().get_server_info(request.environ)
+    server_info = ScmModel().get_server_info(request.environ)
+    # rhodecode-index requires those
+
+    server_info['index_storage'] = server_info['search']['value']['location']
+    server_info['storage'] = server_info['storage']['value']['path']
+
+    return server_info
 
 
 @jsonrpc_method()

rhodecode/api/views/user_api.py

@@ -25,7 +25,7 b' from rhodecode.api.utils import ('
     Optional, OAttr, has_superadmin_permission, get_user_or_error, store_update)
 from rhodecode.lib.auth import AuthUser, PasswordGenerator
 from rhodecode.lib.exceptions import DefaultUserException
-from rhodecode.lib.utils2 import safe_int
+from rhodecode.lib.utils2 import safe_int, str2bool
 from rhodecode.model.db import Session, User, Repository
 from rhodecode.model.user import UserModel
 
@@ -81,6 +81,7 b' def get_user(request, apiuser, userid=Op'
                 "usergroup.read",
                 "hg.repogroup.create.false",
                 "hg.create.none",
+                "hg.password_reset.enabled",
                 "hg.extern_activate.manual",
                 "hg.create.write_on_repogroup.false",
                 "hg.usergroup.create.false",
@@ -154,7 +155,8 b' def create_user(request, apiuser, userna'
                 active=Optional(True), admin=Optional(False),
                 extern_name=Optional('rhodecode'),
                 extern_type=Optional('rhodecode'),
-                force_password_change=Optional(False)):
+                force_password_change=Optional(False),
+                create_personal_repo_group=Optional(None)):
     """
     Creates a new user and returns the new user object.
 
@@ -187,7 +189,8 b' def create_user(request, apiuser, userna'
     :param force_password_change: Force the new user to change password
         on next login.
     :type force_password_change: Optional(``True`` | ``False``)
-
+    :param create_personal_repo_group: Create personal repo group for this user
+    :type create_personal_repo_group: Optional(``True`` | ``False``)
     Example output:
 
     .. code-block:: bash
@@ -229,6 +232,9 b' def create_user(request, apiuser, userna'
                 Optional.extract(extern_name) != 'rhodecode'):
         # generate temporary password if user is external
         password = PasswordGenerator().gen_password(length=16)
+    create_repo_group = Optional.extract(create_personal_repo_group)
+    if isinstance(create_repo_group, basestring):
+        create_repo_group = str2bool(create_repo_group)
 
     try:
         user = UserModel().create_or_update(
@@ -242,6 +248,7 b' def create_user(request, apiuser, userna'
             extern_type=Optional.extract(extern_type),
             extern_name=Optional.extract(extern_name),
             force_password_change=Optional.extract(force_password_change),
+            create_repo_group=create_repo_group
         )
         Session().commit()
         return {

rhodecode/authentication/base.py

@@ -226,6 +226,15 b' class RhodeCodeAuthPluginBase(object):'
         """
         raise NotImplementedError("Not implemented in base class")
 
+    def get_url_slug(self):
+        """
+        Returns a slug which should be used when constructing URLs which refer
+        to this plugin. By default it returns the plugin name. If the name is
+        not suitable for using it in an URL the plugin should override this
+        method.
+        """
+        return self.name
+
     @property
     def is_headers_auth(self):
         """

rhodecode/authentication/routes.py

@@ -45,7 +45,7 b' class AuthnPluginResourceBase(AuthnResou'
 
     def __init__(self, plugin):
         self.plugin = plugin
-        self.__name__ = plugin.name
+        self.__name__ = plugin.get_url_slug()
         self.display_name = plugin.get_display_name()
 
 

rhodecode/authentication/views.py

@@ -84,7 +84,7 b' class AuthnPluginViewBase(object):'
 
         try:
             valid_data = schema.deserialize(data)
-        except colander.Invalid, e:
+        except colander.Invalid as e:
             # Display error message and display form again.
             self.request.session.flash(
                 _('Errors exist when saving plugin settings. '

rhodecode/config/middleware.py

@@ -31,9 +31,9 b' from pyramid.authorization import ACLAut'
 from pyramid.config import Configurator
 from pyramid.settings import asbool, aslist
 from pyramid.wsgi import wsgiapp
-from pyramid.httpexceptions import HTTPError, HTTPInternalServerError, HTTPFound
+from pyramid.httpexceptions import (
+    HTTPError, HTTPInternalServerError, HTTPFound)
 from pyramid.events import ApplicationCreated
-import pyramid.httpexceptions as httpexceptions
 from pyramid.renderers import render_to_response
 from routes.middleware import RoutesMiddleware
 import routes.util
@@ -44,10 +44,10 b' from rhodecode.config import patches'
 from rhodecode.config.routing import STATIC_FILE_PREFIX
 from rhodecode.config.environment import (
     load_environment, load_pyramid_environment)
-from rhodecode.lib.exceptions import VCSServerUnavailable
-from rhodecode.lib.vcs.exceptions import VCSCommunicationError
 from rhodecode.lib.middleware import csrf
 from rhodecode.lib.middleware.appenlight import wrap_in_appenlight_if_enabled
+from rhodecode.lib.middleware.error_handling import (
+    PylonsErrorHandlingMiddleware)
 from rhodecode.lib.middleware.https_fixup import HttpsFixup
 from rhodecode.lib.middleware.vcs import VCSMiddleware
 from rhodecode.lib.plugins.utils import register_rhodecode_plugin
@@ -186,53 +186,27 b' def make_not_found_view(config):'
     pylons_app, appenlight_client = wrap_in_appenlight_if_enabled(
         pylons_app, settings)
 
-    # The VCSMiddleware shall operate like a fallback if pyramid doesn't find
-    # a view to handle the request. Therefore we wrap it around the pylons app.
+    # The pylons app is executed inside of the pyramid 404 exception handler.
+    # Exceptions which are raised inside of it are not handled by pyramid
+    # again. Therefore we add a middleware that invokes the error handler in
+    # case of an exception or error response. This way we return proper error
+    # HTML pages in case of an error.
+    reraise = (settings.get('debugtoolbar.enabled', False) or
+               rhodecode.disable_error_handler)
+    pylons_app = PylonsErrorHandlingMiddleware(
+        pylons_app, error_handler, reraise)
+
+    # The VCSMiddleware shall operate like a fallback if pyramid doesn't find a
+    # view to handle the request. Therefore it is wrapped around the pylons
+    # app. It has to be outside of the error handling otherwise error responses
+    # from the vcsserver are converted to HTML error pages. This confuses the
+    # command line tools and the user won't get a meaningful error message.
     if vcs_server_enabled:
         pylons_app = VCSMiddleware(
             pylons_app, settings, appenlight_client, registry=config.registry)
 
-    pylons_app_as_view = wsgiapp(pylons_app)
-
-    def pylons_app_with_error_handler(context, request):
-        """
-        Handle exceptions from rc pylons app:
-
-        - old webob type exceptions get converted to pyramid exceptions
-        - pyramid exceptions are passed to the error handler view
-        """
-        def is_vcs_response(response):
-            return 'X-RhodeCode-Backend' in response.headers
-
-        def is_http_error(response):
-            # webob type error responses
-            return (400 <= response.status_int <= 599)
-
-        def is_error_handling_needed(response):
-            return is_http_error(response) and not is_vcs_response(response)
-
-        try:
-            response = pylons_app_as_view(context, request)
-            if is_error_handling_needed(response):
-                response = webob_to_pyramid_http_response(response)
-                return error_handler(response, request)
-        except HTTPError as e:  # pyramid type exceptions
-            return error_handler(e, request)
-        except Exception as e:
-            log.exception(e)
-
-            if (settings.get('debugtoolbar.enabled', False) or
-                rhodecode.disable_error_handler):
-                raise
-
-            if isinstance(e, VCSCommunicationError):
-                return error_handler(VCSServerUnavailable(), request)
-
-            return error_handler(HTTPInternalServerError(), request)
-
-        return response
-
-    return pylons_app_with_error_handler
+    # Convert WSGI app to pyramid view and return it.
+    return wsgiapp(pylons_app)
 
 
 def add_pylons_compat_data(registry, global_config, settings):
@@ -243,16 +217,6 b' def add_pylons_compat_data(registry, glo'
     registry._pylons_compat_settings = settings
 
 
-def webob_to_pyramid_http_response(webob_response):
-    ResponseClass = httpexceptions.status_map[webob_response.status_int]
-    pyramid_response = ResponseClass(webob_response.status)
-    pyramid_response.status = webob_response.status
-    pyramid_response.headers.update(webob_response.headers)
-    if pyramid_response.headers['content-type'] == 'text/html':
-        pyramid_response.headers['content-type'] = 'text/html; charset=UTF-8'
-    return pyramid_response
-
-
 def error_handler(exception, request):
     from rhodecode.model.settings import SettingsModel
     from rhodecode.lib.utils2 import AttributeDict
@@ -466,10 +430,11 b' def _sanitize_vcs_settings(settings):'
     """
     _string_setting(settings, 'vcs.svn.compatible_version', '')
     _string_setting(settings, 'git_rev_filter', '--all')
-    _string_setting(settings, 'vcs.hooks.protocol', 'pyro4')
+    _string_setting(settings, 'vcs.hooks.protocol', 'http')
+    _string_setting(settings, 'vcs.scm_app_implementation', 'http')
     _string_setting(settings, 'vcs.server', '')
     _string_setting(settings, 'vcs.server.log_level', 'debug')
-    _string_setting(settings, 'vcs.server.protocol', 'pyro4')
+    _string_setting(settings, 'vcs.server.protocol', 'http')
     _bool_setting(settings, 'startup.import_repos', 'false')
     _bool_setting(settings, 'vcs.hooks.direct_calls', 'false')
     _bool_setting(settings, 'vcs.server.enable', 'true')
@@ -477,6 +442,13 b' def _sanitize_vcs_settings(settings):'
     _list_setting(settings, 'vcs.backends', 'hg, git, svn')
     _int_setting(settings, 'vcs.connection_timeout', 3600)
 
+    # Support legacy values of vcs.scm_app_implementation. Legacy
+    # configurations may use 'rhodecode.lib.middleware.utils.scm_app_http'
+    # which is now mapped to 'http'.
+    scm_app_impl = settings['vcs.scm_app_implementation']
+    if scm_app_impl == 'rhodecode.lib.middleware.utils.scm_app_http':
+        settings['vcs.scm_app_implementation'] = 'http'
+
 
 def _int_setting(settings, name, default):
     settings[name] = int(settings.get(name, default))
@@ -501,5 +473,8 b' def _list_setting(settings, name, defaul'
         settings[name] = aslist(raw_value)
 
 
-def _string_setting(settings, name, default):
-    settings[name] = settings.get(name, default).lower()
+def _string_setting(settings, name, default, lower=True):
+    value = settings.get(name, default)
+    if lower:
+        value = value.lower()
+    settings[name] = value

rhodecode/config/routing.py

@@ -196,7 +196,7 b' def make_map(config):'
     rmap.connect('user_autocomplete_data', '/_users', controller='home',
                  action='user_autocomplete_data', jsroute=True)
     rmap.connect('user_group_autocomplete_data', '/_user_groups', controller='home',
-                 action='user_group_autocomplete_data')
+                 action='user_group_autocomplete_data', jsroute=True)
 
     rmap.connect(
         'user_profile', '/_profiles/{username}', controller='users',
@@ -305,7 +305,7 b' def make_map(config):'
         m.connect('delete_user', '/users/{user_id}',
                   action='delete', conditions={'method': ['DELETE']})
         m.connect('edit_user', '/users/{user_id}/edit',
-                  action='edit', conditions={'method': ['GET']})
+                  action='edit', conditions={'method': ['GET']}, jsroute=True)
         m.connect('user', '/users/{user_id}',
                   action='show', conditions={'method': ['GET']})
         m.connect('force_password_reset_user', '/users/{user_id}/password_reset',
@@ -389,7 +389,7 b' def make_map(config):'
 
         m.connect('edit_user_group_members',
                   '/user_groups/{user_group_id}/edit/members', jsroute=True,
-                  action='edit_members', conditions={'method': ['GET']})
+                  action='user_group_members', conditions={'method': ['GET']})
 
     # ADMIN PERMISSIONS ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
@@ -699,6 +699,9 b' def make_map(config):'
     rmap.connect('repo_refs_changelog_data', '/{repo_name}/refs-data-changelog',
                  controller='summary', action='repo_refs_changelog_data',
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
+    rmap.connect('repo_default_reviewers_data', '/{repo_name}/default-reviewers',
+                 controller='summary', action='repo_default_reviewers_data',
+                 jsroute=True, requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changeset_home', '/{repo_name}/changeset/{revision}',
                  controller='changeset', revision='tip', jsroute=True,
@@ -824,6 +827,10 b' def make_map(config):'
                  controller='admin/repos', action='repo_delete_svn_pattern',
                  conditions={'method': ['DELETE'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
+    rmap.connect('repo_pullrequest_settings', '/{repo_name}/settings/pullrequest',
+                 controller='admin/repos', action='repo_settings_pullrequest',
+                 conditions={'method': ['GET', 'POST'], 'function': check_repo},
+                 requirements=URL_NAME_REQUIREMENTS)
 
     # still working url for backward compat.
     rmap.connect('raw_changeset_home_depraced',

rhodecode/controllers/admin/gists.py

@@ -28,10 +28,9 b' import logging'
 
 import formencode
 import peppercorn
-from formencode import htmlfill
 
 from pylons import request, response, tmpl_context as c, url
-from pylons.controllers.util import abort, redirect
+from pylons.controllers.util import redirect
 from pylons.i18n.translation import _
 from webob.exc import HTTPNotFound, HTTPForbidden
 from sqlalchemy.sql.expression import or_
@@ -45,7 +44,7 b' from rhodecode.lib import helpers as h'
 from rhodecode.lib.base import BaseController, render
 from rhodecode.lib.auth import LoginRequired, NotAnonymous
 from rhodecode.lib.utils import jsonify
-from rhodecode.lib.utils2 import safe_str, safe_int, time_to_datetime
+from rhodecode.lib.utils2 import time_to_datetime
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.vcs.exceptions import VCSError, NodeNotChangedError
 from rhodecode.model import validation_schema

rhodecode/controllers/admin/my_account.py

@@ -39,19 +39,20 b' from rhodecode.lib.auth import ('
     LoginRequired, NotAnonymous, AuthUser, generate_auth_token)
 from rhodecode.lib.base import BaseController, render
 from rhodecode.lib.utils import jsonify
-from rhodecode.lib.utils2 import safe_int, md5
+from rhodecode.lib.utils2 import safe_int, md5, str2bool
 from rhodecode.lib.ext_json import json
 
 from rhodecode.model.validation_schema.schemas import user_schema
 from rhodecode.model.db import (
-    Repository, PullRequest, PullRequestReviewers, UserEmailMap, User,
-    UserFollowing)
+    Repository, PullRequest, UserEmailMap, User, UserFollowing)
 from rhodecode.model.forms import UserForm
 from rhodecode.model.scm import RepoList
 from rhodecode.model.user import UserModel
 from rhodecode.model.repo import RepoModel
 from rhodecode.model.auth_token import AuthTokenModel
 from rhodecode.model.meta import Session
+from rhodecode.model.pull_request import PullRequestModel
+from rhodecode.model.comment import ChangesetCommentsModel
 
 log = logging.getLogger(__name__)
 
@@ -289,25 +290,85 b' class MyAccountController(BaseController'
                 category='success')
         return redirect(url('my_account_emails'))
 
+    def _extract_ordering(self, request):
+        column_index = safe_int(request.GET.get('order[0][column]'))
+        order_dir = request.GET.get('order[0][dir]', 'desc')
+        order_by = request.GET.get(
+            'columns[%s][data][sort]' % column_index, 'name_raw')
+        return order_by, order_dir
+
+    def _get_pull_requests_list(self, statuses):
+        start = safe_int(request.GET.get('start'), 0)
+        length = safe_int(request.GET.get('length'), c.visual.dashboard_items)
+        order_by, order_dir = self._extract_ordering(request)
+
+        pull_requests = PullRequestModel().get_im_participating_in(
+            user_id=c.rhodecode_user.user_id,
+            statuses=statuses,
+            offset=start, length=length, order_by=order_by,
+            order_dir=order_dir)
+
+        pull_requests_total_count = PullRequestModel().count_im_participating_in(
+            user_id=c.rhodecode_user.user_id, statuses=statuses)
+
+        from rhodecode.lib.utils import PartialRenderer
+        _render = PartialRenderer('data_table/_dt_elements.html')
+        data = []
+        for pr in pull_requests:
+            repo_id = pr.target_repo_id
+            comments = ChangesetCommentsModel().get_all_comments(
+                repo_id, pull_request=pr)
+            owned = pr.user_id == c.rhodecode_user.user_id
+            status = pr.calculated_review_status()
+
+            data.append({
+                'target_repo': _render('pullrequest_target_repo',
+                                       pr.target_repo.repo_name),
+                'name': _render('pullrequest_name',
+                                pr.pull_request_id, pr.target_repo.repo_name,
+                                short=True),
+                'name_raw': pr.pull_request_id,
+                'status': _render('pullrequest_status', status),
+                'title': _render(
+                    'pullrequest_title', pr.title, pr.description),
+                'description': h.escape(pr.description),
+                'updated_on': _render('pullrequest_updated_on',
+                                      h.datetime_to_time(pr.updated_on)),
+                'updated_on_raw': h.datetime_to_time(pr.updated_on),
+                'created_on': _render('pullrequest_updated_on',
+                                      h.datetime_to_time(pr.created_on)),
+                'created_on_raw': h.datetime_to_time(pr.created_on),
+                'author': _render('pullrequest_author',
+                                  pr.author.full_contact, ),
+                'author_raw': pr.author.full_name,
+                'comments': _render('pullrequest_comments', len(comments)),
+                'comments_raw': len(comments),
+                'closed': pr.is_closed(),
+                'owned': owned
+            })
+        # json used to render the grid
+        data = ({
+            'data': data,
+            'recordsTotal': pull_requests_total_count,
+            'recordsFiltered': pull_requests_total_count,
+        })
+        return data
+
     def my_account_pullrequests(self):
         c.active = 'pullrequests'
         self.__load_data()
-        c.show_closed = request.GET.get('pr_show_closed')
-
-        def _filter(pr):
-            s = sorted(pr, key=lambda o: o.created_on, reverse=True)
-            if not c.show_closed:
-                s = filter(lambda p: p.status != PullRequest.STATUS_CLOSED, s)
-            return s
+        c.show_closed = str2bool(request.GET.get('pr_show_closed'))
 
-        c.my_pull_requests = _filter(
-            PullRequest.query().filter(
-                PullRequest.user_id == c.rhodecode_user.user_id).all())
-        my_prs = [
-            x.pull_request for x in PullRequestReviewers.query().filter(
-                PullRequestReviewers.user_id == c.rhodecode_user.user_id).all()]
-        c.participate_in_pull_requests = _filter(my_prs)
-        return render('admin/my_account/my_account.html')
+        statuses = [PullRequest.STATUS_NEW, PullRequest.STATUS_OPEN]
+        if c.show_closed:
+            statuses += [PullRequest.STATUS_CLOSED]
+        data = self._get_pull_requests_list(statuses)
+        if not request.is_xhr:
+            c.data_participate = json.dumps(data['data'])
+            c.records_total_participate = data['recordsTotal']
+            return render('admin/my_account/my_account.html')
+        else:
+            return json.dumps(data)
 
     def my_account_auth_tokens(self):
         c.active = 'auth_tokens'

rhodecode/controllers/admin/permissions.py

@@ -57,7 +57,7 b' class PermissionsController(BaseControll'
         super(PermissionsController, self).__before__()
 
     def __load_data(self):
-        PermissionModel().set_global_permission_choices(c, translator=_)
+        PermissionModel().set_global_permission_choices(c, gettext_translator=_)
 
     @HasPermissionAllDecorator('hg.admin')
     def permission_application(self):
@@ -92,6 +92,7 b' class PermissionsController(BaseControll'
         self.__load_data()
         _form = ApplicationPermissionsForm(
             [x[0] for x in c.register_choices],
+            [x[0] for x in c.password_reset_choices],
             [x[0] for x in c.extern_activate_choices])()
 
         try:

rhodecode/controllers/admin/repos.py

@@ -160,6 +160,7 b' class ReposController(BaseRepoController'
         self.__load_defaults()
         form_result = {}
         task_id = None
+        c.personal_repo_group = c.rhodecode_user.personal_repo_group
         try:
             # CanWriteToGroup validators checks permissions of this POST
             form_result = RepoForm(repo_groups=c.repo_groups_choices,
@@ -173,8 +174,6 b' class ReposController(BaseRepoController'
             if isinstance(task, BaseAsyncResult):
                 task_id = task.task_id
         except formencode.Invalid as errors:
-            c.personal_repo_group = RepoGroup.get_by_group_name(
-                c.rhodecode_user.username)
             return htmlfill.render(
                 render('admin/repos/repo_add.html'),
                 defaults=errors.value,
@@ -215,7 +214,7 b' class ReposController(BaseRepoController'
         c.repo_groups = RepoGroup.groups_choices(groups=acl_groups)
         c.repo_groups_choices = map(lambda k: unicode(k[0]), c.repo_groups)
         choices, c.landing_revs = ScmModel().get_repo_landing_revs()
-        c.personal_repo_group = RepoGroup.get_by_group_name(c.rhodecode_user.username)
+        c.personal_repo_group = c.rhodecode_user.personal_repo_group
         c.new_repo = repo_name_slug(new_repo)
 
         ## apply the defaults from defaults page
@@ -299,9 +298,8 b' class ReposController(BaseRepoController'
         repo_model = RepoModel()
         changed_name = repo_name
 
+        c.personal_repo_group = c.rhodecode_user.personal_repo_group
         # override the choices with extracted revisions !
-        c.personal_repo_group = RepoGroup.get_by_group_name(
-            c.rhodecode_user.username)
         repo = Repository.get_by_repo_name(repo_name)
         old_data = {
             'repo_name': repo_name,
@@ -399,8 +397,7 b' class ReposController(BaseRepoController'
 
         c.repo_fields = RepositoryField.query()\
             .filter(RepositoryField.repository == c.repo_info).all()
-        c.personal_repo_group = RepoGroup.get_by_group_name(
-            c.rhodecode_user.username)
+        c.personal_repo_group = c.rhodecode_user.personal_repo_group
         c.active = 'settings'
         return htmlfill.render(
             render('admin/repos/repo_edit.html'),

rhodecode/controllers/admin/settings.py

@@ -34,6 +34,7 b' import packaging.version'
 from pylons import request, tmpl_context as c, url, config
 from pylons.controllers.util import redirect
 from pylons.i18n.translation import _, lazy_ugettext
+from pyramid.threadlocal import get_current_registry
 from webob.exc import HTTPBadRequest
 
 import rhodecode
@@ -54,6 +55,7 b' from rhodecode.model.db import RhodeCode'
 from rhodecode.model.forms import ApplicationSettingsForm, \
     ApplicationUiSettingsForm, ApplicationVisualisationForm, \
     LabsSettingsForm, IssueTrackerPatternsForm
+from rhodecode.model.repo_group import RepoGroupModel
 
 from rhodecode.model.scm import ScmModel
 from rhodecode.model.notification import EmailNotificationModel
@@ -63,6 +65,7 b' from rhodecode.model.settings import ('
     SettingsModel)
 
 from rhodecode.model.supervisor import SupervisorModel, SUPERVISOR_MASTER
+from rhodecode.svn_support.config_keys import generate_config
 
 
 log = logging.getLogger(__name__)
@@ -134,6 +137,10 b' class SettingsController(BaseController)'
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
 
+        # TODO: Replace with request.registry after migrating to pyramid.
+        pyramid_settings = get_current_registry().settings
+        c.svn_proxy_generate_config = pyramid_settings[generate_config]
+
         application_form = ApplicationUiSettingsForm()()
 
         try:
@@ -186,6 +193,10 b' class SettingsController(BaseController)'
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
 
+        # TODO: Replace with request.registry after migrating to pyramid.
+        pyramid_settings = get_current_registry().settings
+        c.svn_proxy_generate_config = pyramid_settings[generate_config]
+
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
@@ -235,6 +246,8 b' class SettingsController(BaseController)'
         """POST /admin/settings/global: All items in the collection"""
         # url('admin_settings_global')
         c.active = 'global'
+        c.personal_repo_group_default_pattern = RepoGroupModel()\
+            .get_personal_group_name_pattern()
         application_form = ApplicationSettingsForm()()
         try:
             form_result = application_form.to_python(dict(request.POST))
@@ -249,16 +262,18 b' class SettingsController(BaseController)'
 
         try:
             settings = [
-                ('title', 'rhodecode_title'),
-                ('realm', 'rhodecode_realm'),
-                ('pre_code', 'rhodecode_pre_code'),
-                ('post_code', 'rhodecode_post_code'),
-                ('captcha_public_key', 'rhodecode_captcha_public_key'),
-                ('captcha_private_key', 'rhodecode_captcha_private_key'),
+                ('title', 'rhodecode_title', 'unicode'),
+                ('realm', 'rhodecode_realm', 'unicode'),
+                ('pre_code', 'rhodecode_pre_code', 'unicode'),
+                ('post_code', 'rhodecode_post_code', 'unicode'),
+                ('captcha_public_key', 'rhodecode_captcha_public_key', 'unicode'),
+                ('captcha_private_key', 'rhodecode_captcha_private_key', 'unicode'),
+                ('create_personal_repo_group', 'rhodecode_create_personal_repo_group', 'bool'),
+                ('personal_repo_group_pattern', 'rhodecode_personal_repo_group_pattern', 'unicode'),
             ]
-            for setting, form_key in settings:
+            for setting, form_key, type_ in settings:
                 sett = SettingsModel().create_or_update_setting(
-                    setting, form_result[form_key])
+                    setting, form_result[form_key], type_)
                 Session().add(sett)
 
             Session().commit()
@@ -277,6 +292,8 b' class SettingsController(BaseController)'
         """GET /admin/settings/global: All items in the collection"""
         # url('admin_settings_global')
         c.active = 'global'
+        c.personal_repo_group_default_pattern = RepoGroupModel()\
+            .get_personal_group_name_pattern()
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
@@ -397,19 +414,20 b' class SettingsController(BaseController)'
         settings_model = IssueTrackerSettingsModel()
 
         form = IssueTrackerPatternsForm()().to_python(request.POST)
-        for uid in form['delete_patterns']:
-            settings_model.delete_entries(uid)
+        if form:
+            for uid in form.get('delete_patterns', []):
+                settings_model.delete_entries(uid)
 
-        for pattern in form['patterns']:
-            for setting, value, type_ in pattern:
-                sett = settings_model.create_or_update_setting(
-                    setting, value, type_)
-                Session().add(sett)
+            for pattern in form.get('patterns', []):
+                for setting, value, type_ in pattern:
+                    sett = settings_model.create_or_update_setting(
+                        setting, value, type_)
+                    Session().add(sett)
 
-            Session().commit()
+                Session().commit()
 
-        SettingsModel().invalidate_settings_cache()
-        h.flash(_('Updated issue tracker entries'), category='success')
+            SettingsModel().invalidate_settings_cache()
+            h.flash(_('Updated issue tracker entries'), category='success')
         return redirect(url('admin_settings_issuetracker'))
 
     @HasPermissionAllDecorator('hg.admin')
@@ -530,65 +548,93 b' class SettingsController(BaseController)'
         """GET /admin/settings/system: All items in the collection"""
         # url('admin_settings_system')
         snapshot = str2bool(request.GET.get('snapshot'))
-        c.active = 'system'
+        defaults = self._form_defaults()
 
-        defaults = self._form_defaults()
-        c.rhodecode_ini = rhodecode.CONFIG
+        c.active = 'system'
         c.rhodecode_update_url = defaults.get('rhodecode_update_url')
         server_info = ScmModel().get_server_info(request.environ)
+
         for key, val in server_info.iteritems():
             setattr(c, key, val)
 
-        if c.disk['percent'] > 90:
-            h.flash(h.literal(_(
-                'Critical: your disk space is very low <b>%s%%</b> used' %
-                c.disk['percent'])), 'error')
-        elif c.disk['percent'] > 70:
-            h.flash(h.literal(_(
-                'Warning: your disk space is running low <b>%s%%</b> used' %
-                c.disk['percent'])), 'warning')
+        def val(name, subkey='human_value'):
+            return server_info[name][subkey]
+
+        def state(name):
+            return server_info[name]['state']
+
+        def val2(name):
+            val = server_info[name]['human_value']
+            state = server_info[name]['state']
+            return val, state
 
-        try:
-            c.uptime_age = h._age(
-                h.time_to_datetime(c.boot_time), False, show_suffix=False)
-        except TypeError:
-            c.uptime_age = c.boot_time
+        c.data_items = [
+            # update info
+            (_('Update info'), h.literal(
+                '<span class="link" id="check_for_update" >%s.</span>' % (
+                _('Check for updates')) +
+                '<br/> <span >%s.</span>' % (_('Note: please make sure this server can access `%s` for the update link to work') % c.rhodecode_update_url)
+            ), ''),
+
+            # RhodeCode specific
+            (_('RhodeCode Version'), val('rhodecode_app')['text'], state('rhodecode_app')),
+            (_('RhodeCode Server IP'), val('server')['server_ip'], state('server')),
+            (_('RhodeCode Server ID'), val('server')['server_id'], state('server')),
+            (_('RhodeCode Configuration'), val('rhodecode_config')['path'], state('rhodecode_config')),
+            ('', '', ''),  # spacer
+
+            # Database
+            (_('Database'), val('database')['url'], state('database')),
+            (_('Database version'), val('database')['version'], state('database')),
+            ('', '', ''),  # spacer
 
-        try:
-            c.system_memory = '%s/%s, %s%% (%s%%) used%s' % (
-                h.format_byte_size_binary(c.memory['used']),
-                h.format_byte_size_binary(c.memory['total']),
-                c.memory['percent2'],
-                c.memory['percent'],
-                ' %s' % c.memory['error'] if 'error' in c.memory else '')
-        except TypeError:
-            c.system_memory = 'NOT AVAILABLE'
+            # Platform/Python
+            (_('Platform'), val('platform')['name'], state('platform')),
+            (_('Platform UUID'), val('platform')['uuid'], state('platform')),
+            (_('Python version'), val('python')['version'], state('python')),
+            (_('Python path'), val('python')['executable'], state('python')),
+            ('', '', ''),  # spacer
+
+            # Systems stats
+            (_('CPU'), val('cpu'), state('cpu')),
+            (_('Load'), val('load')['text'], state('load')),
+            (_('Memory'), val('memory')['text'], state('memory')),
+            (_('Uptime'), val('uptime')['text'], state('uptime')),
+            ('', '', ''),  # spacer
+
+            # Repo storage
+            (_('Storage location'), val('storage')['path'], state('storage')),
+            (_('Storage info'), val('storage')['text'], state('storage')),
+            (_('Storage inodes'), val('storage_inodes')['text'], state('storage_inodes')),
 
-        rhodecode_ini_safe = rhodecode.CONFIG.copy()
-        blacklist = [
-            'rhodecode_license_key',
-            'routes.map',
-            'pylons.h',
-            'pylons.app_globals',
-            'pylons.environ_config',
-            'sqlalchemy.db1.url',
-            ('app_conf', 'sqlalchemy.db1.url')
+            (_('Gist storage location'), val('storage_gist')['path'], state('storage_gist')),
+            (_('Gist storage info'), val('storage_gist')['text'], state('storage_gist')),
+
+            (_('Archive cache storage location'), val('storage_archive')['path'], state('storage_archive')),
+            (_('Archive cache info'), val('storage_archive')['text'], state('storage_archive')),
+
+            (_('Temp storage location'), val('storage_temp')['path'], state('storage_temp')),
+            (_('Temp storage info'), val('storage_temp')['text'], state('storage_temp')),
+
+            (_('Search info'), val('search')['text'], state('search')),
+            (_('Search location'), val('search')['location'], state('search')),
+            ('', '', ''),  # spacer
+
+            # VCS specific
+            (_('VCS Backends'), val('vcs_backends'), state('vcs_backends')),
+            (_('VCS Server'), val('vcs_server')['text'], state('vcs_server')),
+            (_('GIT'), val('git'), state('git')),
+            (_('HG'), val('hg'), state('hg')),
+            (_('SVN'), val('svn'), state('svn')),
+
         ]
-        for k in blacklist:
-            if isinstance(k, tuple):
-                section, key = k
-                if section in rhodecode_ini_safe:
-                    rhodecode_ini_safe[section].pop(key, None)
-            else:
-                rhodecode_ini_safe.pop(k, None)
-
-        c.rhodecode_ini_safe = rhodecode_ini_safe
 
         # TODO: marcink, figure out how to allow only selected users to do this
-        c.allowed_to_snapshot = False
+        c.allowed_to_snapshot = c.rhodecode_user.admin
 
         if snapshot:
             if c.allowed_to_snapshot:
+                c.data_items.pop(0)  # remove server info
                 return render('admin/settings/settings_system_snapshot.html')
             else:
                 h.flash('You are not allowed to do this', category='warning')

rhodecode/controllers/admin/user_groups.py

@@ -25,6 +25,7 b' User Groups crud controller for pylons'
 import logging
 import formencode
 
+import peppercorn
 from formencode import htmlfill
 from pylons import request, tmpl_context as c, url, config
 from pylons.controllers.util import redirect
@@ -40,7 +41,7 b' from rhodecode.lib.utils import jsonify,'
 from rhodecode.lib.utils2 import safe_unicode, str2bool, safe_int
 from rhodecode.lib.auth import (
     LoginRequired, NotAnonymous, HasUserGroupPermissionAnyDecorator,
-    HasPermissionAnyDecorator)
+    HasPermissionAnyDecorator, XHRRequired)
 from rhodecode.lib.base import BaseController, render
 from rhodecode.model.permission import PermissionModel
 from rhodecode.model.scm import UserGroupList
@@ -64,18 +65,13 b' class UserGroupsController(BaseControlle'
     def __before__(self):
         super(UserGroupsController, self).__before__()
         c.available_permissions = config['available_permissions']
-        PermissionModel().set_global_permission_choices(c, translator=_)
+        PermissionModel().set_global_permission_choices(c, gettext_translator=_)
 
     def __load_data(self, user_group_id):
         c.group_members_obj = [x.user for x in c.user_group.members]
         c.group_members_obj.sort(key=lambda u: u.username.lower())
-
         c.group_members = [(x.user_id, x.username) for x in c.group_members_obj]
 
-        c.available_members = [(x.user_id, x.username)
-                               for x in User.query().all()]
-        c.available_members.sort(key=lambda u: u[1].lower())
-
     def __load_defaults(self, user_group_id):
         """
         Load defaults settings for edit, and update
@@ -207,20 +203,21 b' class UserGroupsController(BaseControlle'
         c.active = 'settings'
         self.__load_data(user_group_id)
 
-        available_members = [safe_unicode(x[0]) for x in c.available_members]
-
         users_group_form = UserGroupForm(
-            edit=True, old_data=c.user_group.get_dict(),
-            available_members=available_members, allow_disabled=True)()
+            edit=True, old_data=c.user_group.get_dict(), allow_disabled=True)()
 
         try:
             form_result = users_group_form.to_python(request.POST)
+            pstruct = peppercorn.parse(request.POST.items())
+            form_result['users_group_members'] = pstruct['user_group_members']
+
             UserGroupModel().update(c.user_group, form_result)
-            gr = form_result['users_group_name']
+            updated_user_group = form_result['users_group_name']
             action_logger(c.rhodecode_user,
-                          'admin_updated_users_group:%s' % gr,
+                          'admin_updated_users_group:%s' % updated_user_group,
                           None, self.ip_addr, self.sa)
-            h.flash(_('Updated user group %s') % gr, category='success')
+            h.flash(_('Updated user group %s') % updated_user_group,
+                    category='success')
             Session().commit()
         except formencode.Invalid as errors:
             defaults = errors.value
@@ -462,19 +459,29 b' class UserGroupsController(BaseControlle'
         return render('admin/user_groups/user_group_edit.html')
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
-    def edit_members(self, user_group_id):
+    @XHRRequired()
+    @jsonify
+    def user_group_members(self, user_group_id):
         user_group_id = safe_int(user_group_id)
-        c.user_group = UserGroup.get_or_404(user_group_id)
-        c.active = 'members'
-        c.group_members_obj = sorted((x.user for x in c.user_group.members),
-                                     key=lambda u: u.username.lower())
+        user_group = UserGroup.get_or_404(user_group_id)
+        group_members_obj = sorted((x.user for x in user_group.members),
+                                   key=lambda u: u.username.lower())
 
-        group_members = [(x.user_id, x.username) for x in c.group_members_obj]
+        group_members = [
+            {
+                'id': user.user_id,
+                'first_name': user.name,
+                'last_name': user.lastname,
+                'username': user.username,
+                'icon_link': h.gravatar_url(user.email, 30),
+                'value_display': h.person(user.email),
+                'value': user.username,
+                'value_type': 'user',
+                'active': user.active,
+            }
+            for user in group_members_obj
+        ]
 
-        if request.is_xhr:
-            return jsonify(lambda *a, **k: {
-                'members': group_members
-            })
-
-        c.group_members = group_members
-        return render('admin/user_groups/user_group_edit.html')
+        return {
+            'members': group_members
+        }

rhodecode/controllers/admin/users.py

@@ -45,12 +45,13 b' from rhodecode.model.db import ('
     PullRequestReviewers, User, UserEmailMap, UserIpMap, RepoGroup)
 from rhodecode.model.forms import (
     UserForm, UserPermissionsForm, UserIndividualPermissionsForm)
+from rhodecode.model.repo_group import RepoGroupModel
 from rhodecode.model.user import UserModel
 from rhodecode.model.meta import Session
 from rhodecode.model.permission import PermissionModel
 from rhodecode.lib.utils import action_logger
 from rhodecode.lib.ext_json import json
-from rhodecode.lib.utils2 import datetime_to_time, safe_int
+from rhodecode.lib.utils2 import datetime_to_time, safe_int, AttributeDict
 
 log = logging.getLogger(__name__)
 
@@ -73,7 +74,7 b' class UsersController(BaseController):'
             ('ru', 'Russian (ru)'),
             ('zh', 'Chinese (zh)'),
         ]
-        PermissionModel().set_global_permission_choices(c, translator=_)
+        PermissionModel().set_global_permission_choices(c, gettext_translator=_)
 
     @HasPermissionAllDecorator('hg.admin')
     def index(self):
@@ -120,6 +121,16 b' class UsersController(BaseController):'
         c.data = json.dumps(users_data)
         return render('admin/users/users.html')
 
+    def _get_personal_repo_group_template_vars(self):
+        DummyUser = AttributeDict({
+            'username': '${username}',
+            'user_id': '${user_id}',
+        })
+        c.default_create_repo_group = RepoGroupModel() \
+            .get_default_create_personal_repo_group()
+        c.personal_repo_group_name = RepoGroupModel() \
+            .get_personal_group_name(DummyUser)
+
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def create(self):
@@ -143,6 +154,7 b' class UsersController(BaseController):'
                               % {'user_link': user_link}), category='success')
             Session().commit()
         except formencode.Invalid as errors:
+            self._get_personal_repo_group_template_vars()
             return htmlfill.render(
                 render('admin/users/user_add.html'),
                 defaults=errors.value,
@@ -163,6 +175,7 b' class UsersController(BaseController):'
         """GET /users/new: Form to create a new item"""
         # url('new_user')
         c.default_extern_type = auth_rhodecode.RhodeCodeAuthPlugin.name
+        self._get_personal_repo_group_template_vars()
         return render('admin/users/user_add.html')
 
     @HasPermissionAllDecorator('hg.admin')
@@ -339,22 +352,41 b' class UsersController(BaseController):'
 
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
+        personal_repo_group = RepoGroup.get_user_personal_repo_group(
+            c.user.user_id)
+        if personal_repo_group:
+            return redirect(url('edit_user_advanced', user_id=user_id))
 
+        personal_repo_group_name = RepoGroupModel().get_personal_group_name(
+            c.user)
+        named_personal_group = RepoGroup.get_by_group_name(
+            personal_repo_group_name)
         try:
-            desc = RepoGroupModel.PERSONAL_GROUP_DESC % {
-                'username': c.user.username}
-            if not RepoGroup.get_by_group_name(c.user.username):
-                RepoGroupModel().create(group_name=c.user.username,
-                                        group_description=desc,
-                                        owner=c.user.username)
 
-                msg = _('Created repository group `%s`' % (c.user.username,))
+            if named_personal_group and named_personal_group.user_id == c.user.user_id:
+                # migrate the same named group, and mark it as personal
+                named_personal_group.personal = True
+                Session().add(named_personal_group)
+                Session().commit()
+                msg = _('Linked repository group `%s` as personal' % (
+                    personal_repo_group_name,))
                 h.flash(msg, category='success')
+            elif not named_personal_group:
+                RepoGroupModel().create_personal_repo_group(c.user)
+
+                msg = _('Created repository group `%s`' % (
+                    personal_repo_group_name,))
+                h.flash(msg, category='success')
+            else:
+                msg = _('Repository group `%s` is already taken' % (
+                    personal_repo_group_name,))
+                h.flash(msg, category='warning')
         except Exception:
             log.exception("Exception during repository group creation")
             msg = _(
                 'An error occurred during repository group creation for user')
             h.flash(msg, category='error')
+            Session().rollback()
 
         return redirect(url('edit_user_advanced', user_id=user_id))
 
@@ -397,7 +429,9 b' class UsersController(BaseController):'
 
         c.active = 'advanced'
         c.perm_user = AuthUser(user_id=user_id, ip_addr=self.ip_addr)
-        c.personal_repo_group = RepoGroup.get_by_group_name(user.username)
+        c.personal_repo_group = c.perm_user.personal_repo_group
+        c.personal_repo_group_name = RepoGroupModel()\
+            .get_personal_group_name(user)
         c.first_admin = User.get_first_super_admin()
         defaults = user.get_dict()
 

rhodecode/controllers/changeset.py

@@ -32,7 +32,7 b' from pylons.i18n.translation import _'
 from pylons.controllers.util import redirect
 
 from rhodecode.lib import auth
-from rhodecode.lib import diffs
+from rhodecode.lib import diffs, codeblocks
 from rhodecode.lib.auth import (
     LoginRequired, HasRepoPermissionAnyDecorator, NotAnonymous)
 from rhodecode.lib.base import BaseRepoController, render
@@ -43,7 +43,7 b' from rhodecode.lib.utils import action_l'
 from rhodecode.lib.utils2 import safe_unicode
 from rhodecode.lib.vcs.backends.base import EmptyCommit
 from rhodecode.lib.vcs.exceptions import (
-    RepositoryError, CommitDoesNotExistError)
+    RepositoryError, CommitDoesNotExistError, NodeDoesNotExistError)
 from rhodecode.model.db import ChangesetComment, ChangesetStatus
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import ChangesetCommentsModel
@@ -156,15 +156,24 b' class ChangesetController(BaseRepoContro'
         c.ignorews_url = _ignorews_url
         c.context_url = _context_url
         c.fulldiff = fulldiff = request.GET.get('fulldiff')
+
+        # fetch global flags of ignore ws or context lines
+        context_lcl = get_line_ctx('', request.GET)
+        ign_whitespace_lcl = get_ignore_ws('', request.GET)
+
+        # diff_limit will cut off the whole diff if the limit is applied
+        # otherwise it will just hide the big files from the front-end
+        diff_limit = self.cut_off_limit_diff
+        file_limit = self.cut_off_limit_file
+
         # get ranges of commit ids if preset
         commit_range = commit_id_range.split('...')[:2]
-        enable_comments = True
+
         try:
             pre_load = ['affected_files', 'author', 'branch', 'date',
                         'message', 'parents']
 
             if len(commit_range) == 2:
-                enable_comments = False
                 commits = c.rhodecode_repo.get_commits(
                     start_id=commit_range[0], end_id=commit_range[1],
                     pre_load=pre_load)
@@ -190,88 +199,78 b' class ChangesetController(BaseRepoContro'
         c.lines_deleted = 0
 
         c.commit_statuses = ChangesetStatus.STATUSES
-        c.comments = []
-        c.statuses = []
         c.inline_comments = []
         c.inline_cnt = 0
         c.files = []
 
+        c.statuses = []
+        c.comments = []
+        if len(c.commit_ranges) == 1:
+            commit = c.commit_ranges[0]
+            c.comments = ChangesetCommentsModel().get_comments(
+                c.rhodecode_db_repo.repo_id,
+                revision=commit.raw_id)
+            c.statuses.append(ChangesetStatusModel().get_status(
+                c.rhodecode_db_repo.repo_id, commit.raw_id))
+            # comments from PR
+            statuses = ChangesetStatusModel().get_statuses(
+                c.rhodecode_db_repo.repo_id, commit.raw_id,
+                with_revisions=True)
+            prs = set(st.pull_request for st in statuses
+                      if st.pull_request is not None)
+            # from associated statuses, check the pull requests, and
+            # show comments from them
+            for pr in prs:
+                c.comments.extend(pr.comments)
+
         # Iterate over ranges (default commit view is always one commit)
         for commit in c.commit_ranges:
-            if method == 'show':
-                c.statuses.extend([ChangesetStatusModel().get_status(
-                    c.rhodecode_db_repo.repo_id, commit.raw_id)])
-
-                c.comments.extend(ChangesetCommentsModel().get_comments(
-                    c.rhodecode_db_repo.repo_id,
-                    revision=commit.raw_id))
-
-                # comments from PR
-                st = ChangesetStatusModel().get_statuses(
-                    c.rhodecode_db_repo.repo_id, commit.raw_id,
-                    with_revisions=True)
-
-                # from associated statuses, check the pull requests, and
-                # show comments from them
-
-                prs = set(x.pull_request for x in
-                          filter(lambda x: x.pull_request is not None, st))
-                for pr in prs:
-                    c.comments.extend(pr.comments)
-
-                inlines = ChangesetCommentsModel().get_inline_comments(
-                    c.rhodecode_db_repo.repo_id, revision=commit.raw_id)
-                c.inline_comments.extend(inlines.iteritems())
-
             c.changes[commit.raw_id] = []
 
             commit2 = commit
             commit1 = commit.parents[0] if commit.parents else EmptyCommit()
 
-            # fetch global flags of ignore ws or context lines
-            context_lcl = get_line_ctx('', request.GET)
-            ign_whitespace_lcl = get_ignore_ws('', request.GET)
-
             _diff = c.rhodecode_repo.get_diff(
                 commit1, commit2,
                 ignore_whitespace=ign_whitespace_lcl, context=context_lcl)
+            diff_processor = diffs.DiffProcessor(
+                _diff, format='newdiff', diff_limit=diff_limit,
+                file_limit=file_limit, show_full_diff=fulldiff)
 
-            # diff_limit will cut off the whole diff if the limit is applied
-            # otherwise it will just hide the big files from the front-end
-            diff_limit = self.cut_off_limit_diff
-            file_limit = self.cut_off_limit_file
-
-            diff_processor = diffs.DiffProcessor(
-                _diff, format='gitdiff', diff_limit=diff_limit,
-                file_limit=file_limit, show_full_diff=fulldiff)
             commit_changes = OrderedDict()
             if method == 'show':
                 _parsed = diff_processor.prepare()
                 c.limited_diff = isinstance(_parsed, diffs.LimitedDiffContainer)
-                for f in _parsed:
-                    c.files.append(f)
-                    st = f['stats']
-                    c.lines_added += st['added']
-                    c.lines_deleted += st['deleted']
-                    fid = h.FID(commit.raw_id, f['filename'])
-                    diff = diff_processor.as_html(enable_comments=enable_comments,
-                                                  parsed_lines=[f])
-                    commit_changes[fid] = [
-                        commit1.raw_id, commit2.raw_id,
-                        f['operation'], f['filename'], diff, st, f]
+
+                _parsed = diff_processor.prepare()
+
+                def _node_getter(commit):
+                    def get_node(fname):
+                        try:
+                            return commit.get_node(fname)
+                        except NodeDoesNotExistError:
+                            return None
+                    return get_node
+
+                inline_comments = ChangesetCommentsModel().get_inline_comments(
+                    c.rhodecode_db_repo.repo_id, revision=commit.raw_id)
+                c.inline_cnt += len(inline_comments)
+
+                diffset = codeblocks.DiffSet(
+                    repo_name=c.repo_name,
+                    source_node_getter=_node_getter(commit1),
+                    target_node_getter=_node_getter(commit2),
+                    comments=inline_comments
+                ).render_patchset(_parsed, commit1.raw_id, commit2.raw_id)
+                c.changes[commit.raw_id] = diffset
             else:
                 # downloads/raw we only need RAW diff nothing else
                 diff = diff_processor.as_raw()
-                commit_changes[''] = [None, None, None, None, diff, None, None]
-            c.changes[commit.raw_id] = commit_changes
+                c.changes[commit.raw_id] = [None, None, None, None, diff, None, None]
 
         # sort comments by how they were generated
         c.comments = sorted(c.comments, key=lambda x: x.comment_id)
 
-        # count inline comments
-        for __, lines in c.inline_comments:
-            for comments in lines.values():
-                c.inline_cnt += len(comments)
 
         if len(c.commit_ranges) == 1:
             c.commit = c.commit_ranges[0]

rhodecode/controllers/compare.py

@@ -31,13 +31,14 b' from pylons.i18n.translation import _'
 
 from rhodecode.controllers.utils import parse_path_ref, get_commit_from_ref_name
 from rhodecode.lib import helpers as h
-from rhodecode.lib import diffs
+from rhodecode.lib import diffs, codeblocks
 from rhodecode.lib.auth import LoginRequired, HasRepoPermissionAnyDecorator
 from rhodecode.lib.base import BaseRepoController, render
 from rhodecode.lib.utils import safe_str
 from rhodecode.lib.utils2 import safe_unicode, str2bool
 from rhodecode.lib.vcs.exceptions import (
-    EmptyRepositoryError, RepositoryError, RepositoryRequirementError)
+    EmptyRepositoryError, RepositoryError, RepositoryRequirementError,
+    NodeDoesNotExistError)
 from rhodecode.model.db import Repository, ChangesetStatus
 
 log = logging.getLogger(__name__)
@@ -78,7 +79,7 b' class CompareController(BaseRepoControll'
     def index(self, repo_name):
         c.compare_home = True
         c.commit_ranges = []
-        c.files = []
+        c.diffset = None
         c.limited_diff = False
         source_repo = c.rhodecode_db_repo.repo_name
         target_repo = request.GET.get('target_repo', source_repo)
@@ -239,28 +240,24 b' class CompareController(BaseRepoControll'
             commit1=source_commit, commit2=target_commit,
             path1=source_path, path=target_path)
         diff_processor = diffs.DiffProcessor(
-            txtdiff, format='gitdiff', diff_limit=diff_limit,
+            txtdiff, format='newdiff', diff_limit=diff_limit,
             file_limit=file_limit, show_full_diff=c.fulldiff)
         _parsed = diff_processor.prepare()
 
-        c.limited_diff = False
-        if isinstance(_parsed, diffs.LimitedDiffContainer):
-            c.limited_diff = True
+        def _node_getter(commit):
+            """ Returns a function that returns a node for a commit or None """
+            def get_node(fname):
+                try:
+                    return commit.get_node(fname)
+                except NodeDoesNotExistError:
+                    return None
+            return get_node
 
-        c.files = []
-        c.changes = {}
-        c.lines_added = 0
-        c.lines_deleted = 0
-        for f in _parsed:
-            st = f['stats']
-            if not st['binary']:
-                c.lines_added += st['added']
-                c.lines_deleted += st['deleted']
-            fid = h.FID('', f['filename'])
-            c.files.append([fid, f['operation'], f['filename'], f['stats'], f])
-            htmldiff = diff_processor.as_html(
-                enable_comments=False, parsed_lines=[f])
-            c.changes[fid] = [f['operation'], f['filename'], htmldiff, f]
+        c.diffset = codeblocks.DiffSet(
+            repo_name=source_repo.repo_name,
+            source_node_getter=_node_getter(source_commit),
+            target_node_getter=_node_getter(target_commit),
+        ).render_patchset(_parsed, source_ref, target_ref)
 
         c.preview_mode = merge
 

rhodecode/controllers/files.py

@@ -36,6 +36,8 b' from webob.exc import HTTPNotFound, HTTP'
 from rhodecode.controllers.utils import parse_path_ref
 from rhodecode.lib import diffs, helpers as h, caches
 from rhodecode.lib.compat import OrderedDict
+from rhodecode.lib.codeblocks import (
+    filenode_as_lines_tokens, filenode_as_annotated_lines_tokens)
 from rhodecode.lib.utils import jsonify, action_logger
 from rhodecode.lib.utils2 import (
     convert_line_endings, detect_mode, safe_str, str2bool)
@@ -221,9 +223,19 b' class FilesController(BaseRepoController'
             c.file_author = True
             c.file_tree = ''
             if c.file.is_file():
-                c.renderer = (
-                    c.renderer and h.renderer_from_filename(c.file.path))
+                c.file_source_page = 'true'
                 c.file_last_commit = c.file.last_commit
+                if c.file.size < self.cut_off_limit_file:
+                    if c.annotate:  # annotation has precedence over renderer
+                        c.annotated_lines = filenode_as_annotated_lines_tokens(
+                            c.file
+                        )
+                    else:
+                        c.renderer = (
+                            c.renderer and h.renderer_from_filename(c.file.path)
+                        )
+                        if not c.renderer:
+                            c.lines = filenode_as_lines_tokens(c.file)
 
                 c.on_branch_head = self._is_valid_head(
                     commit_id, c.rhodecode_repo)
@@ -233,6 +245,7 b' class FilesController(BaseRepoController'
                 c.authors = [(h.email(author),
                               h.person(author, 'username_or_name_or_email'))]
             else:
+                c.file_source_page = 'false'
                 c.authors = []
                 c.file_tree = self._get_tree_at_commit(
                     repo_name, c.commit.raw_id, f_path)

rhodecode/controllers/forks.py

@@ -60,8 +60,7 b' class ForksController(BaseRepoController'
         c.repo_groups_choices = map(lambda k: unicode(k[0]), c.repo_groups)
         choices, c.landing_revs = ScmModel().get_repo_landing_revs()
         c.landing_revs_choices = choices
-        c.personal_repo_group = RepoGroup.get_by_group_name(
-            c.rhodecode_user.username)
+        c.personal_repo_group = c.rhodecode_user.personal_repo_group
 
     def __load_data(self, repo_name=None):
         """

rhodecode/controllers/home.py

@@ -286,4 +286,3 b' class HomeController(BaseController):'
         _user_groups = _user_groups
 
         return {'suggestions': _user_groups}
-

rhodecode/controllers/pullrequests.py

@@ -22,6 +22,7 b''
 pull requests controller for rhodecode for initializing pull requests
 """
 
+import peppercorn
 import formencode
 import logging
 
@@ -29,22 +30,26 b' from webob.exc import HTTPNotFound, HTTP'
 from pylons import request, tmpl_context as c, url
 from pylons.controllers.util import redirect
 from pylons.i18n.translation import _
+from pyramid.threadlocal import get_current_registry
 from sqlalchemy.sql import func
 from sqlalchemy.sql.expression import or_
 
 from rhodecode import events
-from rhodecode.lib import auth, diffs, helpers as h
+from rhodecode.lib import auth, diffs, helpers as h, codeblocks
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.base import (
     BaseRepoController, render, vcs_operation_context)
 from rhodecode.lib.auth import (
     LoginRequired, HasRepoPermissionAnyDecorator, NotAnonymous,
     HasAcceptedRepoType, XHRRequired)
+from rhodecode.lib.channelstream import channelstream_request
+from rhodecode.lib.compat import OrderedDict
 from rhodecode.lib.utils import jsonify
 from rhodecode.lib.utils2 import safe_int, safe_str, str2bool, safe_unicode
-from rhodecode.lib.vcs.backends.base import EmptyCommit
+from rhodecode.lib.vcs.backends.base import EmptyCommit, UpdateFailureReason
 from rhodecode.lib.vcs.exceptions import (
-    EmptyRepositoryError, CommitDoesNotExistError, RepositoryRequirementError)
+    EmptyRepositoryError, CommitDoesNotExistError, RepositoryRequirementError,
+    NodeDoesNotExistError)
 from rhodecode.lib.diffs import LimitedDiffContainer
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import ChangesetCommentsModel
@@ -61,7 +66,7 b' class PullrequestsController(BaseRepoCon'
     def __before__(self):
         super(PullrequestsController, self).__before__()
 
-    def _load_compare_data(self, pull_request, enable_comments=True):
+    def _load_compare_data(self, pull_request, inline_comments, enable_comments=True):
         """
         Load context data needed for generating compare diff
 
@@ -114,6 +119,7 b' class PullrequestsController(BaseRepoCon'
         except RepositoryRequirementError:
             c.missing_requirements = True
 
+        c.changes = {}
         c.missing_commits = False
         if (c.missing_requirements or
                 isinstance(source_commit, EmptyCommit) or
@@ -123,11 +129,31 b' class PullrequestsController(BaseRepoCon'
         else:
             vcs_diff = PullRequestModel().get_diff(pull_request)
             diff_processor = diffs.DiffProcessor(
-                vcs_diff, format='gitdiff', diff_limit=diff_limit,
+                vcs_diff, format='newdiff', diff_limit=diff_limit,
                 file_limit=file_limit, show_full_diff=c.fulldiff)
             _parsed = diff_processor.prepare()
 
-        c.limited_diff = isinstance(_parsed, LimitedDiffContainer)
+            commit_changes = OrderedDict()
+            _parsed = diff_processor.prepare()
+            c.limited_diff = isinstance(_parsed, diffs.LimitedDiffContainer)
+
+            _parsed = diff_processor.prepare()
+
+            def _node_getter(commit):
+                def get_node(fname):
+                    try:
+                        return commit.get_node(fname)
+                    except NodeDoesNotExistError:
+                        return None
+                return get_node
+
+            c.diffset = codeblocks.DiffSet(
+                repo_name=c.repo_name,
+                source_node_getter=_node_getter(target_commit),
+                target_node_getter=_node_getter(source_commit),
+                comments=inline_comments
+            ).render_patchset(_parsed, target_commit.raw_id, source_commit.raw_id)
+
 
         c.files = []
         c.changes = {}
@@ -136,17 +162,17 b' class PullrequestsController(BaseRepoCon'
         c.included_files = []
         c.deleted_files = []
 
-        for f in _parsed:
-            st = f['stats']
-            c.lines_added += st['added']
-            c.lines_deleted += st['deleted']
+        # for f in _parsed:
+        #     st = f['stats']
+        #     c.lines_added += st['added']
+        #     c.lines_deleted += st['deleted']
 
-            fid = h.FID('', f['filename'])
-            c.files.append([fid, f['operation'], f['filename'], f['stats']])
-            c.included_files.append(f['filename'])
-            html_diff = diff_processor.as_html(enable_comments=enable_comments,
-                                               parsed_lines=[f])
-            c.changes[fid] = [f['operation'], f['filename'], html_diff, f]
+        #     fid = h.FID('', f['filename'])
+        #     c.files.append([fid, f['operation'], f['filename'], f['stats']])
+        #     c.included_files.append(f['filename'])
+        #     html_diff = diff_processor.as_html(enable_comments=enable_comments,
+        #                                        parsed_lines=[f])
+        #     c.changes[fid] = [f['operation'], f['filename'], html_diff, f]
 
     def _extract_ordering(self, request):
         column_index = safe_int(request.GET.get('order[0][column]'))
@@ -295,10 +321,12 b' class PullrequestsController(BaseRepoCon'
             redirect(url('pullrequest_home', repo_name=source_repo.repo_name))
 
         default_target_repo = source_repo
-        if (source_repo.parent and
-                not source_repo.parent.scm_instance().is_empty()):
-            # change default if we have a parent repo
-            default_target_repo = source_repo.parent
+
+        if source_repo.parent:
+            parent_vcs_obj = source_repo.parent.scm_instance()
+            if parent_vcs_obj and not parent_vcs_obj.is_empty():
+                # change default if we have a parent repo
+                default_target_repo = source_repo.parent
 
         target_repo_data = PullRequestModel().generate_repo_data(
             default_target_repo)
@@ -360,7 +388,8 b' class PullrequestsController(BaseRepoCon'
         add_parent = False
         if repo.parent:
             if filter_query in repo.parent.repo_name:
-                if not repo.parent.scm_instance().is_empty():
+                parent_vcs_obj = repo.parent.scm_instance()
+                if parent_vcs_obj and not parent_vcs_obj.is_empty():
                     add_parent = True
 
         limit = 20 - 1 if add_parent else 20
@@ -397,8 +426,10 b' class PullrequestsController(BaseRepoCon'
         if not repo:
             raise HTTPNotFound
 
+        controls = peppercorn.parse(request.POST.items())
+
         try:
-            _form = PullRequestForm(repo.repo_id)().to_python(request.POST)
+            _form = PullRequestForm(repo.repo_id)().to_python(controls)
         except formencode.Invalid as errors:
             if errors.error_dict.get('revisions'):
                 msg = 'Revisions: %s' % errors.error_dict['revisions']
@@ -417,7 +448,8 b' class PullrequestsController(BaseRepoCon'
         target_repo = _form['target_repo']
         target_ref = _form['target_ref']
         commit_ids = _form['revisions'][::-1]
-        reviewers = _form['review_members']
+        reviewers = [
+            (r['user_id'], r['reasons']) for r in _form['review_members']]
 
         # find the ancestor for this pr
         source_db_repo = Repository.get_by_repo_name(_form['source_repo'])
@@ -476,8 +508,11 b' class PullrequestsController(BaseRepoCon'
         allowed_to_update = PullRequestModel().check_user_update(
             pull_request, c.rhodecode_user)
         if allowed_to_update:
-            if 'reviewers_ids' in request.POST:
-                self._update_reviewers(pull_request_id)
+            controls = peppercorn.parse(request.POST.items())
+
+            if 'review_members' in controls:
+                self._update_reviewers(
+                    pull_request_id, controls['review_members'])
             elif str2bool(request.POST.get('update_commits', 'false')):
                 self._update_commits(pull_request)
             elif str2bool(request.POST.get('close_pull_request', 'false')):
@@ -506,32 +541,51 b' class PullrequestsController(BaseRepoCon'
         return
 
     def _update_commits(self, pull_request):
-        try:
-            if PullRequestModel().has_valid_update_type(pull_request):
-                updated_version, changes = PullRequestModel().update_commits(
-                    pull_request)
-                if updated_version:
-                    msg = _(
-                        u'Pull request updated to "{source_commit_id}" with '
-                        u'{count_added} added, {count_removed} removed '
-                        u'commits.'
-                    ).format(
-                        source_commit_id=pull_request.source_ref_parts.commit_id,
-                        count_added=len(changes.added),
-                        count_removed=len(changes.removed))
-                    h.flash(msg, category='success')
-                else:
-                    h.flash(_("Nothing changed in pull request."),
-                            category='warning')
-            else:
-                msg = _(
-                    u"Skipping update of pull request due to reference "
-                    u"type: {reference_type}"
-                ).format(reference_type=pull_request.source_ref_parts.type)
-                h.flash(msg, category='warning')
-        except CommitDoesNotExistError:
-            h.flash(
-                _(u'Update failed due to missing commits.'), category='error')
+        resp = PullRequestModel().update_commits(pull_request)
+
+        if resp.executed:
+            msg = _(
+                u'Pull request updated to "{source_commit_id}" with '
+                u'{count_added} added, {count_removed} removed commits.')
+            msg = msg.format(
+                source_commit_id=pull_request.source_ref_parts.commit_id,
+                count_added=len(resp.changes.added),
+                count_removed=len(resp.changes.removed))
+            h.flash(msg, category='success')
+
+            registry = get_current_registry()
+            rhodecode_plugins = getattr(registry, 'rhodecode_plugins', {})
+            channelstream_config = rhodecode_plugins.get('channelstream', {})
+            if channelstream_config.get('enabled'):
+                message = msg + (
+                    ' - <a onclick="window.location.reload()">'
+                    '<strong>{}</strong></a>'.format(_('Reload page')))
+                channel = '/repo${}$/pr/{}'.format(
+                    pull_request.target_repo.repo_name,
+                    pull_request.pull_request_id
+                )
+                payload = {
+                    'type': 'message',
+                    'user': 'system',
+                    'exclude_users': [request.user.username],
+                    'channel': channel,
+                    'message': {
+                        'message': message,
+                        'level': 'success',
+                        'topic': '/notifications'
+                    }
+                }
+                channelstream_request(
+                    channelstream_config, [payload], '/message',
+                    raise_exc=False)
+        else:
+            msg = PullRequestModel.UPDATE_STATUS_MESSAGES[resp.reason]
+            warning_reasons = [
+                UpdateFailureReason.NO_CHANGE,
+                UpdateFailureReason.WRONG_REF_TPYE,
+            ]
+            category = 'warning' if resp.reason in warning_reasons else 'error'
+            h.flash(msg, category=category)
 
     @auth.CSRFRequired()
     @LoginRequired()
@@ -601,11 +655,10 b' class PullrequestsController(BaseRepoCon'
                 merge_resp.failure_reason)
             h.flash(msg, category='error')
 
-    def _update_reviewers(self, pull_request_id):
-        reviewers_ids = map(int, filter(
-            lambda v: v not in [None, ''],
-            request.POST.get('reviewers_ids', '').split(',')))
-        PullRequestModel().update_reviewers(pull_request_id, reviewers_ids)
+    def _update_reviewers(self, pull_request_id, review_members):
+        reviewers = [
+            (int(r['user_id']), r['reasons']) for r in review_members]
+        PullRequestModel().update_reviewers(pull_request_id, reviewers)
         Session().commit()
 
     def _reject_close(self, pull_request):
@@ -655,6 +708,10 b' class PullrequestsController(BaseRepoCon'
             c.pull_request, c.rhodecode_user) and not c.pull_request.is_closed()
         c.allowed_to_merge = PullRequestModel().check_user_merge(
             c.pull_request, c.rhodecode_user) and not c.pull_request.is_closed()
+        c.shadow_clone_url = PullRequestModel().get_shadow_clone_url(
+            c.pull_request)
+        c.allowed_to_delete = PullRequestModel().check_user_delete(
+            c.pull_request, c.rhodecode_user) and not c.pull_request.is_closed()
 
         cc_model = ChangesetCommentsModel()
 
@@ -669,23 +726,13 b' class PullrequestsController(BaseRepoCon'
             c.pr_merge_status = False
         # load compare data into template context
         enable_comments = not c.pull_request.is_closed()
-        self._load_compare_data(c.pull_request, enable_comments=enable_comments)
 
-        # this is a hack to properly display links, when creating PR, the
-        # compare view and others uses different notation, and
-        # compare_commits.html renders links based on the target_repo.
-        # We need to swap that here to generate it properly on the html side
-        c.target_repo = c.source_repo
 
         # inline comments
-        c.inline_cnt = 0
         c.inline_comments = cc_model.get_inline_comments(
             c.rhodecode_db_repo.repo_id,
-            pull_request=pull_request_id).items()
-        # count inline comments
-        for __, lines in c.inline_comments:
-            for comments in lines.values():
-                c.inline_cnt += len(comments)
+            pull_request=pull_request_id)
+        c.inline_cnt = len(c.inline_comments)
 
         # outdated comments
         c.outdated_cnt = 0
@@ -702,6 +749,15 b' class PullrequestsController(BaseRepoCon'
         else:
             c.outdated_comments = {}
 
+        self._load_compare_data(
+            c.pull_request, c.inline_comments, enable_comments=enable_comments)
+
+        # this is a hack to properly display links, when creating PR, the
+        # compare view and others uses different notation, and
+        # compare_commits.html renders links based on the target_repo.
+        # We need to swap that here to generate it properly on the html side
+        c.target_repo = c.source_repo
+
         # comments
         c.comments = cc_model.get_comments(c.rhodecode_db_repo.repo_id,
                                            pull_request=pull_request_id)

rhodecode/controllers/summary.py

@@ -251,6 +251,16 b' class SummaryController(BaseRepoControll'
         }
         return data
 
+    @LoginRequired()
+    @HasRepoPermissionAnyDecorator('repository.read', 'repository.write',
+                                   'repository.admin')
+    @jsonify
+    def repo_default_reviewers_data(self, repo_name):
+        return {
+            'reviewers': [utils.reviewer_as_json(
+                user=c.rhodecode_db_repo.user, reasons=None)]
+        }
+
     @jsonify
     def repo_refs_changelog_data(self, repo_name):
         repo = c.rhodecode_repo

rhodecode/controllers/utils.py

@@ -86,3 +86,21 b' def get_commit_from_ref_name(repo, ref_n'
                     '%s "%s" does not exist' % (ref_type, ref_name))
 
     return repo_scm.get_commit(commit_id)
+
+
+def reviewer_as_json(user, reasons):
+    """
+    Returns json struct of a reviewer for frontend
+
+    :param user: the reviewer
+    :param reasons: list of strings of why they are reviewers
+    """
+
+    return {
+        'user_id': user.user_id,
+        'reasons': reasons,
+        'username': user.username,
+        'firstname': user.firstname,
+        'lastname': user.lastname,
+        'gravatar_link': h.gravatar_url(user.email, 14),
+    }

rhodecode/events/__init__.py

@@ -48,6 +48,7 b' from rhodecode.events.base import Rhodec'
 
 from rhodecode.events.user import (  # noqa
     UserPreCreate,
+    UserPostCreate,
     UserPreUpdate,
     UserRegistered
 )

rhodecode/events/user.py

@@ -51,6 +51,19 b' class UserPreCreate(RhodecodeEvent):'
         self.user_data = user_data
 
 
+@implementer(IUserPreCreate)
+class UserPostCreate(RhodecodeEvent):
+    """
+    An instance of this class is emitted as an :term:`event` after a new user
+    object is created.
+    """
+    name = 'user-post-create'
+    display_name = lazy_ugettext('user post create')
+
+    def __init__(self, user_data):
+        self.user_data = user_data
+
+
 @implementer(IUserPreUpdate)
 class UserPreUpdate(RhodecodeEvent):
     """

rhodecode/integrations/types/hipchat.py

@@ -161,7 +161,7 b' class HipchatIntegrationType(Integration'
         comment_text = data['comment']['text']
         if len(comment_text) > 200:
             comment_text = '{comment_text}<a href="{comment_url}">...<a/>'.format(
-                comment_text=comment_text[:200],
+                comment_text=h.html_escape(comment_text[:200]),
                 comment_url=data['comment']['url'],
             )
 
@@ -179,8 +179,8 b' class HipchatIntegrationType(Integration'
                 number=data['pullrequest']['pull_request_id'],
                 pr_url=data['pullrequest']['url'],
                 pr_status=data['pullrequest']['status'],
-                pr_title=data['pullrequest']['title'],
-                comment_text=comment_text
+                pr_title=h.html_escape(data['pullrequest']['title']),
+                comment_text=h.html_escape(comment_text)
             )
         )
 
@@ -193,7 +193,7 b' class HipchatIntegrationType(Integration'
                 number=data['pullrequest']['pull_request_id'],
                 pr_url=data['pullrequest']['url'],
                 pr_status=data['pullrequest']['status'],
-                pr_title=data['pullrequest']['title'],
+                pr_title=h.html_escape(data['pullrequest']['title']),
             )
         )
 
@@ -206,21 +206,20 b' class HipchatIntegrationType(Integration'
         }.get(event.__class__, str(event.__class__))
 
         return ('Pull request <a href="{url}">#{number}</a> - {title} '
-                '{action} by {user}').format(
+                '{action} by <b>{user}</b>').format(
             user=data['actor']['username'],
             number=data['pullrequest']['pull_request_id'],
             url=data['pullrequest']['url'],
-            title=data['pullrequest']['title'],
+            title=h.html_escape(data['pullrequest']['title']),
             action=action
         )
 
     def format_repo_push_event(self, data):
         branch_data = {branch['name']: branch
-                      for branch in data['push']['branches']}
+                       for branch in data['push']['branches']}
 
         branches_commits = {}
         for commit in data['push']['commits']:
-            log.critical(commit)
             if commit['branch'] not in branches_commits:
                 branch_commits = {'branch': branch_data[commit['branch']],
                                   'commits': []}
@@ -238,7 +237,7 b' class HipchatIntegrationType(Integration'
     def format_repo_create_event(self, data):
         return '<a href="{}">{}</a> ({}) repository created by <b>{}</b>'.format(
             data['repo']['url'],
-            data['repo']['repo_name'],
+            h.html_escape(data['repo']['repo_name']),
             data['repo']['repo_type'],
             data['actor']['username'],
         )

rhodecode/integrations/types/slack.py

@@ -173,7 +173,7 b' class SlackIntegrationType(IntegrationTy'
 
         return (textwrap.dedent(
             '''
-            {user} commented on pull request <{pr_url}|#{number}> - {pr_title}:
+            *{user}* commented on pull request <{pr_url}|#{number}> - {pr_title}:
             >>> {comment_status}{comment_text}
             ''').format(
                 comment_status=comment_status,
@@ -208,7 +208,7 b' class SlackIntegrationType(IntegrationTy'
         }.get(event.__class__, str(event.__class__))
 
         return ('Pull request <{url}|#{number}> - {title} '
-                '{action} by {user}').format(
+                '`{action}` by *{user}*').format(
             user=data['actor']['username'],
             number=data['pullrequest']['pull_request_id'],
             url=data['pullrequest']['url'],
@@ -218,11 +218,10 b' class SlackIntegrationType(IntegrationTy'
 
     def format_repo_push_event(self, data):
         branch_data = {branch['name']: branch
-                      for branch in data['push']['branches']}
+                       for branch in data['push']['branches']}
 
         branches_commits = {}
         for commit in data['push']['commits']:
-            log.critical(commit)
             if commit['branch'] not in branches_commits:
                 branch_commits = {'branch': branch_data[commit['branch']],
                                   'commits': []}

rhodecode/integrations/types/webhook.py

@@ -19,13 +19,15 b''
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 from __future__ import unicode_literals
+import string
+from collections import OrderedDict
 
 import deform
 import logging
 import requests
 import colander
 from celery.task import task
-from mako.template import Template
+from requests.packages.urllib3.util.retry import Retry
 
 from rhodecode import events
 from rhodecode.translation import _
@@ -33,12 +35,127 b' from rhodecode.integrations.types.base i'
 
 log = logging.getLogger(__name__)
 
+# updating this required to update the `base_vars` passed in url calling func
+WEBHOOK_URL_VARS = [
+    'repo_name',
+    'repo_type',
+    'repo_id',
+    'repo_url',
+
+    # special attrs below that we handle, using multi-call
+    'branch',
+    'commit_id',
+
+    # pr events vars
+    'pull_request_id',
+    'pull_request_url',
+
+]
+URL_VARS = ', '.join('${' + x + '}' for x in WEBHOOK_URL_VARS)
+
+
+class WebhookHandler(object):
+    def __init__(self, template_url, secret_token):
+        self.template_url = template_url
+        self.secret_token = secret_token
+
+    def get_base_parsed_template(self, data):
+        """
+        initially parses the passed in template with some common variables
+        available on ALL calls
+        """
+        # note: make sure to update the `WEBHOOK_URL_VARS` if this changes
+        common_vars = {
+            'repo_name': data['repo']['repo_name'],
+            'repo_type': data['repo']['repo_type'],
+            'repo_id': data['repo']['repo_id'],
+            'repo_url': data['repo']['url'],
+        }
+
+        return string.Template(
+            self.template_url).safe_substitute(**common_vars)
+
+    def repo_push_event_handler(self, event, data):
+        url = self.get_base_parsed_template(data)
+        url_cals = []
+        branch_data = OrderedDict()
+        for obj in data['push']['branches']:
+            branch_data[obj['name']] = obj
+
+        branches_commits = OrderedDict()
+        for commit in data['push']['commits']:
+            if commit['branch'] not in branches_commits:
+                branch_commits = {'branch': branch_data[commit['branch']],
+                                  'commits': []}
+                branches_commits[commit['branch']] = branch_commits
+
+            branch_commits = branches_commits[commit['branch']]
+            branch_commits['commits'].append(commit)
+
+        if '${branch}' in url:
+            # call it multiple times, for each branch if used in variables
+            for branch, commit_ids in branches_commits.items():
+                branch_url = string.Template(url).safe_substitute(branch=branch)
+                # call further down for each commit if used
+                if '${commit_id}' in branch_url:
+                    for commit_data in commit_ids['commits']:
+                        commit_id = commit_data['raw_id']
+                        commit_url = string.Template(branch_url).safe_substitute(
+                            commit_id=commit_id)
+                        # register per-commit call
+                        log.debug(
+                            'register webhook call(%s) to url %s', event, commit_url)
+                        url_cals.append((commit_url, self.secret_token, data))
+
+                else:
+                    # register per-branch call
+                    log.debug(
+                        'register webhook call(%s) to url %s', event, branch_url)
+                    url_cals.append((branch_url, self.secret_token, data))
+
+        else:
+            log.debug(
+                'register webhook call(%s) to url %s', event, url)
+            url_cals.append((url, self.secret_token, data))
+
+        return url_cals
+
+    def repo_create_event_handler(self, event, data):
+        url = self.get_base_parsed_template(data)
+        log.debug(
+            'register webhook call(%s) to url %s', event, url)
+        return [(url, self.secret_token, data)]
+
+    def pull_request_event_handler(self, event, data):
+        url = self.get_base_parsed_template(data)
+        log.debug(
+            'register webhook call(%s) to url %s', event, url)
+        url = string.Template(url).safe_substitute(
+            pull_request_id=data['pullrequest']['pull_request_id'],
+            pull_request_url=data['pullrequest']['url'])
+        return [(url, self.secret_token, data)]
+
+    def __call__(self, event, data):
+        if isinstance(event, events.RepoPushEvent):
+            return self.repo_push_event_handler(event, data)
+        elif isinstance(event, events.RepoCreateEvent):
+            return self.repo_create_event_handler(event, data)
+        elif isinstance(event, events.PullRequestEvent):
+            return self.pull_request_event_handler(event, data)
+        else:
+            raise ValueError('event type not supported: %s' % events)
+
 
 class WebhookSettingsSchema(colander.Schema):
     url = colander.SchemaNode(
         colander.String(),
         title=_('Webhook URL'),
-        description=_('URL of the webhook to receive POST event.'),
+        description=
+            _('URL of the webhook to receive POST event. Following variables '
+              'are allowed to be used: {vars}. Some of the variables would '
+              'trigger multiple calls, like ${{branch}} or ${{commit_id}}. '
+              'Webhook will be called as many times as unique objects in '
+              'data in such cases.').format(vars=URL_VARS),
         missing=colander.required,
         required=True,
         validator=colander.url,
@@ -58,8 +175,6 b' class WebhookSettingsSchema(colander.Sch'
     )
 
 
-
-
 class WebhookIntegrationType(IntegrationTypeBase):
     key = 'webhook'
     display_name = _('Webhook')
@@ -104,14 +219,30 b' class WebhookIntegrationType(Integration'
             return
 
         data = event.as_dict()
-        post_to_webhook(data, self.settings)
+        template_url = self.settings['url']
+
+        handler = WebhookHandler(template_url, self.settings['secret_token'])
+        url_calls = handler(event, data)
+        log.debug('webhook: calling following urls: %s',
+                  [x[0] for x in url_calls])
+        post_to_webhook(url_calls)
 
 
 @task(ignore_result=True)
-def post_to_webhook(data, settings):
-    log.debug('sending event:%s to webhook %s', data['name'], settings['url'])
-    resp = requests.post(settings['url'], json={
-        'token': settings['secret_token'],
-        'event': data
-    })
-    resp.raise_for_status()  # raise exception on a failed request
+def post_to_webhook(url_calls):
+    max_retries = 3
+    for url, token, data in url_calls:
+        # retry max N times
+        retries = Retry(
+            total=max_retries,
+            backoff_factor=0.15,
+            status_forcelist=[500, 502, 503, 504])
+        req_session = requests.Session()
+        req_session.mount(
+            'http://', requests.adapters.HTTPAdapter(max_retries=retries))
+
+        resp = req_session.post(url, json={
+            'token': token,
+            'event': data
+        })
+        resp.raise_for_status()  # raise exception on a failed request

rhodecode/lib/auth.py

@@ -49,7 +49,7 b' from rhodecode.model.meta import Session'
 from rhodecode.model.user import UserModel
 from rhodecode.model.db import (
     User, Repository, Permission, UserToPerm, UserGroupToPerm, UserGroupMember,
-    UserIpMap, UserApiKeys)
+    UserIpMap, UserApiKeys, RepoGroup)
 from rhodecode.lib import caches
 from rhodecode.lib.utils2 import safe_unicode, aslist, safe_str, md5
 from rhodecode.lib.utils import (
@@ -983,6 +983,9 b' class AuthUser(object):'
         inherit = self.inherit_default_permissions
         return AuthUser.check_ip_allowed(self.user_id, self.ip_addr,
                                          inherit_from_default=inherit)
+    @property
+    def personal_repo_group(self):
+        return RepoGroup.get_user_personal_repo_group(self.user_id)
 
     @classmethod
     def check_ip_allowed(cls, user_id, ip_addr, inherit_from_default):

rhodecode/lib/base.py

@@ -163,7 +163,8 b' def get_access_path(environ):'
 
 
 def vcs_operation_context(
-        environ, repo_name, username, action, scm, check_locking=True):
+        environ, repo_name, username, action, scm, check_locking=True,
+        is_shadow_repo=False):
     """
     Generate the context for a vcs operation, e.g. push or pull.
 
@@ -200,6 +201,7 b' def vcs_operation_context('
         'locked_by': locked_by,
         'server_url': utils2.get_server_url(environ),
         'hooks': get_enabled_hook_classes(ui_settings),
+        'is_shadow_repo': is_shadow_repo,
     }
     return extras
 
@@ -363,6 +365,18 b' def attach_context_attributes(context, r'
     # Fix this and remove it from base controller.
     # context.repo_name = get_repo_slug(request)  # can be empty
 
+    diffmode = 'sideside'
+    if request.GET.get('diffmode'):
+        if request.GET['diffmode'] == 'unified':
+            diffmode = 'unified'
+    elif request.session.get('diffmode'):
+        diffmode = request.session['diffmode']
+
+    context.diffmode = diffmode
+
+    if request.session.get('diffmode') != diffmode:
+        request.session['diffmode'] = diffmode
+
     context.csrf_token = auth.get_csrf_token()
     context.backends = rhodecode.BACKENDS.keys()
     context.backends.sort()

rhodecode/lib/caches.py

@@ -24,9 +24,10 b' import logging'
 import threading
 
 from beaker.cache import _cache_decorate, cache_regions, region_invalidate
+from sqlalchemy.exc import IntegrityError
 
 from rhodecode.lib.utils import safe_str, md5
-from rhodecode.model.db import Session, CacheKey, IntegrityError
+from rhodecode.model.db import Session, CacheKey
 
 log = logging.getLogger(__name__)
 

rhodecode/lib/channelstream.py

@@ -18,16 +18,14 b''
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
+import hashlib
+import itsdangerous
 import logging
 import os
-
-import itsdangerous
 import requests
-
 from dogpile.core import ReadWriteMutex
 
 import rhodecode.lib.helpers as h
-
 from rhodecode.lib.auth import HasRepoPermissionAny
 from rhodecode.lib.ext_json import json
 from rhodecode.model.db import User
@@ -81,7 +79,7 b' def get_user_data(user_id):'
         'username': user.username,
         'first_name': user.name,
         'last_name': user.lastname,
-        'icon_link': h.gravatar_url(user.email, 14),
+        'icon_link': h.gravatar_url(user.email, 60),
         'display_name': h.person(user, 'username_or_name_or_email'),
         'display_link': h.link_to_user(user),
         'notifications': user.user_data.get('notification_status', True)
@@ -169,7 +167,9 b' def parse_channels_info(info_result, inc'
 
 
 def log_filepath(history_location, channel_name):
-    filename = '{}.log'.format(channel_name.encode('hex'))
+    hasher = hashlib.sha256()
+    hasher.update(channel_name.encode('utf8'))
+    filename = '{}.log'.format(hasher.hexdigest())
     filepath = os.path.join(history_location, filename)
     return filepath