release: Merge default into stable for release preparation

docs/contributing/checklist-tickets.rst

@@ -0,0 +1,137 b''
+.. _checklist-tickets:
+
+=================
+Ticket Checklists
+=================
+
+
+Ticket Description
+==================
+
+In general these things really matter in the description:
+
+- Reasoning / Rationale. Explain "WHY" it makes sense and is important.
+
+- How to reproduce. Easy to follow steps, that’s important.
+
+  - Observation: The problem (short)
+
+  - Expectation: How it should be (short)
+
+- Specs: It is fine to draft them as good as it works.
+
+  If anything is unclear, please ask for a review or help on this via the
+  Community Portal or Slack channel.
+
+
+Checklists for Tickets
+======================
+
+BUG
+---
+
+Definition: An existing function that does not work as expected for the user.
+
+- Problem description
+- Steps needed to recreate (gherkin)
+- Link to the screen in question and/or description of how to find it via 
+  navigation
+- Explanation of what the expected outcome is
+- Any hints into the source of the problem
+- Information about platform/browser/db/etc. where applicable
+- Examples of other similar cases which have different behaviour
+
+DESIGN
+------
+
+Definition: Styling and user interface issues, including cosmetic improvements
+or appearance and behaviour of frontend functionality.
+
+- Screenshot/animation of existing page/behaviour
+- Sketches or wireframes if available
+- Link to the screen in question and/or description of how to find it via 
+  navigation
+- Problem description
+- Explanation of what the expected outcome is
+- Since this may be examined by a designer; it should be written in a way that a
+  non-developer can understand
+
+EPIC
+----
+
+Definition: A collection of tickets which together complete a larger overall
+project.
+
+- Benefit explanation
+- Clear objective - when is this complete?
+- Explanations of exceptions/corner cases
+- Documentation subtask
+- Comprehensive wireframes and/or design subtasks
+- Links to subtasks
+
+FEATURE
+-------
+
+Definition: A new function in the software which previously did not exist.
+
+- Benefit explanation
+- Clear objective
+- Explanations of exceptions/corner cases
+- Documentation subtask
+- Comprehensive wireframes and/or design subtasks
+
+SUPPORT
+-------
+
+Definition: An issue related to a customer report.
+
+- Link to support ticket, if available
+- Problem description
+- Steps needed to recreate (gherkin)
+- Link to the screen in question and/or description of how to find it via 
+  navigation
+- Explanation of what the expected outcome is
+- Any hints into the source of the problem
+- Information about platform/browser/db/etc. where applicable
+- Examples of other similar cases which have different behaviour
+
+TASK
+----
+
+Definition: An improvement or step towards implementing a feature or fixing
+a bug. Includes refactoring and other tech debt.
+
+- Clear objective
+- Benefit explanation
+- Links to parent/related tickets
+
+
+All details below.
+
+
+External links:
+
+- Avoid linking to external images; they disappear over time. Please attach any
+  relevant images to the ticket itself.
+
+- External links in general: They also disappear over time, consider copying the
+  relevant bit of information into a comment or write a paragraph to sum up the
+  general idea.
+
+
+Hints
+=====
+
+Change Description
+------------------
+
+It can be tricky to figure out how to change the description of a ticket. There
+is a very small pencil which has to be clicked once you see the edit form of a
+ticket.
+
+
+.. figure:: images/redmine-description.png
+   :alt: Example of pencil to change the ticket description
+
+   Shows an example of the pencil which lets you change the description.
+

docs/contributing/frontend.rst

@@ -0,0 +1,153 b''
+
+==================================================
+ Code style and structure guide for frontend work
+==================================================
+
+About: Outline of frontend development practices.
+
+
+
+
+Templates
+=========
+
+- Indent with 4 spaces in general.
+- Embedded Python code follows the same conventions as in the backend.
+
+  A common problem is missed spaces around operators.
+
+
+
+
+Grunt
+=====
+
+We use Grunt to compile our JavaScript and LESS files. This is done automatically
+when you start an instance. If you are changing these files, however, it is
+recommended to amend `--reload` to the `runserver` command, or use `grunt watch`
+- the Gruntfile is located in the base directory. For more info on Grunt, see
+http://gruntjs.com/
+
+
+
+
+LESS CSS
+========
+
+
+Style
+-----
+
+- Use 4 spaces instead of tabs.
+- Avoid ``!important``; it is very often an indicator for a problem.
+
+
+
+
+Structure
+---------
+
+It is important that we maintain consistency in the LESS files so that things
+scale properly. CSS is organized using LESS and then compiled into a CSS file
+to be used in production. Find the class you need to change and change it
+there. Do not add overriding styles at the end of the file. The LESS file will
+be minified; use plenty of spacing and comments for readability.
+
+These will be kept in auxillary LESS files to be imported (in this order) at the top:
+
+- `fonts.less` (font-face declarations)
+- `mixins` (place all LESS mixins here)
+- `helpers` (basic classes for hiding mobile elements, centering, etc)
+- `variables` (theme-specific colors, spacing, and fonts which might change later)
+
+
+Sections of the primary LESS file are as follows. Add comments describing
+layout and modules.
+
+.. code-block:: css
+
+   //--- BASE ------------------//
+           Very basic, sitewide styles.
+
+   //--- LAYOUT ------------------//
+           Essential layout, ex. containers and wrappers.
+           Do not put type styles in here.
+
+   //--- MODULES ------------------//
+           Reusable sections, such as sidebars and menus.
+
+   //--- THEME ------------------//
+           Theme styles, typography, etc.
+
+
+
+Formatting rules
+~~~~~~~~~~~~~~~~
+
+- Each rule should be indented on a separate line (this is helpful for diff
+  checking).
+
+- Use a space after each colon and a semicolon after each last rule.
+
+- Put a blank line between each class.
+
+- Nested classes should be listed after the parent class' rules, separated with a
+  blank line, and indented.
+
+- Using the below as a guide, place each rule in order of its effect on content,
+  layout, sizing, and last listing minor style changes such as font color and
+  backgrounds. Not every possible rule is listed here; when adding new ones,
+  judge where it should go in the list based on that hierarchy.
+
+  .. code-block:: scss
+
+     .class {
+             content
+             list-style-type
+             position
+             float
+             top
+             right
+             bottom
+             left
+             height
+             max-height
+             min-height
+             width
+             max-width
+             min-width
+             margin
+             padding
+             indent
+             vertical-align
+             text-align
+             border
+             border-radius
+             font-size
+             line-height
+             font
+             font-style
+             font-variant
+             font-weight
+             color
+             text-shadow
+             background
+             background-color
+             box-shadow
+             background-url
+             background-position
+             background-repeat
+             background-cover
+             transitions
+             cursor
+             pointer-events
+
+             .nested-class {
+                     position
+                     background-color
+
+                     &:hover {
+                             color
+                     }
+             }
+     }

docs/contributing/overview.rst

@@ -0,0 +1,111 b''
+
+=======================
+ Contributing Overview
+=======================
+
+
+RhodeCode Community Edition is an open source code management platform. We
+encourage contributions to our project from the community. This is a basic
+overview of the procedures for adding your contribution to RhodeCode.
+
+
+
+Check the Issue Tracker
+=======================
+
+Make an account at https://issues.rhodecode.com/account/register and browse the
+current tickets for bugs to fix and tasks to do. Have a bug or feature that you
+can't find in the tracker? Create a new issue for it. When you select a ticket,
+make sure to assign it to yourself and mark it "in progress" to avoid duplicated
+work. 
+
+
+
+Sign Up at code.rhodecode.com
+=============================
+
+Make an account at https://code.rhodecode.com/ using an email or your existing
+GitHub, Bitbucket, Google, or Twitter account. Fork the repo you'd like to
+contribute to; we suggest adding your username to the fork name. Clone your fork
+to your computer. We use Mercurial for source control management; see
+https://www.mercurial-scm.org/guide to get started quickly.
+
+
+
+Set Up A Local Instance
+=======================
+
+You will need to set up an instance of RhodeCode CE using VCSServer so that you
+can see your work locally as you make changes. We recommend using Linux for this
+but it can also be built on OSX.
+
+See :doc:`dev-setup` for instructions.
+
+
+
+Code!
+=====
+
+You can now make, see, and test your changes locally. We are always improving to
+keep our code clean and the cost of maintaining it low. This applies in the same
+way for contributions. We run automated checks on our pull requests, and expect
+understandable code. We also aim to provide test coverage for as much of our
+codebase as possible; any new features should be augmented with tests.
+
+Keep in mind that when we accept your contribution, we also take responsibility
+for it; we must understand it to take on that responsibility.
+
+See :doc:`standards` for more detailed information.
+
+
+
+Commit And Push Your Changes
+============================
+
+We highly recommend making a new bookmark for each feature, bug, or set of
+commits you make so that you can point to it when creating your pull request.
+Please also reference the ticket number in your commit messages. Don't forget to
+push the bookmark!
+
+
+
+Submit a Pull Request
+=====================
+
+Go to your fork, and choose "Create Pull Request" from the Options menu. Use
+your bookmark as the source, and choose someone to review it. Don't worry about
+chosing the right person; we'll assign the best contributor for the job. You'll
+get feedback and an assigned status.
+
+Be prepared to make updates to your pull request after some feedback.
+Collaboration is part of the process and improvements can often be made.
+
+
+
+Sign the Contributor License Agreement
+======================================
+
+If your contribution is approved, you will need to virtually sign the license
+agreement in order for it to be merged into the project's codebase. You can read
+it on our website here: https://rhodecode.com/static/pdf/RhodeCode-CLA.pdf
+
+To sign, go to code.rhodecode.com
+and clone the CLA repository. Add your name and make a pull request to add it to
+the contributor agreement; this serves as your virtual signature. Once your
+signature is merged, add a link to the relevant commit to your contribution
+pull request.
+
+
+
+That's it! We'll take it from there. Thanks for your contribution!
+------------------------------------------------------------------
+
+.. note:: If you have any questions or comments, feel free to contact us through
+          either the community portal(community.rhodecode.com), IRC
+          (irc.freenode.net), or Slack (rhodecode.com/join).
+
+
+
+
+
+

docs/contributing/standards.rst

@@ -0,0 +1,177 b''
+
+======================
+Contribution Standards
+======================
+
+Standards help to improve the quality of our product and its development. Herein
+we define our standards for processes and development to maintain consistency
+and function well as a community. It is a work in progress; modifications to this
+document should be discussed and agreed upon by the community.
+
+
+--------------------------------------------------------------------------------
+
+Code
+====
+
+This provides an outline for standards we use in our codebase to keep our code
+easy to read and easy to maintain. Much of our code guidelines are based on the
+book `Clean Code <http://www.pearsonhighered.com/educator/product/Clean-Code-A-Handbook-of-Agile-Software-Craftsmanship/9780132350884.page>`_
+by Robert Martin.
+
+We maintain a Tech Glossary to provide consistency in terms and symbolic names
+used for items and concepts within the application. This is found in the CE
+project in /docs-internal/glossary.rst
+
+
+Refactoring
+-----------
+Make it better than you found it!
+
+Our codebase can always use improvement and often benefits from refactoring.
+New code should be refactored as it is being written, and old code should be
+treated with the same care as if it was new. Before doing any refactoring,
+ensure that there is test coverage on the affected code; this will help
+minimize issues.
+
+
+Python
+------
+For Python, we use `PEP8 <https://www.python.org/dev/peps/pep-0008/>`_.
+We adjust lines of code to under 80 characters and use 4 spaces for indentation.
+
+
+JavaScript
+----------
+This currently remains undefined. Suggestions welcome!
+
+
+HTML
+----
+Unfortunately, we currently have no strict HTML standards, but there are a few
+guidelines we do follow. Templates must work in all modern browsers. HTML should
+be clean and easy to read, and additionally should be free of inline CSS or
+JavaScript. It is recommended to use data attributes for JS actions where
+possible in order to separate it from styling and prevent unintentional changes.
+
+
+LESS/CSS
+--------
+We use LESS for our CSS; see :doc:`frontend` for structure and formatting
+guidelines.
+
+
+Linters
+-------
+Currently, we have a linter for pull requests which checks code against PEP8.
+We intend to add more in the future as we clarify standards.
+
+
+--------------------------------------------------------------------------------
+
+Naming Conventions
+==================
+
+These still need to be defined for naming everything from Python variables to
+HTML classes to files and folders.
+
+
+--------------------------------------------------------------------------------
+
+Testing
+=======
+
+Testing is a very important aspect of our process, especially as we are our own
+quality control team. While it is of course unrealistic to hit every potential
+combination, our goal is to cover every line of Python code with a test. 
+
+The following is a brief introduction to our test suite. Our tests are primarily
+written using `py.test <http://pytest.org/>`_
+
+
+Acceptance Tests
+----------------
+Also known as "ac tests", these test from the user and business perspective to
+check if the requirements of a feature are met. Scenarios are created at a
+feature's inception and help to define its value.
+
+py.test is used for ac tests; they are located in a repo separate from the
+other tests which follow. Each feature has a .feature file which contains a
+brief description and the scenarios to be tested.
+
+
+Functional Tests
+----------------
+These test specific functionality in the application which checks through the
+entire stack. Typically these are user actions or permissions which go through
+the web browser. They are located in rhodecode/tests.
+
+
+Unit Tests
+----------
+These test isolated, individual modules to ensure that they function correctly.
+They are located in rhodecode/tests.
+
+
+Integration Tests
+-----------------
+These are used for testing performance of larger systems than the unit tests.
+They are located in rhodecode/tests.
+
+
+JavaScript Testing
+------------------
+Currently, we have not defined how to test our JavaScript code.
+
+
+--------------------------------------------------------------------------------
+
+Pull Requests
+=============
+
+Pull requests should generally contain only one thing: a single feature, one bug
+fix, etc.. The commit history should be concise and clean, and the pull request
+should contain the ticket number (also a good idea for the commits themselves)
+to provide context for the reviewer.
+
+See also: :doc:`checklist-pull-request`
+
+
+Reviewers
+---------
+Each pull request must be approved by at least one member of the RhodeCode
+team. Assignments may be based on expertise or familiarity with a particular
+area of code, or simply availability. Switching up or adding extra community
+members for different pull requests helps to share knowledge as well as provide
+other perspectives.
+
+
+Responsibility
+--------------
+The community is responsible for maintaining features and this must be taken
+into consideration. External contributions must be held to the same standards
+as internal contributions.
+
+
+Feature Switch
+--------------
+Experimental and work-in-progress features can be hidden behind one of two
+switches:
+
+* A setting can be added to the Labs page in the Admin section which may allow
+  customers to access and toggle additional features.
+* For work-in-progress or other features where customer access is not desired,
+  use a custom setting in the .ini file as a trigger.
+
+
+--------------------------------------------------------------------------------
+
+Tickets
+=======
+
+Redmine tickets are a crucial part of our development process. Any code added or
+changed in our codebase should have a corresponding ticket to document it. With
+this in mind, it is important that tickets be as clear and concise as possible,
+including what the expected outcome is.
+
+See also: :doc:`checklist-tickets`

docs/release-notes/release-notes-4.2.0.rst

@@ -0,0 +1,70 b''
+|RCE| 4.2.0 |RNS|
+-----------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2016-06-30
+
+
+General
+^^^^^^^
+
+- Autocomplete: add GET flag support to show/hide active users on autocomplete,
+  also display this information in autocomplete data.  ref #3374
+- Gravatar: add flag to show current gravatar + user as disabled user (non-active)
+- Repos, repo groups, user groups: allow to use disabled users in owner field.
+  This fixes #3374.
+- Repos, repo groups, user groups: visually show what user is an owner of a
+  resource, and if potentially he is disabled in the system.
+- Pull requests: reorder navigation on repo pull requests, fixes #2995
+- Dependencies: bump dulwich to 0.13.0
+
+New Features
+^^^^^^^^^^^^
+
+- My account: made pull requests aggregate view for users look like not
+  created in 1995. Now uses a consistent look with repo one.
+- emails: expose profile link on registation email that super-admins receive.
+  Implements #3999.
+- Social auth: move the buttons to the top of nav so they are easier to reach.
+
+
+Security
+^^^^^^^^
+
+- Encryption: allow to pass in alternative key for encryption values. Now
+  users can use `rhodecode.encrypted_values.secret` that is alternative key,
+  de-coupled from `beaker.session` key.
+- Encryption: Implement a slightly improved AesCipher encryption.
+  This addresses issues from #4036.
+- Encryption: encrypted values now by default uses HMAC signatures to detect
+  if data or secret key is incorrect. The strict checks can be disabled using
+  `rhodecode.encrypted_values.strict = false` .ini setting
+
+
+Performance
+^^^^^^^^^^^
+
+- Sql: use smarter JOINs when fetching repository information
+- Helpers: optimize slight calls for link_to_user to save some intense queries.
+- App settings: use computed caches for repository settings, this in some cases
+  brings almost 4x performance increase for large repos with a lot of issue
+  tracker patterns.
+
+
+Fixes
+^^^^^
+
+- Fixed events on user pre/post create actions
+- Authentication: fixed problem with saving forms with errors on auth plugins
+- Svn: Avoid chunked transfer for Subversion that caused checkout issues in some cases.
+- Users: fix generate new user password helper.
+- Celery: fixed problem with workers running action in sync mode in some cases.
+- Setup-db: fix redundant question on writable dir. The question needs to be
+  asked only when the dir is actually not writable.
+- Elasticsearch: fixed issues when searching single repo using elastic search
+- Social auth: fix issues with non-active users using social authentication
+  causing a 500 error.
+- Fixed problem with largefiles extensions on per-repo settings using local
+  .hgrc files present inside the repo directory.

rhodecode/admin/__init__.py

@@ -0,0 +1,40 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+from rhodecode.admin.navigation import NavigationRegistry
+from rhodecode.config.routing import ADMIN_PREFIX
+from rhodecode.lib.utils2 import str2bool
+
+
+def includeme(config):
+    settings = config.get_settings()
+
+    # Create admin navigation registry and add it to the pyramid registry.
+    labs_active = str2bool(settings.get('labs_settings_active', False))
+    navigation_registry = NavigationRegistry(labs_active=labs_active)
+    config.registry.registerUtility(navigation_registry)
+
+    config.add_route(
+        name='admin_settings_open_source',
+        pattern=ADMIN_PREFIX + '/settings/open_source')
+
+    # Scan module for configuration decorators.
+    config.scan()

rhodecode/admin/interfaces.py

@@ -0,0 +1,29 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+from zope.interface import Interface
+
+
+class IAdminNavigationRegistry(Interface):
+    """
+    Interface for the admin navigation registry. Currently this is only
+    used to register and retrieve it via pyramids registry.
+    """
+    pass

rhodecode/admin/navigation.py

@@ -0,0 +1,124 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+import logging
+import collections
+
+from pylons import url
+from zope.interface import implementer
+
+from rhodecode.admin.interfaces import IAdminNavigationRegistry
+from rhodecode.lib.utils import get_registry
+from rhodecode.translation import _
+
+
+log = logging.getLogger(__name__)
+
+NavListEntry = collections.namedtuple('NavListEntry', ['key', 'name', 'url'])
+
+
+class NavEntry(object):
+    """
+    Represents an entry in the admin navigation.
+
+    :param key: Unique identifier used to store reference in an OrderedDict.
+    :param name: Display name, usually a translation string.
+    :param view_name: Name of the view, used generate the URL.
+    :param pyramid: Indicator to use pyramid for URL generation. This should
+        be removed as soon as we are fully migrated to pyramid.
+    """
+
+    def __init__(self, key, name, view_name, pyramid=False):
+        self.key = key
+        self.name = name
+        self.view_name = view_name
+        self.pyramid = pyramid
+
+    def generate_url(self, request):
+        if self.pyramid:
+            if hasattr(request, 'route_path'):
+                return request.route_path(self.view_name)
+            else:
+                # TODO: johbo: Remove this after migrating to pyramid.
+                # We need the pyramid request here to generate URLs to pyramid
+                # views from within pylons views.
+                from pyramid.threadlocal import get_current_request
+                pyramid_request = get_current_request()
+                return pyramid_request.route_path(self.view_name)
+        else:
+            return url(self.view_name)
+
+
+@implementer(IAdminNavigationRegistry)
+class NavigationRegistry(object):
+
+    _base_entries = [
+        NavEntry('global', _('Global'), 'admin_settings_global'),
+        NavEntry('vcs', _('VCS'), 'admin_settings_vcs'),
+        NavEntry('visual', _('Visual'), 'admin_settings_visual'),
+        NavEntry('mapping', _('Remap and Rescan'), 'admin_settings_mapping'),
+        NavEntry('issuetracker', _('Issue Tracker'),
+                 'admin_settings_issuetracker'),
+        NavEntry('email', _('Email'), 'admin_settings_email'),
+        NavEntry('hooks', _('Hooks'), 'admin_settings_hooks'),
+        NavEntry('search', _('Full Text Search'), 'admin_settings_search'),
+        NavEntry('system', _('System Info'), 'admin_settings_system'),
+        NavEntry('open_source', _('Open Source Licenses'),
+                 'admin_settings_open_source', pyramid=True),
+        # TODO: marcink: we disable supervisor now until the supervisor stats
+        # page is fixed in the nix configuration
+        # NavEntry('supervisor', _('Supervisor'), 'admin_settings_supervisor'),
+    ]
+
+    _labs_entry = NavEntry('labs', _('Labs'),
+                           'admin_settings_labs')
+
+    def __init__(self, labs_active=False):
+        self._registered_entries = collections.OrderedDict([
+            (item.key, item) for item in self.__class__._base_entries
+        ])
+
+        if labs_active:
+            self.add_entry(self._labs_entry)
+
+    def add_entry(self, entry):
+        self._registered_entries[entry.key] = entry
+
+    def get_navlist(self, request):
+        navlist = [NavListEntry(i.key, i.name, i.generate_url(request))
+                   for i in self._registered_entries.values()]
+        return navlist
+
+
+def navigation_registry(request):
+    """
+    Helper that returns the admin navigation registry.
+    """
+    pyramid_registry = get_registry(request)
+    nav_registry = pyramid_registry.queryUtility(IAdminNavigationRegistry)
+    return nav_registry
+
+
+def navigation_list(request):
+    """
+    Helper that returns the admin navigation as list of NavListEntry objects.
+    """
+    return navigation_registry(request).get_navlist(request)

rhodecode/admin/views.py

@@ -0,0 +1,55 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import collections
+import logging
+
+from pylons import tmpl_context as c
+from pyramid.view import view_config
+
+from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator
+from rhodecode.lib.utils import read_opensource_licenses
+
+from .navigation import navigation_list
+
+
+log = logging.getLogger(__name__)
+
+
+class AdminSettingsView(object):
+
+    def __init__(self, context, request):
+        self.request = request
+        self.context = context
+        self.session = request.session
+        self._rhodecode_user = request.user
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
+        route_name='admin_settings_open_source', request_method='GET',
+        renderer='rhodecode:templates/admin/settings/settings.html')
+    def open_source_licenses(self):
+        c.active = 'open_source'
+        c.navlist = navigation_list(self.request)
+        c.opensource_licenses = collections.OrderedDict(
+            sorted(read_opensource_licenses().items(), key=lambda t: t[0]))
+
+        return {}

rhodecode/authentication/tests/conftest.py

@@ -0,0 +1,92 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+import pytest
+
+
+class EnabledAuthPlugin():
+    """
+    Context manager that updates the 'auth_plugins' setting in DB to enable
+    a plugin. Previous setting is restored on exit. The rhodecode auth plugin
+    is also enabled because it is needed to login the test users.
+    """
+
+    def __init__(self, plugin):
+        self.new_value = set([
+            'egg:rhodecode-enterprise-ce#rhodecode',
+            plugin.get_id()
+        ])
+
+    def __enter__(self):
+        from rhodecode.model.settings import SettingsModel
+        self._old_value = SettingsModel().get_auth_plugins()
+        SettingsModel().create_or_update_setting(
+            'auth_plugins', ','.join(self.new_value))
+
+    def __exit__(self, type, value, traceback):
+        from rhodecode.model.settings import SettingsModel
+        SettingsModel().create_or_update_setting(
+            'auth_plugins', ','.join(self._old_value))
+
+
+class DisabledAuthPlugin():
+    """
+    Context manager that updates the 'auth_plugins' setting in DB to disable
+    a plugin. Previous setting is restored on exit.
+    """
+
+    def __init__(self, plugin):
+        self.plugin_id = plugin.get_id()
+
+    def __enter__(self):
+        from rhodecode.model.settings import SettingsModel
+        self._old_value = SettingsModel().get_auth_plugins()
+        new_value = [id_ for id_ in self._old_value if id_ != self.plugin_id]
+        SettingsModel().create_or_update_setting(
+            'auth_plugins', ','.join(new_value))
+
+    def __exit__(self, type, value, traceback):
+        from rhodecode.model.settings import SettingsModel
+        SettingsModel().create_or_update_setting(
+            'auth_plugins', ','.join(self._old_value))
+
+
[e-mail](params=[
+    ('rhodecode.authentication.plugins.auth_crowd', 'egg:rhodecode-enterprise-ce#crowd'),
+    ('rhodecode.authentication.plugins.auth_headers', 'egg:rhodecode-enterprise-ce#headers'),
+    ('rhodecode.authentication.plugins.auth_jasig_cas', 'egg:rhodecode-enterprise-ce#jasig_cas'),
+    ('rhodecode.authentication.plugins.auth_ldap', 'egg:rhodecode-enterprise-ce#ldap'),
+    ('rhodecode.authentication.plugins.auth_pam', 'egg:rhodecode-enterprise-ce#pam'),
+    ('rhodecode.authentication.plugins.auth_rhodecode', 'egg:rhodecode-enterprise-ce#rhodecode'),
+    ('rhodecode.authentication.plugins.auth_token', 'egg:rhodecode-enterprise-ce#token'),
+])
+def auth_plugin(request):
+    """
+    Fixture that provides instance for each authentication plugin. These
+    instances are NOT the instances which are registered to the authentication
+    registry.
+    """
+    from importlib import import_module
+
+    # Create plugin instance.
+    module, plugin_id = request.param
+    plugin_module = import_module(module)
+    return plugin_module.plugin_factory(plugin_id)

rhodecode/authentication/tests/functional/test_settings.py

@@ -0,0 +1,77 b''
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2016-2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+
+import pytest
+
+from rhodecode.authentication.tests.conftest import (
+    EnabledAuthPlugin, DisabledAuthPlugin)
+from rhodecode.config.routing import ADMIN_PREFIX
+
+
[e-mail]('autologin_user', 'app')
+class TestAuthenticationSettings:
+
+    def test_auth_settings_global_view_get(self, app):
+        url = '{prefix}/auth/'.format(prefix=ADMIN_PREFIX)
+        response = app.get(url)
+        assert response.status_code == 200
+
+    def test_plugin_settings_view_get(self, app, auth_plugin):
+        url = '{prefix}/auth/{name}'.format(
+            prefix=ADMIN_PREFIX,
+            name=auth_plugin.name)
+        with EnabledAuthPlugin(auth_plugin):
+            response = app.get(url)
+            assert response.status_code == 200
+
+    def test_plugin_settings_view_post(self, app, auth_plugin, csrf_token):
+        url = '{prefix}/auth/{name}'.format(
+            prefix=ADMIN_PREFIX,
+            name=auth_plugin.name)
+        params = {
+            'enabled': True,
+            'cache_ttl': 0,
+            'csrf_token': csrf_token,
+        }
+        with EnabledAuthPlugin(auth_plugin):
+            response = app.post(url, params=params)
+            assert response.status_code in [200, 302]
+
+    def test_plugin_settings_view_get_404(self, app, auth_plugin):
+        url = '{prefix}/auth/{name}'.format(
+            prefix=ADMIN_PREFIX,
+            name=auth_plugin.name)
+        with DisabledAuthPlugin(auth_plugin):
+            response = app.get(url, status=404)
+            assert response.status_code == 404
+
+    def test_plugin_settings_view_post_404(self, app, auth_plugin, csrf_token):
+        url = '{prefix}/auth/{name}'.format(
+            prefix=ADMIN_PREFIX,
+            name=auth_plugin.name)
+        params = {
+            'enabled': True,
+            'cache_ttl': 0,
+            'csrf_token': csrf_token,
+        }
+        with DisabledAuthPlugin(auth_plugin):
+            response = app.post(url, params=params, status=404)
+            assert response.status_code == 404

rhodecode/authentication/tests/test_auth_plugin_base.py

@@ -0,0 +1,28 b''
+# Copyright (C) 2016  RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+"""
+Checks around the API of the class RhodeCodeAuthPluginBase.
+"""
+
+from rhodecode.authentication.base import RhodeCodeAuthPluginBase
+
+
+def test_str_returns_plugin_id():
+    plugin = RhodeCodeAuthPluginBase(plugin_id='stub_plugin_id')
+    assert str(plugin) == 'stub_plugin_id'

.bumpversion.cfg

@@ -1,6 +1,6 b''
 [bumpversion]
-current_version = 4.1.2
+current_version = 4.2.0
 message = release: Bump version {current_version} to {new_version}
 
 [bumpversion:file:rhodecode/VERSION]
 

.release.cfg

@@ -1,34 +1,28 b''
 [DEFAULT]
 done = false
 
+[task:fixes_on_stable]
+
+[task:changelog_updated]
+
 [task:bump_version]
 done = true
 
-[task:rc_tools_pinned]
-done = true
+[task:generate_api_docs]
+
+[task:updated_translation]
 
-[task:fixes_on_stable]
-done = true
+[release]
+state = in_progress
+version = 4.2.0
+
+[task:rc_tools_pinned]
 
 [task:pip2nix_generated]
-done = true
-
-[task:changelog_updated]
-done = true
-
-[task:generate_api_docs]
-done = true
 
 [task:generate_js_routes]
-done = true
-
-[release]
-state = prepared
-version = 4.1.2
-
-[task:updated_translation]
 
 [task:updated_trial_license]
 
 [task:generate_oss_licenses]
 

README.rst

@@ -1,101 +1,114 b''
 =========
 RhodeCode
 =========
 
 About
 -----
 
 ``RhodeCode`` is a fast and powerful management tool for Mercurial_ and GIT_
 and Subversion_ with a built in push/pull server, full text search,
 pull requests and powerfull code-review system. It works on http/https and
 has a few unique features like:
  - plugable architecture
  - advanced permission system with IP restrictions
  - rich set of authentication plugins including LDAP,
    ActiveDirectory, Atlassian Crowd, Http-Headers, Pam, Token-Auth.
  - live code-review chat
  - full web based file editing
  - unified multi vcs support
  - snippets (gist) system
  - integration with all 3rd party issue trackers
 
 RhodeCode also provides rich API, and multiple event hooks so it's easy
 integrable with existing external systems.
 
 RhodeCode is similar in some respects to gitlab_, github_ or bitbucket_,
 however RhodeCode can be run as standalone hosted application on your own server.
 RhodeCode can be installed on \*nix or Windows systems.
 
 RhodeCode uses `PEP386 versioning <http://www.python.org/dev/peps/pep-0386/>`_
 
 Installation
 ------------
 Please visit https://docs.rhodecode.com/RhodeCode-Control/tasks/install-cli.html
 for more details
 
 
 Source code
 -----------
 
 The latest sources can be obtained from official RhodeCode instance
 https://code.rhodecode.com
 
 
+Contributions
+-------------
+
+RhodeCode is open-source; contributions are welcome!
+
+Please see the contribution documentation inside of the docs folder, which is
+also available at
+https://docs.rhodecode.com/RhodeCode-Enterprise/contributing/contributing.html
+
+For additional information about collaboration tools, our issue tracker,
+licensing, and contribution credit, visit https://rhodecode.com/open-source 
+
+
 RhodeCode Features
 ------------------
 
 Check out all features of RhodeCode at https://rhodecode.com/features
 
 License
 -------
 
 ``RhodeCode`` is dual-licensed with AGPLv3 and commercial license.
 Please see LICENSE.txt file for details.
 
 
 Getting help
 ------------
 
 Listed bellow are various support resources that should help.
 
 .. note::
 
    Please try to read the documentation before posting any issues, especially
    the **troubleshooting section**
 
 - Official issue tracker `RhodeCode Issue tracker <https://issues.rhodecode.com>`_
 
 - Search our community portal `Community portal <https://community.rhodecode.com>`_
 
 - Join #rhodecode on FreeNode (irc.freenode.net)
   or use http://webchat.freenode.net/?channels=rhodecode for web access to irc.
 
 - You can also follow RhodeCode on twitter **@RhodeCode** where we often post
   news and other interesting stuff about RhodeCode.
 
 
 Online documentation
 --------------------
 
 Online documentation for the current version of RhodeCode is available at
  - http://rhodecode.com/docs
 
 You may also build the documentation for yourself - go into ``docs/`` and run::
 
     nix-build default.nix -o result && make clean html
 
 (You need to have sphinx_ installed to build the documentation. If you don't
 have sphinx_ installed you can install it via the command:
 ``pip install sphinx``)
 
 .. _virtualenv: http://pypi.python.org/pypi/virtualenv
 .. _python: http://www.python.org/
 .. _sphinx: http://sphinx.pocoo.org/
 .. _mercurial: http://mercurial.selenic.com/
 .. _bitbucket: http://bitbucket.org/
 .. _github: http://github.com/
 .. _gitlab: http://gitlab.com/
 .. _subversion: http://subversion.tigris.org/
 .. _git: http://git-scm.com/
 .. _celery: http://celeryproject.org/
 .. _vcs: http://pypi.python.org/pypi/vcs

acceptance_tests/README.rst

@@ -1,47 +1,47 b''
 README - Quickstart
 ===================
 
-This folder contains functional tests and the automation of specification
+This folder contains the functional tests and automation of specification
 examples. Details about testing can be found in
 `/docs-internal/testing/index.rst`.
 
 
 Setting up your Rhodecode Enterprise instance
 ---------------------------------------------
 
 The tests will create users and repositories as needed, so you can start with a
 new and empty instance.
 
 Use the following example call for the database setup of Enterprise::
 
    paster setup-rhodecode \
      --user=admin \
      --email=[e-mail] \
      --password=secret \
      --api-key=9999999999999999999999999999999999999999 \
      your-enterprise-config.ini
 
-This way the username, password and auth token of the admin user will match the
+This way the username, password, and auth token of the admin user will match the
 defaults from the test run.
 
 
 Usage
 -----
 
 1. Make sure your Rhodecode Enterprise instance is running at
    http://localhost:5000.
 
 2. Enter `nix-shell` from the acceptance_tests folder::
 
      cd acceptance_tests
-     nix-shell -I ~/dev
+     nix-shell
 
    Make sure that `rcpkgs` and `rcnixpkgs` are available on the nix path.
 
 3. Run the tests::
 
      py.test -c example.ini -vs
 
    The parameter ``-vs`` allows you to see debugging output during the test
    run. Check ``py.test --help`` and the documentation at http://pytest.org to
    learn all details about the test runner.

configs/development.ini

@@ -1,608 +1,612 b''
 ################################################################################
 ################################################################################
 # RhodeCode Enterprise - configuration file                                    #
 # Built-in functions and variables                                             #
 # The %(here)s variable will be replaced with the parent directory of this file#
 #                                                                              #
 ################################################################################
 
 [DEFAULT]
 debug = true
-pdebug = false
 ################################################################################
 ## Uncomment and replace with the email address which should receive          ##
 ## any error reports after an application crash                               ##
 ## Additionally these settings will be used by the RhodeCode mailing system   ##
 ################################################################################
 #email_to = admin@localhost
 #error_email_from = paste_error@localhost
 #app_email_from = rhodecode-noreply@localhost
 #error_message =
 #email_prefix = [RhodeCode]
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 ## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
 #smtp_auth =
 
 [server:main]
 ## COMMON ##
 host = 127.0.0.1
 port = 5000
 
 ##################################
 ##     WAITRESS WSGI SERVER     ##
 ## Recommended for Development  ##
 ##################################
 use = egg:waitress#main
 ## number of worker threads
 threads = 5
 ## MAX BODY SIZE 100GB
 max_request_body_size = 107374182400
 ## Use poll instead of select, fixes file descriptors limits problems.
 ## May not work on old windows systems.
 asyncore_use_poll = true
 
 
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
 ## run with gunicorn --log-config <inifile.ini> --paste <inifile.ini>
 #use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
 ## when this option is set to more than one worker, recommended
 ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
 ## The `instance_id = *` must be set in the [app:main] section below
 #workers = 2
 ## number of threads for each of the worker, must be set to 1 for gevent
 ## generally recommened to be at 1
 #threads = 1
 ## process name
 #proc_name = rhodecode
 ## type of worker class, one of sync, gevent
 ## recommended for bigger setup is using of of other than sync one
 #worker_class = sync
 ## The maximum number of simultaneous clients. Valid only for Gevent
 #worker_connections = 10
 ## max number of requests that worker will handle before being gracefully
 ## restarted, could prevent memory leaks
 #max_requests = 1000
 #max_requests_jitter = 30
 ## amount of time a worker can spend with handling a request before it
 ## gets killed and restarted. Set to 6hrs
 #timeout = 21600
 
 
 ## prefix middleware for RhodeCode, disables force_https flag.
 ## allows to set RhodeCode under a prefix in server.
 ## eg https://server.com/<prefix>. Enable `filter-with =` option below as well.
 #[filter:proxy-prefix]
 #use = egg:PasteDeploy#prefix
 #prefix = /<your-prefix>
 
 [app:main]
 use = egg:rhodecode-enterprise-ce
 ## enable proxy prefix middleware, defined below
 #filter-with = proxy-prefix
 
 # During development the we want to have the debug toolbar enabled
 pyramid.includes =
     pyramid_debugtoolbar
     rhodecode.utils.debugtoolbar
     rhodecode.lib.middleware.request_wrapper
 
 pyramid.reload_templates = true
 
 debugtoolbar.hosts = 0.0.0.0/0
 debugtoolbar.exclude_prefixes =
     /css
     /fonts
     /images
     /js
 
 ## RHODECODE PLUGINS ##
 rhodecode.includes =
     rhodecode.api
 
 
 # api prefix url
 rhodecode.api.url = /_admin/api
 
 
 ## END RHODECODE PLUGINS ##
 
+## encryption key used to encrypt social plugin tokens,
+## remote_urls with credentials etc, if not set it defaults to
+## `beaker.session.secret`
+#rhodecode.encrypted_values.secret =
+
+## decryption strict mode (enabled by default). It controls if decryption raises
+## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
+#rhodecode.encrypted_values.strict = false
+
 full_stack = true
 
 ## Serve static files via RhodeCode, disable to serve them via HTTP server
 static_files = true
 
+# autogenerate javascript routes file on startup
+generate_js_files = false
+
 ## Optional Languages
 ## en(default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ## perform a full repository scan on each server start, this should be
 ## set to false after first startup, to allow faster server restarts.
 startup.import_repos = false
 
 ## Uncomment and set this path to use archive download cache.
 ## Once enabled, generated archives will be cached at this location
 ## and served from the cache during subsequent requests for the same archive of
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
 ## change this to unique ID for security
 app_instance_uuid = rc-production
 
 ## cut off limit for large diffs (size in bytes)
 cut_off_limit_diff = 1024000
 cut_off_limit_file = 256000
 
 ## use cache version of scm repo everywhere
 vcs_full_cache = true
 
 ## force https in RhodeCode, fixes https redirects, assumes it's always https
 ## Normally this is controlled by proper http flags sent from http server
 force_https = false
 
 ## use Strict-Transport-Security headers
 use_htsts = false
 
 ## number of commits stats will parse on each iteration
 commit_parse_limit = 25
 
 ## git rev filter option, --all is the default filter, if you need to
 ## hide all refs in changelog switch this to --branches --tags
 git_rev_filter = --branches --tags
 
 # Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ## RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
 ## url that does rewrites to _admin/gists/<gistid>.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/<gistid>
 gist_alias_url =
 
 ## List of controllers (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
 ## Adding ?auth_token = <token> to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ##
 ## Syntax is <ControllerClass>:<function_pattern>.
 ## To enable access to raw_files put `FilesController:raw`.
 ## To enable access to patches add `ChangesetController:changeset_patch`.
 ## The list should be "," separated and on a single line.
 ##
 ## Recommended controllers to enable:
 #    ChangesetController:changeset_patch,
 #    ChangesetController:changeset_raw,
 #    FilesController:raw,
 #    FilesController:archivefile,
 #    GistsController:*,
 api_access_controllers_whitelist =
 
 ## default encoding used to convert from and to unicode
 ## can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
 ## multiple instances of rhodecode, make sure it's globally unique for
 ## all running rhodecode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
 ## authentication settings. Then you can enable e.g. the internal rhodecode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
 ## egg:rhodecode-enterprise-ce#rhodecode
 ## egg:rhodecode-enterprise-ce#pam
 ## egg:rhodecode-enterprise-ce#ldap
 ## egg:rhodecode-enterprise-ce#jasig_cas
 ## egg:rhodecode-enterprise-ce#headers
 ## egg:rhodecode-enterprise-ce#crowd
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ## alternative return HTTP header for failed authentication. Default HTTP
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ## This will be served instead of default 401 on bad authnetication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
 ## ret_code directly, use 401 initially (Which triggers credentials prompt)
 ## and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ## locking return code. When repository is locked return this HTTP code. 2XX
 ## codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ## allows to change the repository location in settings page
 allow_repo_location_change = true
 
 ## allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ## generated license token, goto license page in RhodeCode settings to obtain
 ## new token
 license_token =
 
 ## supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 ## supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = dev
 
 ## Display extended labs settings
 labs_settings_active = true
 
 ####################################
 ###        CELERY CONFIG        ####
 ####################################
 use_celery = false
 broker.host = localhost
 broker.vhost = rabbitmqhost
 broker.port = 5672
 broker.user = rabbitmq
 broker.password = qweqwe
 
 celery.imports = rhodecode.lib.celerylib.tasks
 
 celery.result.backend = amqp
 celery.result.dburi = amqp://
 celery.result.serialier = json
 
 #celery.send.task.error.emails = true
 #celery.amqp.task.result.expires = 18000
 
 celeryd.concurrency = 2
 #celeryd.log.file = celeryd.log
 celeryd.log.level = debug
 celeryd.max.tasks.per.child = 1
 
 ## tasks will never be sent to the queue, but executed locally instead.
 celery.always.eager = false
 
 ####################################
 ###         BEAKER CACHE        ####
 ####################################
 # default cache dir for templates.  Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk
 cache_dir = %(here)s/data
 
 ## locking and default file storage for Beaker. Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk/cache/beaker_data
 beaker.cache.data_dir = %(here)s/data/cache/beaker_data
 beaker.cache.lock_dir = %(here)s/data/cache/beaker_lock
 
 beaker.cache.regions = super_short_term, short_term, long_term, sql_cache_short, auth_plugins, repo_cache_long
 
 beaker.cache.super_short_term.type = memory
 beaker.cache.super_short_term.expire = 10
 beaker.cache.super_short_term.key_length = 256
 
 beaker.cache.short_term.type = memory
 beaker.cache.short_term.expire = 60
 beaker.cache.short_term.key_length = 256
 
 beaker.cache.long_term.type = memory
 beaker.cache.long_term.expire = 36000
 beaker.cache.long_term.key_length = 256
 
 beaker.cache.sql_cache_short.type = memory
 beaker.cache.sql_cache_short.expire = 10
 beaker.cache.sql_cache_short.key_length = 256
 
 # default is memory cache, configure only if required
 # using multi-node or multi-worker setup
 #beaker.cache.auth_plugins.type = ext:database
 #beaker.cache.auth_plugins.lock_dir = %(here)s/data/cache/auth_plugin_lock
 #beaker.cache.auth_plugins.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.cache.auth_plugins.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.cache.auth_plugins.sa.pool_recycle = 3600
 #beaker.cache.auth_plugins.sa.pool_size = 10
 #beaker.cache.auth_plugins.sa.max_overflow = 0
 
 beaker.cache.repo_cache_long.type = memorylru_base
 beaker.cache.repo_cache_long.max_items = 4096
 beaker.cache.repo_cache_long.expire = 2592000
 
 # default is memorylru_base cache, configure only if required
 # using multi-node or multi-worker setup
 #beaker.cache.repo_cache_long.type = ext:memcached
 #beaker.cache.repo_cache_long.url = localhost:11211
 #beaker.cache.repo_cache_long.expire = 1209600
 #beaker.cache.repo_cache_long.key_length = 256
 
 ####################################
 ###       BEAKER SESSION        ####
 ####################################
 
 ## .session.type is type of storage options for the session, current allowed
 ## types are file, ext:memcached, ext:database, and memory (default).
 beaker.session.type = file
 beaker.session.data_dir = %(here)s/data/sessions/data
 
 ## db based session, fast, and allows easy management over logged in users ##
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = develop-rc-uytcxaz
 beaker.session.lock_dir = %(here)s/data/sessions/lock
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = <key_for_encryption>
 #beaker.session.validate_key = <validation_key>
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 #beaker.session.cookie_path = /<your-prefix>
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
 
 ## auto save the session to not to use .save()
 beaker.session.auto = false
 
 ## default cookie expiration time in seconds, set to `true` to set expire
 ## at browser close
 #beaker.session.cookie_expires = 3600
 
 ###################################
 ## SEARCH INDEXING CONFIGURATION ##
 ###################################
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
 # WHOOSH Backend, doesn't require additional services to run
 # it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
-
 ###################################
 ##       APPENLIGHT CONFIG       ##
 ###################################
 
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
 ## appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
 # used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
 
 ## enables 404 error logging (default False)
 appenlight.report_404 = false
 
 ## time in seconds after request is considered being slow (default 1)
 appenlight.slow_request_time = 1
 
 ## record slow requests in application
 ## (needs to be enabled for slow datastore recording and time tracking)
 appenlight.slow_requests = true
 
 ## enable hooking to application loggers
 appenlight.logging = true
 
 ## minimum log level for log capture
 appenlight.logging.level = WARNING
 
 ## send logs only from erroneous/slow requests
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
 ## list of additonal keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
 ## start with HTTP* this list be extended with additional keywords here
 appenlight.environ_keys_whitelist =
 
 ## list of keywords that should be blanked from request object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always blank keys that contain following words
 ## 'password', 'passwd', 'pwd', 'auth_tkt', 'secret', 'csrf'
 ## this list be extended with additional keywords set here
 appenlight.request_keys_blacklist =
 
 ## list of namespaces that should be ignores when gathering log entries
 ## can be string with comma separated list of namespaces
 ## (by default the client ignores own entries: appenlight_client.client)
 appenlight.log_namespace_blacklist =
 
 
 ################################################################################
 ## WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*  ##
 ## Debug mode will enable the interactive debugging tool, allowing ANYONE to  ##
 ## execute malicious code after an exception is raised.                       ##
 ################################################################################
 #set debug = false
 
 
 ##############
 ## STYLING  ##
 ##############
 debug_style = true
 
 #########################################################
 ### DB CONFIGS - EACH DB WILL HAVE IT'S OWN CONFIG    ###
 #########################################################
 sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
 
 # see sqlalchemy docs for other advanced settings
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
 ## recycle the connections after this ammount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
 ## the number of connections to keep open inside the connection pool.
 ## 0 indicates no limit
 #sqlalchemy.db1.pool_size = 5
 
 ## the number of connections to allow in connection pool "overflow", that is
 ## connections that can be opened above and beyond the pool_size setting,
 ## which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 
 ##################
 ### VCS CONFIG ###
 ##################
 vcs.server.enable = true
 vcs.server = localhost:9900
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
 ## `pyro4` - using pyro4 server
 ## `http` - using http-rpc backend
 #vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
 ## `pyro4` - using pyro4 server
 ## `rhodecode.lib.middleware.utils.scm_app_http` - Http based, recommended
 ## `vcsserver.scm_app` - internal app (EE only)
 #vcs.scm_app_implementation = rhodecode.lib.middleware.utils.scm_app_http
 
 ## Push/Pull operations hooks protocol, available options are:
 ## `pyro4` - using pyro4 server
 ## `http` - using http-rpc backend
 #vcs.hooks.protocol = http
 
 vcs.server.log_level = debug
 ## Start VCSServer with this instance as a subprocess, usefull for development
 vcs.start_server = true
 vcs.backends = hg, git, svn
 vcs.connection_timeout = 3600
 ## Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ## Available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible
 #vcs.svn.compatible_version = pre-1.8-compatible
 
 ################################
 ### LOGGING CONFIGURATION   ####
 ################################
 [loggers]
-keys = root, routes, rhodecode, sqlalchemy, beaker, pyro4, templates, whoosh_indexer
+keys = root, routes, rhodecode, sqlalchemy, beaker, pyro4, templates
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, color_formatter, color_formatter_sql
 
 #############
 ## LOGGERS ##
 #############
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_routes]
 level = DEBUG
 handlers =
 qualname = routes.middleware
 ## "level = DEBUG" logs the route matched and routing variables.
 propagate = 1
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_pyro4]
 level = DEBUG
 handlers =
 qualname = Pyro4
 propagate = 1
 
 [logger_templates]
 level = INFO
 handlers =
 qualname = pylons.templating
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
-[logger_whoosh_indexer]
-level = DEBUG
-handlers =
-qualname = whoosh_indexer
-propagate = 1
-
 ##############
 ## HANDLERS ##
 ##############
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr,)
 level = DEBUG
 formatter = color_formatter
 
 [handler_console_sql]
 class = StreamHandler
 args = (sys.stderr,)
 level = DEBUG
 formatter = color_formatter_sql
 
 ################
 ## FORMATTERS ##
 ################
 
 [formatter_generic]
 class = rhodecode.lib.logging_formatter.Pyro4AwareFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S

configs/production.ini

@@ -1,577 +1,581 b''
 ################################################################################
 ################################################################################
 # RhodeCode Enterprise - configuration file                                    #
 # Built-in functions and variables                                             #
 # The %(here)s variable will be replaced with the parent directory of this file#
 #                                                                              #
 ################################################################################
 
 [DEFAULT]
 debug = true
-pdebug = false
 ################################################################################
 ## Uncomment and replace with the email address which should receive          ##
 ## any error reports after an application crash                               ##
 ## Additionally these settings will be used by the RhodeCode mailing system   ##
 ################################################################################
 #email_to = admin@localhost
 #error_email_from = paste_error@localhost
 #app_email_from = rhodecode-noreply@localhost
 #error_message =
 #email_prefix = [RhodeCode]
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 ## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
 #smtp_auth =
 
 [server:main]
 ## COMMON ##
 host = 127.0.0.1
 port = 5000
 
 ##################################
 ##     WAITRESS WSGI SERVER     ##
 ## Recommended for Development  ##
 ##################################
 #use = egg:waitress#main
 ## number of worker threads
 #threads = 5
 ## MAX BODY SIZE 100GB
 #max_request_body_size = 107374182400
 ## Use poll instead of select, fixes file descriptors limits problems.
 ## May not work on old windows systems.
 #asyncore_use_poll = true
 
 
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
 ## run with gunicorn --log-config <inifile.ini> --paste <inifile.ini>
 use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
 ## when this option is set to more than one worker, recommended
 ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
 ## The `instance_id = *` must be set in the [app:main] section below
 workers = 2
 ## number of threads for each of the worker, must be set to 1 for gevent
 ## generally recommened to be at 1
 #threads = 1
 ## process name
 proc_name = rhodecode
 ## type of worker class, one of sync, gevent
 ## recommended for bigger setup is using of of other than sync one
 worker_class = sync
 ## The maximum number of simultaneous clients. Valid only for Gevent
 #worker_connections = 10
 ## max number of requests that worker will handle before being gracefully
 ## restarted, could prevent memory leaks
 max_requests = 1000
 max_requests_jitter = 30
 ## amount of time a worker can spend with handling a request before it
 ## gets killed and restarted. Set to 6hrs
 timeout = 21600
 
 
 ## prefix middleware for RhodeCode, disables force_https flag.
 ## allows to set RhodeCode under a prefix in server.
 ## eg https://server.com/<prefix>. Enable `filter-with =` option below as well.
 #[filter:proxy-prefix]
 #use = egg:PasteDeploy#prefix
 #prefix = /<your-prefix>
 
 [app:main]
 use = egg:rhodecode-enterprise-ce
 ## enable proxy prefix middleware, defined below
 #filter-with = proxy-prefix
 
+## encryption key used to encrypt social plugin tokens,
+## remote_urls with credentials etc, if not set it defaults to
+## `beaker.session.secret`
+#rhodecode.encrypted_values.secret =
+
+## decryption strict mode (enabled by default). It controls if decryption raises
+## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
+#rhodecode.encrypted_values.strict = false
+
 full_stack = true
 
 ## Serve static files via RhodeCode, disable to serve them via HTTP server
 static_files = true
 
+# autogenerate javascript routes file on startup
+generate_js_files = false
+
 ## Optional Languages
 ## en(default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ## perform a full repository scan on each server start, this should be
 ## set to false after first startup, to allow faster server restarts.
 startup.import_repos = false
 
 ## Uncomment and set this path to use archive download cache.
 ## Once enabled, generated archives will be cached at this location
 ## and served from the cache during subsequent requests for the same archive of
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
 ## change this to unique ID for security
 app_instance_uuid = rc-production
 
 ## cut off limit for large diffs (size in bytes)
 cut_off_limit_diff = 1024000
 cut_off_limit_file = 256000
 
 ## use cache version of scm repo everywhere
 vcs_full_cache = true
 
 ## force https in RhodeCode, fixes https redirects, assumes it's always https
 ## Normally this is controlled by proper http flags sent from http server
 force_https = false
 
 ## use Strict-Transport-Security headers
 use_htsts = false
 
 ## number of commits stats will parse on each iteration
 commit_parse_limit = 25
 
 ## git rev filter option, --all is the default filter, if you need to
 ## hide all refs in changelog switch this to --branches --tags
 git_rev_filter = --branches --tags
 
 # Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ## RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
 ## url that does rewrites to _admin/gists/<gistid>.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/<gistid>
 gist_alias_url =
 
 ## List of controllers (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
 ## Adding ?auth_token = <token> to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ##
 ## Syntax is <ControllerClass>:<function_pattern>.
 ## To enable access to raw_files put `FilesController:raw`.
 ## To enable access to patches add `ChangesetController:changeset_patch`.
 ## The list should be "," separated and on a single line.
 ##
 ## Recommended controllers to enable:
 #    ChangesetController:changeset_patch,
 #    ChangesetController:changeset_raw,
 #    FilesController:raw,
 #    FilesController:archivefile,
 #    GistsController:*,
 api_access_controllers_whitelist =
 
 ## default encoding used to convert from and to unicode
 ## can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
 ## multiple instances of rhodecode, make sure it's globally unique for
 ## all running rhodecode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
 ## authentication settings. Then you can enable e.g. the internal rhodecode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
 ## egg:rhodecode-enterprise-ce#rhodecode
 ## egg:rhodecode-enterprise-ce#pam
 ## egg:rhodecode-enterprise-ce#ldap
 ## egg:rhodecode-enterprise-ce#jasig_cas
 ## egg:rhodecode-enterprise-ce#headers
 ## egg:rhodecode-enterprise-ce#crowd
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ## alternative return HTTP header for failed authentication. Default HTTP
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ## This will be served instead of default 401 on bad authnetication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
 ## ret_code directly, use 401 initially (Which triggers credentials prompt)
 ## and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ## locking return code. When repository is locked return this HTTP code. 2XX
 ## codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ## allows to change the repository location in settings page
 allow_repo_location_change = true
 
 ## allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ## generated license token, goto license page in RhodeCode settings to obtain
 ## new token
 license_token =
 
 ## supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 ## supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = prod
 
 ## Display extended labs settings
 labs_settings_active = true
 
 ####################################
 ###        CELERY CONFIG        ####
 ####################################
 use_celery = false
 broker.host = localhost
 broker.vhost = rabbitmqhost
 broker.port = 5672
 broker.user = rabbitmq
 broker.password = qweqwe
 
 celery.imports = rhodecode.lib.celerylib.tasks
 
 celery.result.backend = amqp
 celery.result.dburi = amqp://
 celery.result.serialier = json
 
 #celery.send.task.error.emails = true
 #celery.amqp.task.result.expires = 18000
 
 celeryd.concurrency = 2
 #celeryd.log.file = celeryd.log
 celeryd.log.level = debug
 celeryd.max.tasks.per.child = 1
 
 ## tasks will never be sent to the queue, but executed locally instead.
 celery.always.eager = false
 
 ####################################
 ###         BEAKER CACHE        ####
 ####################################
 # default cache dir for templates.  Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk
 cache_dir = %(here)s/data
 
 ## locking and default file storage for Beaker. Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk/cache/beaker_data
 beaker.cache.data_dir = %(here)s/data/cache/beaker_data
 beaker.cache.lock_dir = %(here)s/data/cache/beaker_lock
 
 beaker.cache.regions = super_short_term, short_term, long_term, sql_cache_short, auth_plugins, repo_cache_long
 
 beaker.cache.super_short_term.type = memory
 beaker.cache.super_short_term.expire = 10
 beaker.cache.super_short_term.key_length = 256
 
 beaker.cache.short_term.type = memory
 beaker.cache.short_term.expire = 60
 beaker.cache.short_term.key_length = 256
 
 beaker.cache.long_term.type = memory
 beaker.cache.long_term.expire = 36000
 beaker.cache.long_term.key_length = 256
 
 beaker.cache.sql_cache_short.type = memory
 beaker.cache.sql_cache_short.expire = 10
 beaker.cache.sql_cache_short.key_length = 256
 
 # default is memory cache, configure only if required
 # using multi-node or multi-worker setup
 #beaker.cache.auth_plugins.type = ext:database
 #beaker.cache.auth_plugins.lock_dir = %(here)s/data/cache/auth_plugin_lock
 #beaker.cache.auth_plugins.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.cache.auth_plugins.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.cache.auth_plugins.sa.pool_recycle = 3600
 #beaker.cache.auth_plugins.sa.pool_size = 10
 #beaker.cache.auth_plugins.sa.max_overflow = 0
 
 beaker.cache.repo_cache_long.type = memorylru_base
 beaker.cache.repo_cache_long.max_items = 4096
 beaker.cache.repo_cache_long.expire = 2592000
 
 # default is memorylru_base cache, configure only if required
 # using multi-node or multi-worker setup
 #beaker.cache.repo_cache_long.type = ext:memcached
 #beaker.cache.repo_cache_long.url = localhost:11211
 #beaker.cache.repo_cache_long.expire = 1209600
 #beaker.cache.repo_cache_long.key_length = 256
 
 ####################################
 ###       BEAKER SESSION        ####
 ####################################
 
 ## .session.type is type of storage options for the session, current allowed
 ## types are file, ext:memcached, ext:database, and memory (default).
 beaker.session.type = file
 beaker.session.data_dir = %(here)s/data/sessions/data
 
 ## db based session, fast, and allows easy management over logged in users ##
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = production-rc-uytcxaz
 beaker.session.lock_dir = %(here)s/data/sessions/lock
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = <key_for_encryption>
 #beaker.session.validate_key = <validation_key>
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 #beaker.session.cookie_path = /<your-prefix>
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
 
 ## auto save the session to not to use .save()
 beaker.session.auto = false
 
 ## default cookie expiration time in seconds, set to `true` to set expire
 ## at browser close
 #beaker.session.cookie_expires = 3600
 
 ###################################
 ## SEARCH INDEXING CONFIGURATION ##
 ###################################
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
 # WHOOSH Backend, doesn't require additional services to run
 # it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
-
 ###################################
 ##       APPENLIGHT CONFIG       ##
 ###################################
 
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
 ## appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
 # used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
 
 ## enables 404 error logging (default False)
 appenlight.report_404 = false
 
 ## time in seconds after request is considered being slow (default 1)
 appenlight.slow_request_time = 1
 
 ## record slow requests in application
 ## (needs to be enabled for slow datastore recording and time tracking)
 appenlight.slow_requests = true
 
 ## enable hooking to application loggers
 appenlight.logging = true
 
 ## minimum log level for log capture
 appenlight.logging.level = WARNING
 
 ## send logs only from erroneous/slow requests
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
 ## list of additonal keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
 ## start with HTTP* this list be extended with additional keywords here
 appenlight.environ_keys_whitelist =
 
 ## list of keywords that should be blanked from request object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always blank keys that contain following words
 ## 'password', 'passwd', 'pwd', 'auth_tkt', 'secret', 'csrf'
 ## this list be extended with additional keywords set here
 appenlight.request_keys_blacklist =
 
 ## list of namespaces that should be ignores when gathering log entries
 ## can be string with comma separated list of namespaces
 ## (by default the client ignores own entries: appenlight_client.client)
 appenlight.log_namespace_blacklist =
 
 
 ################################################################################
 ## WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*  ##
 ## Debug mode will enable the interactive debugging tool, allowing ANYONE to  ##
 ## execute malicious code after an exception is raised.                       ##
 ################################################################################
 set debug = false
 
 
 #########################################################
 ### DB CONFIGS - EACH DB WILL HAVE IT'S OWN CONFIG    ###
 #########################################################
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
 
 # see sqlalchemy docs for other advanced settings
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
 ## recycle the connections after this ammount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
 ## the number of connections to keep open inside the connection pool.
 ## 0 indicates no limit
 #sqlalchemy.db1.pool_size = 5
 
 ## the number of connections to allow in connection pool "overflow", that is
 ## connections that can be opened above and beyond the pool_size setting,
 ## which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 
 ##################
 ### VCS CONFIG ###
 ##################
 vcs.server.enable = true
 vcs.server = localhost:9900
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
 ## `pyro4` - using pyro4 server
 ## `http` - using http-rpc backend
 #vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
 ## `pyro4` - using pyro4 server
 ## `rhodecode.lib.middleware.utils.scm_app_http` - Http based, recommended
 ## `vcsserver.scm_app` - internal app (EE only)
 #vcs.scm_app_implementation = rhodecode.lib.middleware.utils.scm_app_http
 
 ## Push/Pull operations hooks protocol, available options are:
 ## `pyro4` - using pyro4 server
 ## `http` - using http-rpc backend
 #vcs.hooks.protocol = http
 
 vcs.server.log_level = info
 ## Start VCSServer with this instance as a subprocess, usefull for development
 vcs.start_server = false
 vcs.backends = hg, git, svn
 vcs.connection_timeout = 3600
 ## Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ## Available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible
 #vcs.svn.compatible_version = pre-1.8-compatible
 
 ################################
 ### LOGGING CONFIGURATION   ####
 ################################
 [loggers]
-keys = root, routes, rhodecode, sqlalchemy, beaker, pyro4, templates, whoosh_indexer
+keys = root, routes, rhodecode, sqlalchemy, beaker, pyro4, templates
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, color_formatter, color_formatter_sql
 
 #############
 ## LOGGERS ##
 #############
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_routes]
 level = DEBUG
 handlers =
 qualname = routes.middleware
 ## "level = DEBUG" logs the route matched and routing variables.
 propagate = 1
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_pyro4]
 level = DEBUG
 handlers =
 qualname = Pyro4
 propagate = 1
 
 [logger_templates]
 level = INFO
 handlers =
 qualname = pylons.templating
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
-[logger_whoosh_indexer]
-level = DEBUG
-handlers =
-qualname = whoosh_indexer
-propagate = 1
-
 ##############
 ## HANDLERS ##
 ##############
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr,)
 level = INFO
 formatter = generic
 
 [handler_console_sql]
 class = StreamHandler
 args = (sys.stderr,)
 level = WARN
 formatter = generic
 
 ################
 ## FORMATTERS ##
 ################
 
 [formatter_generic]
 class = rhodecode.lib.logging_formatter.Pyro4AwareFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S

default.nix

@@ -1,219 +1,227 b''
 # Nix environment for the community edition
 #
 # This shall be as lean as possible, just producing the Enterprise
 # derivation. For advanced tweaks to pimp up the development environment we use
 # "shell.nix" so that it does not have to clutter this file.
 
 { pkgs ? (import <nixpkgs> {})
 , pythonPackages ? "python27Packages"
 , pythonExternalOverrides ? self: super: {}
 , doCheck ? true
 }:
 
 let pkgs_ = pkgs; in
 
 let
   pkgs = pkgs_.overridePackages (self: super: {
     # Override subversion derivation to
     #  - activate python bindings
     #  - set version to 1.8
     subversion = super.subversion18.override {
        httpSupport = true;
        pythonBindings = true;
        python = self.python27Packages.python;
     };
   });
 
   inherit (pkgs.lib) fix extends;
 
   basePythonPackages = with builtins; if isAttrs pythonPackages
     then pythonPackages
     else getAttr pythonPackages pkgs;
 
   elem = builtins.elem;
   basename = path: with pkgs.lib; last (splitString "/" path);
   startsWith = prefix: full: let
     actualPrefix = builtins.substring 0 (builtins.stringLength prefix) full;
   in actualPrefix == prefix;
 
   src-filter = path: type: with pkgs.lib;
     let
       ext = last (splitString "." path);
     in
       !elem (basename path) [
         ".git" ".hg" "__pycache__" ".eggs" "node_modules"
         "build" "data" "tmp"] &&
       !elem ext ["egg-info" "pyc"] &&
       !startsWith "result" path;
 
+  sources = pkgs.config.rc.sources or {};
   rhodecode-enterprise-ce-src = builtins.filterSource src-filter ./.;
 
   # Load the generated node packages
   nodePackages = pkgs.callPackage "${pkgs.path}/pkgs/top-level/node-packages.nix" rec {
     self = nodePackages;
     generated = pkgs.callPackage ./pkgs/node-packages.nix { inherit self; };
   };
 
   # TODO: Should be taken automatically out of the generates packages.
   # apps.nix has one solution for this, although I'd prefer to have the deps
   # from package.json mapped in here.
   nodeDependencies = with nodePackages; [
     grunt
     grunt-contrib-concat
     grunt-contrib-jshint
     grunt-contrib-less
     grunt-contrib-watch
     jshint
   ];
 
   pythonGeneratedPackages = self: basePythonPackages.override (a: {
     inherit self;
   })
   // (scopedImport {
     self = self;
     super = basePythonPackages;
     inherit pkgs;
     inherit (pkgs) fetchurl fetchgit;
   } ./pkgs/python-packages.nix);
 
   pythonOverrides = import ./pkgs/python-packages-overrides.nix {
     inherit
       basePythonPackages
       pkgs;
   };
 
   pythonLocalOverrides = self: super: {
     rhodecode-enterprise-ce =
       let
         version = builtins.readFile ./rhodecode/VERSION;
         linkNodeModules = ''
           echo "Link node packages"
           # TODO: check if this adds stuff as a dependency, closure size
           rm -fr node_modules
           mkdir -p node_modules
           ${pkgs.lib.concatMapStrings (dep: ''
             ln -sfv ${dep}/lib/node_modules/${dep.pkgName} node_modules/
           '') nodeDependencies}
           echo "DONE: Link node packages"
         '';
       in super.rhodecode-enterprise-ce.override (attrs: {
 
-      inherit doCheck;
+      inherit
+        doCheck
+        version;
       name = "rhodecode-enterprise-ce-${version}";
-      version = version;
+      releaseName = "RhodeCodeEnterpriseCE-${version}";
       src = rhodecode-enterprise-ce-src;
 
       buildInputs =
         attrs.buildInputs ++
         (with self; [
           pkgs.nodePackages.grunt-cli
           pkgs.subversion
           pytest-catchlog
-          rc_testdata
+          rhodecode-testdata
         ]);
 
       propagatedBuildInputs = attrs.propagatedBuildInputs ++ (with self; [
         rhodecode-tools
       ]);
 
       # TODO: johbo: Make a nicer way to expose the parts. Maybe
       # pkgs/default.nix?
       passthru = {
         inherit
           pythonLocalOverrides
           myPythonPackagesUnfix;
         pythonPackages = self;
       };
 
       LC_ALL = "en_US.UTF-8";
       LOCALE_ARCHIVE =
         if pkgs.stdenv ? glibc
         then "${pkgs.glibcLocales}/lib/locale/locale-archive"
         else "";
 
       # Somewhat snappier setup of the development environment
       # TODO: move into shell.nix
       # TODO: think of supporting a stable path again, so that multiple shells
       #       can share it.
       shellHook = ''
         tmp_path=$(mktemp -d)
         export PATH="$tmp_path/bin:$PATH"
         export PYTHONPATH="$tmp_path/${self.python.sitePackages}:$PYTHONPATH"
         mkdir -p $tmp_path/${self.python.sitePackages}
         python setup.py develop --prefix $tmp_path --allow-hosts ""
       '' + linkNodeModules;
 
       preCheck = ''
         export PATH="$out/bin:$PATH"
       '';
 
       postCheck = ''
         rm -rf $out/lib/${self.python.libPrefix}/site-packages/pytest_pylons
         rm -rf $out/lib/${self.python.libPrefix}/site-packages/rhodecode/tests
       '';
 
       preBuild = linkNodeModules + ''
         grunt
         rm -fr node_modules
       '';
 
       postInstall = ''
         # python based programs need to be wrapped
         ln -s ${self.supervisor}/bin/supervisor* $out/bin/
         ln -s ${self.gunicorn}/bin/gunicorn $out/bin/
         ln -s ${self.PasteScript}/bin/paster $out/bin/
         ln -s ${self.pyramid}/bin/* $out/bin/  #*/
 
         # rhodecode-tools
         # TODO: johbo: re-think this. Do the tools import anything from enterprise?
         ln -s ${self.rhodecode-tools}/bin/rhodecode-* $out/bin/
 
         # note that condition should be restricted when adding further tools
         for file in $out/bin/*; do  #*/
           wrapProgram $file \
               --prefix PYTHONPATH : $PYTHONPATH \
               --prefix PATH : $PATH \
               --set PYTHONHASHSEED random
         done
 
         mkdir $out/etc
         cp configs/production.ini $out/etc
 
         echo "Writing meta information for rccontrol to nix-support/rccontrol"
         mkdir -p $out/nix-support/rccontrol
         cp -v rhodecode/VERSION $out/nix-support/rccontrol/version
         echo "DONE: Meta information for rccontrol written"
 
         # TODO: johbo: Make part of ac-tests
         if [ ! -f rhodecode/public/js/scripts.js ]; then
           echo "Missing scripts.js"
           exit 1
         fi
         if [ ! -f rhodecode/public/css/style.css ]; then
           echo "Missing style.css"
           exit 1
         fi
       '';
 
     });
 
-    rc_testdata = self.buildPythonPackage rec {
-      name = "rc_testdata-0.7.0";
-      src = pkgs.fetchhg {
-        url = "https://code.rhodecode.com/upstream/rc_testdata";
-        rev = "v0.7.0";
-        sha256 = "0w3z0zn8lagr707v67lgys23sl6pbi4xg7pfvdbw58h3q384h6rx";
-      };
+    rhodecode-testdata = import "${rhodecode-testdata-src}/default.nix" {
+    inherit
+      doCheck
+      pkgs
+      pythonPackages;
     };
 
   };
 
+  rhodecode-testdata-src = sources.rhodecode-testdata or (
+    pkgs.fetchhg {
+      url = "https://code.rhodecode.com/upstream/rc_testdata";
+      rev = "v0.8.0";
+      sha256 = "0hy1ba134rq2f9si85yx7j4qhc9ky0hjzdk553s3q026i7km809m";
+  });
+
   # Apply all overrides and fix the final package set
   myPythonPackagesUnfix =
     (extends pythonExternalOverrides
     (extends pythonLocalOverrides
     (extends pythonOverrides
              pythonGeneratedPackages)));
   myPythonPackages = (fix myPythonPackagesUnfix);
 
 in myPythonPackages.rhodecode-enterprise-ce

docs/admin/enable-debug.rst

@@ -1,136 +1,130 b''
 .. _debug-mode:
 
 Enabling Debug Mode
 -------------------
 
 To enable debug mode on a |RCE| instance you need to set the debug property
 in the :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file. To
 do this, use the following steps
 
 1. Open the file and set the ``debug`` line to ``true``
 2. Restart you instance using the ``rccontrol restart`` command,
    see the following example:
 
 You can also set the log level, the follow are the valid options;
 ``debug``, ``info``, ``warning``, or ``fatal``.
 
 .. code-block:: ini
 
     [DEFAULT]
     debug = true
     pdebug = false
 
 .. code-block:: bash
 
     # Restart your instance
     $ rccontrol restart enterprise-1
     Instance "enterprise-1" successfully stopped.
     Instance "enterprise-1" successfully started.
 
 Debug and Logging Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Further debugging and logging settings can also be set in the
 :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
 
 In the logging section, the various packages that run with |RCE| can have
 different debug levels set. If you want to increase the logging level change
 ``level = DEBUG`` line to one of the valid options.
 
 You also need to change the log level for handlers. See the example
 ``##handler`` section below. The ``handler`` level takes the same options as
 the ``debug`` level.
 
 .. code-block:: ini
 
     ################################
     ### LOGGING CONFIGURATION   ####
     ################################
     [loggers]
-    keys = root, routes, rhodecode, sqlalchemy, beaker, pyro4, templates, whoosh_indexer
+    keys = root, routes, rhodecode, sqlalchemy, beaker, pyro4, templates
 
     [handlers]
     keys = console, console_sql, file, file_rotating
 
     [formatters]
     keys = generic, color_formatter, color_formatter_sql
 
     #############
     ## LOGGERS ##
     #############
     [logger_root]
     level = NOTSET
     handlers = console
 
     [logger_routes]
     level = DEBUG
     handlers =
     qualname = routes.middleware
     ## "level = DEBUG" logs the route matched and routing variables.
     propagate = 1
 
     [logger_beaker]
     level = DEBUG
     handlers =
     qualname = beaker.container
     propagate = 1
 
     [logger_pyro4]
     level = DEBUG
     handlers =
     qualname = Pyro4
     propagate = 1
 
     [logger_templates]
     level = INFO
     handlers =
     qualname = pylons.templating
     propagate = 1
 
     [logger_rhodecode]
     level = DEBUG
     handlers =
     qualname = rhodecode
     propagate = 1
 
     [logger_sqlalchemy]
     level = INFO
     handlers = console_sql
     qualname = sqlalchemy.engine
     propagate = 0
 
-    [logger_whoosh_indexer]
-    level = DEBUG
-    handlers =
-    qualname = whoosh_indexer
-    propagate = 1
-
     ##############
     ## HANDLERS ##
     ##############
 
     [handler_console]
     class = StreamHandler
     args = (sys.stderr,)
     level = INFO
     formatter = generic
 
     [handler_console_sql]
     class = StreamHandler
     args = (sys.stderr,)
     level = WARN
     formatter = generic
 
     [handler_file]
     class = FileHandler
     args = ('rhodecode.log', 'a',)
     level = INFO
     formatter = generic
 
     [handler_file_rotating]
     class = logging.handlers.TimedRotatingFileHandler
     # 'D', 5 - rotate every 5days
     # you can set 'h', 'midnight'
     args = ('rhodecode.log', 'D', 5, 10,)
     level = INFO
     formatter = generic

docs/contributing/contributing.rst

@@ -1,19 +1,20 b''
 .. _contributing:
 
 Contributing to RhodeCode
 =========================
 
 
 
-Welcome to contribution guides and development docs of RhodeCode.
+Welcome to the contribution guides and development docs of RhodeCode.
 
 
 
 .. toctree::
    :maxdepth: 1
 
+   overview
    testing/index
    dev-setup
    db-schema
    dev-settings
    api

docs/contributing/db-schema.rst

@@ -1,52 +1,52 b''
 =======================
 DB Schema and Migration
 =======================
 
-To create or alter tables in the database it's necessary to change a couple of
+To create or alter tables in the database, it's necessary to change a couple of
 files, apart from configuring the settings pointing to the latest database
 schema.
 
 
 Database Model and ORM
 ----------------------
 
 On ``rhodecode.model.db`` you will find the database definition of all tables and 
-fields. Any fresh install database will be correctly created by the definitions 
-here. So, any change to this files will affect the tests without having to change
+fields. Any freshly installed database will be correctly created by the definitions 
+here. So, any change to this file will affect the tests without having to change
 any other file.
 
-A second layer are the businness classes that are inside ``rhodecode.model``. 
+A second layer are the business classes inside ``rhodecode.model``. 
 
 
 Database Migration
 ------------------
 
 Three files play a role when creating database migrations:
 
     * Database schema inside ``rhodecode.lib.dbmigrate``
     * Database version inside ``rhodecode.lib.dbmigrate`` 
     * Configuration ``__dbversion__`` at ``rhodecode.__init__``
 
 
 Schema is a snapshot of the database version BEFORE the migration. So, it's
 the initial state before any changes were added. The name convention is
-the latest release version where the snapshot were created, and not the 
+the latest release version where the snapshot was created, and not the 
 target version of this code.
 
 Version is the method that will define how to UPGRADE/DOWNGRADE the database.
 
 ``rhodecode.__init__`` contains only a variable that defines up to which version of 
 the database will be used to upgrade. Eg.: ``__dbversion__ = 45``
 
 
 For examples on how to create those files, please see the existing code.
 
 
 Migration Command
 ^^^^^^^^^^^^^^^^^
 
-After you changed the database ORM and migration files, you can run::
+After you've changed the database ORM and migration files, you can run::
 
    paster upgrade-db <ini-file> 
 
-And the database will be upgraded up to the version defined in the ``__init__`` file.
  No newline at end of file
+The database will be upgraded up to the version defined in the ``__init__`` file.
  No newline at end of file

docs/contributing/dev-settings.rst

@@ -1,46 +1,46 b''
 
 ==========================
  Settings for Development
 ==========================
 
 
 We have a few settings which are intended to be used only for development
 purposes. This section contains an overview of them.
 
 
 
 `debug_style`
 =============
 
 Enables the section "Style" in the application. This section provides an
-overview of all components which are found in the frontend style of the
+overview of all components which are found in the frontend of the
 application.
 
 
 
 `vcs.start_server`
 ==================
 
 Starts the server as a subprocess while the system comes up. Intended usage is
 to ease development.
 
 
 
 `[logging]`
 ===========
 
-Use this to configure loggig to your current needs. The documentation of
-Python's `logging` module explains all details. The following snippets are
-useful for day to day development work.
+Use this to configure logging to your current needs. The documentation of
+Python's `logging` module explains all of the details. The following snippets
+are useful for day to day development work.
 
 
 Mute SQL output
 ---------------
 
 They come out of the package `sqlalchemy.engine`::
 
   [logger_sqlalchemy]
   level = WARNING
   handlers = console_sql
   qualname = sqlalchemy.engine
   propagate = 0

docs/contributing/dev-setup.rst

@@ -1,141 +1,144 b''
+.. _dev-setup:
 
 ===================
  Development setup
 ===================
 
 
 RhodeCode Enterprise runs inside a Nix managed environment. This ensures build
 environment dependencies are correctly declared and installed during setup.
 It also enables atomic upgrades, rollbacks, and multiple instances of RhodeCode
-Enterprise for efficient cluster management.
+Enterprise running with isolation.
 
-To set up RhodeCode Enterprise inside the Nix environment use the following steps:
+To set up RhodeCode Enterprise inside the Nix environment, use the following steps:
 
 
 
 Setup Nix Package Manager
 -------------------------
 
-To install the Nix Package Manager please run::
+To install the Nix Package Manager, please run::
 
    $ curl https://nixos.org/nix/install | sh
 
-or go to https://nixos.org/nix/ and follow their installation instructions.
-Once this is correctly set up on your system you should be able to use the
+or go to https://nixos.org/nix/ and follow the installation instructions.
+Once this is correctly set up on your system, you should be able to use the
 following commands:
 
 * `nix-env`
 
 * `nix-shell`
 
 
 .. tip::
 
    Update your channels frequently by running ``nix-channel --upgrade``.
 
 
-Switch nix to latest STABLE channel
------------------------------------
+Switch nix to the latest STABLE channel
+---------------------------------------
 
 run::
 
    nix-channel --add https://nixos.org/channels/nixos-16.03 nixpkgs
 
 Followed by::
 
    nix-channel --update
 
 
 Clone the required repositories
 -------------------------------
 
-After Nix is set up, clone the RhodeCode Enterprise Community Edition, and
+After Nix is set up, clone the RhodeCode Enterprise Community Edition and
 RhodeCode VCSServer repositories into the same directory.
 To do this, use the following example::
 
     mkdir rhodecode-develop && cd rhodecode-develop
     hg clone https://code.rhodecode.com/rhodecode-enterprise-ce
     hg clone https://code.rhodecode.com/rhodecode-vcsserver
 
 .. note::
 
-   If you cannot clone the repository, please request read permissions.
+   If you cannot clone the repository, please request read permissions
+   via [e-mail]
 
 
 
 Enter the Development Shell
 ---------------------------
 
-The final step is to start into the development shell. To do this run the
+The final step is to start the development shell. To do this, run the
 following command from inside the cloned repository::
 
    cd ~/rhodecode-enterprise-ce
-   nix-shell --arg dev true
+   nix-shell
 
 .. note::
 
    On the first run, this will take a while to download and optionally compile
-   a few things. The next runs of it will be faster.
+   a few things. The following runs will be faster. The development shell works
+   fine on MacOS and Linux platforms.
 
 
 
 Creating a Development Configuration
 ------------------------------------
 
 To create a development environment for RhodeCode Enterprise,
 use the following steps:
 
 1. Create a copy of `~/rhodecode-enterprise-ce/configs/development.ini`
 2. Adjust the configuration settings to your needs
 
    .. note::
 
-      It is recommended to call it `dev.ini`.
+      It is recommended to use the name `dev.ini`.
 
 
 Setup the Development Database
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-To create a development database use the following example. This is a one
+To create a development database, use the following example. This is a one
 time operation::
 
     paster setup-rhodecode dev.ini \
         --user=admin --password=secret \
         --email=[e-mail] \
         --repos=~/my_dev_repos
 
 
 Start the Development Server
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 When starting the development server, you should start the vcsserver as a
-separate process. To do this use one of the following examples:
+separate process. To do this, use one of the following examples:
 
 1. Set the `start.vcs_server` flag in the ``dev.ini`` file to true. For example:
 
    .. code-block:: python
 
       ### VCS CONFIG ###
       ##################
       vcs.start_server = true
       vcs.server = localhost:9900
       vcs.server.log_level = debug
 
    Then start the server using the following command: ``rcserver dev.ini``
 
 2. Start the development server using the following example::
 
       rcserver --with-vcsserver dev.ini
 
 3. Start the development server in a different terminal using the following
    example::
 
       vcsserver
 
 
 Run the Environment Tests
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Please make sure that the test are passing to verify that your environment is
-set up correctly. More details about the tests are described in:
-:file:`/docs/dev/testing`.
+Please make sure that the tests are passing to verify that your environment is
+set up correctly. RhodeCode uses py.test to run tests.
+Please simply run ``make test`` to run the basic test suite.

docs/contributing/testing/index.rst

@@ -1,28 +1,28 b''
 
 
 ============================
  Testing and Specifications
 ============================
 
 
 .. toctree::
    :maxdepth: 2
 
    unit-and-functional
    spec-by-example
    naming-conventions
 
 
 
 Overview
 ========
 
-We have a quite big test suite inside of :file:`rhodecode/tests` which is a mix
+We have a quite large test suite inside of :file:`rhodecode/tests` which is a mix
 of unit tests and functional or integration tests. More details are in
 :ref:`test-unit-and-functional`.
 
 
-Apart from that we start to apply "Specification by Example" and maintain a
-collection of such specifications together with an implementation so that it can
-be validated in an automatic way. The files can be found in
+Apart from that, we are starting to apply "Specification by Example" and maintain
+a collection of such specifications together with an implementation so that it
+can be validated in an automatic way. The files can be found in
 :file:`acceptance_tests`. More details are in :ref:`test-spec-by-example`.

docs/contributing/testing/spec-by-example.rst

@@ -1,75 +1,75 b''
 
 .. _test-spec-by-example:
 
 ==========================
  Specification by Example
 ==========================
 
 
 .. Avoid duplicating the quickstart instructions by importing the README
    file.
 
 .. include:: ../../../acceptance_tests/README.rst
 
 
 
 Choices of technology and tools
 ===============================
 
 
 `nix` as runtime environment
 ----------------------------
 
 We settled to use the `nix` tools to provide us the needed environment for
 running the tests.
 
 
 
 `Gherkins` as specification language
 ------------------------------------
 
 To specify by example, we settled on Gherkins as the semi-formal specification
 language.
 
 
 `py.test` as a runner
 ---------------------
 
 After experimenting with `behave` and `py.test` our choice was `pytest-bdd`
 because it allows us to use our existing knowledge about `py.test` and avoids
 that we have to learn another tool.
 
 
 
 Concepts
 ========
 
 The logic is structured around the design pattern of "page objects". The
 documentation of `python-selemium` contains a few more details about this
 pattern.
 
 
 
 Page Objects
 ------------
 
 We introduce an abstraction class for every page which we have to interact with
 in order to validate the specifications.
 
 The implementation for the page objects is inside of the module
 :mod:`page_objects`. The class :class:`page_objects.base.BasePage` should be
 used as a base for all page object implementations.
 
 
 
 Locators
 --------
 
 The specific information how to locate an element inside of the DOM tree of a
-page is kept in a separate class. This class serves mainly as a data container,
+page is kept in a separate class. This class serves mainly as a data container;
 it shall not contain any logic.
 
 The reason for keeping the locators separate is that we expect a frequent need
-for change whenever we work on our templates. In such a case it is more
-efficient to have all locators together and update them there instead of having
-to find all locators inside of the logic of a page object.
+for change whenever we work on our templates. In such a case, it is more
+efficient to have all of thelocators together and update them there instead of
+having to find every locator inside of the logic of a page object.

docs/contributing/testing/unit-and-functional.rst

@@ -1,61 +1,61 b''
 
 .. _test-unit-and-functional:
 
 ===========================
  Unit and Functional Tests
 ===========================
 
 
 
 py.test based test suite
 ========================
 
 
 The test suite is in the folder :file:`rhodecode/tests/` and should be run with
 the test runner `py.test` inside of your `nix-shell` environment::
 
    # In case you need the cythonized version
    CYTHONIZE=1 python setup.py develop --prefix=$tmp_path
 
    py.test rhodecode
 
 
 
 py.test integration
 -------------------
 
 The integration with the test runner is based on the following three parts:
 
 - `pytest_pylons` is a py.test plugin which does the integration with the
-  Pylons web framework. It sets up the Pylons environment based on a given ini
+  Pylons web framework. It sets up the Pylons environment based on the given ini
   file.
 
   Tests which depend on the Pylons environment to be set up must request the
   fixture `pylonsapp`.
 
 - :file:`rhodecode/tests/plugin.py` contains the integration of py.test with
   RhodeCode Enterprise itself.
 
 - :file:`conftest.py` plugins are used to provide a special integration for
   certain groups of tests based on the directory location.
 
 
 
 VCS backend selection
 ---------------------
 
 The py.test integration provides a parameter `--backends`. It will skip all
 tests which are marked for other backends.
 
 To run only Subversion tests::
 
    py.test rhodecode --backends=svn
 
 
 
 Frontend / Styling support
 ==========================
 
 All relevant style components have an example inside of the "Style" section
 within the application. Enable the setting `debug_style` to make this section
 visible in your local instance of the application.

docs/release-notes/release-notes.rst

@@ -1,82 +1,83 b''
 .. _rhodecode-release-notes-ref:
 
 Release Notes
 =============
 
 |RCE| 4.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
+   release-notes-4.2.0.rst
    release-notes-4.1.2.rst
    release-notes-4.1.1.rst
    release-notes-4.1.0.rst
    release-notes-4.0.1.rst
    release-notes-4.0.0.rst
 
 |RCE| 3.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-3.8.4.rst
    release-notes-3.8.3.rst
    release-notes-3.8.2.rst
    release-notes-3.8.1.rst
    release-notes-3.8.0.rst
    release-notes-3.7.1.rst
    release-notes-3.7.0.rst
    release-notes-3.6.1.rst
    release-notes-3.6.0.rst
    release-notes-3.5.2.rst
    release-notes-3.5.1.rst
    release-notes-3.5.0.rst
    release-notes-3.4.1.rst
    release-notes-3.4.0.rst
    release-notes-3.3.4.rst
    release-notes-3.3.3.rst
    release-notes-3.3.2.rst
    release-notes-3.3.1.rst
    release-notes-3.3.0.rst
    release-notes-3.2.3.rst
    release-notes-3.2.2.rst
    release-notes-3.2.1.rst
    release-notes-3.2.0.rst
    release-notes-3.1.1.rst
    release-notes-3.1.0.rst
    release-notes-3.0.2.rst
    release-notes-3.0.1.rst
    release-notes-3.0.0.rst
 
 |RCE| 2.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-2.2.8.rst
    release-notes-2.2.7.rst
    release-notes-2.2.6.rst
    release-notes-2.2.5.rst
    release-notes-2.2.4.rst
    release-notes-2.2.3.rst
    release-notes-2.2.2.rst
    release-notes-2.2.1.rst
    release-notes-2.2.0.rst
    release-notes-2.1.0.rst
    release-notes-2.0.2.rst
    release-notes-2.0.1.rst
    release-notes-2.0.0.rst
 
 |RCE| 1.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-1.7.2.rst
    release-notes-1.7.1.rst
    release-notes-1.7.0.rst
    release-notes-1.6.0.rst

license.nix

@@ -1,163 +1,160 b''
 # Utility to generate the license information
 #
 # Usage:
 #
 #   nix-build -I ~/dev license.nix -A result
 #
 # Afterwards ./result will contain the license information as JSON files.
 #
 #
 # Overview
 #
 # Uses two steps to get the relevant license information:
 #
 # 1. Walk down the derivations based on "buildInputs" and
 #    "propagatedBuildInputs". This results in all dependencies based on the nix
 #    declartions.
 #
 # 2. Build Enterprise and query nix-store to get a list of runtime
 #    dependencies. The results from step 1 are then limited to the ones which
 #    are in this list.
 #
 # The result is then available in ./result/license.json.
 #
 
 
 let
 
   nixpkgs = import <nixpkgs> {};
 
   stdenv = nixpkgs.stdenv;
 
   # Enterprise as simple as possible, goal here is just to identify the runtime
   # dependencies. Ideally we could avoid building Enterprise at all and somehow
   # figure it out without calling into nix-store.
   enterprise = import ./default.nix {
     doCheck = false;
-    with_vcsserver = false;
-    with_pyramid = false;
-    cythonize = false;
   };
 
   # For a given derivation, return the list of all dependencies
   drvToDependencies = drv: nixpkgs.lib.flatten [
     drv.nativeBuildInputs or []
     drv.propagatedNativeBuildInputs or []
   ];
 
   # Transform the given derivation into the meta information which we need in
   # the resulting JSON files.
   drvToMeta = drv: {
     name = drv.name or "UNNAMED";
     license = if drv ? meta.license then drv.meta.license else "UNKNOWN";
   };
 
   # Walk the tree of buildInputs and propagatedBuildInputs and return it as a
   # flat list.  Duplicates are avoided.
   listDrvDependencies = drv: let
     addElement = element: seen:
       if (builtins.elem element seen)
       then seen
       else let
         newSeen = seen ++ [ element ];
         newDeps = drvToDependencies element;
       in nixpkgs.lib.fold addElement newSeen newDeps;
     initialElements = drvToDependencies drv;
   in nixpkgs.lib.fold addElement [] initialElements;
 
   # Reads in a file with store paths and returns a list of derivation names.
   #
   # Reads the file, splits the lines, then removes the prefix, so that we
   # end up with a list of derivation names in the end.
   storePathsToDrvNames = srcPath: let
     rawStorePaths = nixpkgs.lib.removeSuffix "\n" (
       builtins.readFile srcPath);
     storePaths = nixpkgs.lib.splitString "\n" rawStorePaths;
     # TODO: johbo: Would be nice to use some sort of utility here to convert
     # the path to a derivation name.
     storePathPrefix = (
       builtins.stringLength "/nix/store/zwy7aavnif9ayw30rya1k6xiacafzzl6-");
     storePathToName = path:
       builtins.substring storePathPrefix (builtins.stringLength path) path;
   in (map storePathToName storePaths);
 
 in rec {
 
   # Build Enterprise and call nix-store to retrieve the runtime
   # dependencies. The result is available in the nix store.
   runtimeDependencies = stdenv.mkDerivation {
     name = "runtime-dependencies";
     buildInputs = [
       # Needed to query the store
       nixpkgs.nix
     ];
     unpackPhase = ''
       echo "Nothing to unpack"
     '';
     buildPhase = ''
       # Get a list of runtime dependencies
       nix-store -q --references ${enterprise} > nix-store-references
     '';
     installPhase = ''
       mkdir -p $out
       cp -v nix-store-references $out/
     '';
   };
 
   # Produce the license overview files.
   result = let
 
     # Dependencies according to the nix-store
     runtimeDependencyNames = (
       storePathsToDrvNames "${runtimeDependencies}/nix-store-references");
 
     # Dependencies based on buildInputs and propagatedBuildInputs
     enterpriseAllDependencies = listDrvDependencies enterprise;
     enterpriseRuntimeDependencies = let
       elemName = element: element.name or "UNNAMED";
       isRuntime = element: builtins.elem (elemName element) runtimeDependencyNames;
     in builtins.filter isRuntime enterpriseAllDependencies;
 
     # Extract relevant meta information
     enterpriseAllLicenses = map drvToMeta enterpriseAllDependencies;
     enterpriseRuntimeLicenses = map drvToMeta enterpriseRuntimeDependencies;
 
   in stdenv.mkDerivation {
 
     name = "licenses";
 
     buildInputs = [];
 
     unpackPhase = ''
       echo "Nothing to unpack"
     '';
 
     buildPhase = ''
       mkdir build
 
       # Copy list of runtime dependencies for the Python processor
       cp "${runtimeDependencies}/nix-store-references" ./build/nix-store-references
 
       # All licenses which we found by walking buildInputs and
       # propagatedBuildInputs
       cat > build/all-licenses.json <<EOF
       ${builtins.toJSON enterpriseAllLicenses}
       EOF
 
       # License information for our runtime dependencies only. Basically all
       # licenses limited to the items which where also reported by nix-store as
       # a dependency.
       cat > build/licenses.json <<EOF
       ${builtins.toJSON enterpriseRuntimeLicenses}
       EOF
     '';
 
     installPhase = ''
       mkdir -p $out
 
       # Store it all, that helps when things go wrong
       cp -rv ./build/* $out
     '';
   };
 
 }

pkgs/python-packages-overrides.nix

@@ -1,165 +1,273 b''
 # Overrides for the generated python-packages.nix
 #
 # This function is intended to be used as an extension to the generated file
 # python-packages.nix. The main objective is to add needed dependencies of C
 # libraries and tweak the build instructions where needed.
 
 { pkgs, basePythonPackages }:
 
 let
   sed = "sed -i";
+  localLicenses = {
+    repoze = {
+      fullName = "Repoze License";
+      url =  http://www.repoze.org/LICENSE.txt;
+    };
+  };
 in
 
 self: super: {
 
+  appenlight-client = super.appenlight-client.override (attrs: {
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
+  });
+
+  future = super.future.override (attrs: {
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  });
+
   gnureadline = super.gnureadline.override (attrs: {
     buildInputs = attrs.buildInputs ++ [
       pkgs.ncurses
     ];
     patchPhase = ''
       substituteInPlace setup.py --replace "/bin/bash" "${pkgs.bash}/bin/bash"
     '';
   });
 
   gunicorn = super.gunicorn.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       # johbo: futures is needed as long as we are on Python 2, otherwise
       # gunicorn explodes if used with multiple threads per worker.
       self.futures
     ];
   });
 
   ipython = super.ipython.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       self.gnureadline
     ];
   });
 
   kombu = super.kombu.override (attrs: {
     # The current version of kombu needs some patching to work with the
     # other libs. Should be removed once we update celery and kombu.
     patches = [
       ./patch-kombu-py-2-7-11.diff
       ./patch-kombu-msgpack.diff
     ];
   });
 
   lxml = super.lxml.override (attrs: {
     buildInputs = with self; [
       pkgs.libxml2
       pkgs.libxslt
     ];
   });
 
   MySQL-python = super.MySQL-python.override (attrs: {
     buildInputs = attrs.buildInputs ++ [
       pkgs.openssl
     ];
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.mysql.lib
       pkgs.zlib
     ];
   });
 
   psutil = super.psutil.override (attrs: {
     buildInputs = attrs.buildInputs ++
       pkgs.lib.optional pkgs.stdenv.isDarwin pkgs.darwin.IOKit;
   });
 
   psycopg2 = super.psycopg2.override (attrs: {
     buildInputs = attrs.buildInputs ++
       pkgs.lib.optional pkgs.stdenv.isDarwin pkgs.openssl;
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.postgresql
     ];
+    meta = {
+      license = pkgs.lib.licenses.lgpl3Plus;
+    };
   });
 
   pycurl = super.pycurl.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.curl
       pkgs.openssl
     ];
     preConfigure = ''
       substituteInPlace setup.py --replace '--static-libs' '--libs'
       export PYCURL_SSL_LIBRARY=openssl
     '';
+    meta = {
+      # TODO: It is LGPL and MIT
+      license = pkgs.lib.licenses.mit;
+    };
   });
 
   Pylons = super.Pylons.override (attrs: {
     name = "Pylons-1.0.1-patch1";
     src = pkgs.fetchgit {
       url = "https://code.rhodecode.com/upstream/pylons";
       rev = "707354ee4261b9c10450404fc9852ccea4fd667d";
       sha256 = "b2763274c2780523a335f83a1df65be22ebe4ff413a7bc9e9288d23c1f62032e";
     };
   });
 
   pyramid = super.pyramid.override (attrs: {
     postFixup = ''
       wrapPythonPrograms
       # TODO: johbo: "wrapPython" adds this magic line which
       # confuses pserve.
       ${sed} '/import sys; sys.argv/d' $out/bin/.pserve-wrapped
     '';
+    meta = {
+      license = localLicenses.repoze;
+    };
+  });
+
+  pyramid-debugtoolbar = super.pyramid-debugtoolbar.override (attrs: {
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal localLicenses.repoze ];
+    };
   });
 
   Pyro4 = super.Pyro4.override (attrs: {
     # TODO: Was not able to generate this version, needs further
     # investigation.
     name = "Pyro4-4.35";
     src = pkgs.fetchurl {
       url = "https://pypi.python.org/packages/source/P/Pyro4/Pyro4-4.35.src.tar.gz";
       md5 = "cbe6cb855f086a0f092ca075005855f3";
     };
   });
 
   pysqlite = super.pysqlite.override (attrs: {
     propagatedBuildInputs = [
       pkgs.sqlite
     ];
+    meta = {
+      license = [ pkgs.lib.licenses.zlib pkgs.lib.licenses.libpng ];
+    };
   });
 
   pytest-runner = super.pytest-runner.override (attrs: {
     propagatedBuildInputs = [
       self.setuptools-scm
     ];
   });
 
   python-ldap = super.python-ldap.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.cyrus_sasl
       pkgs.openldap
       pkgs.openssl
     ];
     NIX_CFLAGS_COMPILE = "-I${pkgs.cyrus_sasl}/include/sasl";
   });
 
   python-pam = super.python-pam.override (attrs:
     let
       includeLibPam = pkgs.stdenv.isLinux;
     in {
       # TODO: johbo: Move the option up into the default.nix, we should
       # include python-pam only on supported platforms.
       propagatedBuildInputs = attrs.propagatedBuildInputs ++
         pkgs.lib.optional includeLibPam [
           pkgs.pam
         ];
       # TODO: johbo: Check if this can be avoided, or transform into
       # a real patch
       patchPhase = pkgs.lib.optionals includeLibPam ''
         substituteInPlace pam.py \
           --replace 'find_library("pam")' '"${pkgs.pam}/lib/libpam.so.0"'
       '';
     });
 
   rhodecode-tools = super.rhodecode-tools.override (attrs: {
     patches = [
       ./patch-rhodecode-tools-setup.diff
     ];
   });
 
+  URLObject = super.URLObject.override (attrs: {
+    meta = {
+      license = {
+        spdxId = "Unlicense";
+        fullName = "The Unlicense";
+        url = http://unlicense.org/;
+      };
+    };
+  });
+
+  amqplib = super.amqplib.override (attrs: {
+    meta = {
+      license = pkgs.lib.licenses.lgpl3;
+    };
+  });
+
+  docutils = super.docutils.override (attrs: {
+    meta = {
+      license = pkgs.lib.licenses.bsd2;
+    };
+  });
+
+  colander = super.colander.override (attrs: {
+    meta = {
+      license = localLicenses.repoze;
+    };
+  });
+
+  pyramid-beaker  = super.pyramid-beaker.override (attrs: {
+    meta = {
+      license = localLicenses.repoze;
+    };
+  });
+
+  pyramid-mako = super.pyramid-mako.override (attrs: {
+    meta = {
+      license = localLicenses.repoze;
+    };
+  });
+
+  repoze.lru = super.repoze.lru.override (attrs: {
+    meta = {
+      license = localLicenses.repoze;
+    };
+  });
+
+  recaptcha-client = super.recaptcha-client.override (attrs: {
+    meta = {
+      # TODO: It is MIT/X11
+      license = pkgs.lib.licenses.mit;
+    };
+  });
+
+  python-editor = super.python-editor.override (attrs: {
+    meta = {
+      license = pkgs.lib.licenses.asl20;
+    };
+  });
+
+  translationstring = super.translationstring.override (attrs: {
+    meta = {
+      license = localLicenses.repoze;
+    };
+  });
+
+  venusian = super.venusian.override (attrs: {
+    meta = {
+      license = localLicenses.repoze;
+    };
+  });
+
   # Avoid that setuptools is replaced, this leads to trouble
   # with buildPythonPackage.
   setuptools = basePythonPackages.setuptools;
 
 }

pkgs/python-packages.nix

@@ -1,1263 +1,1641 b''
 {
   Babel = super.buildPythonPackage {
     name = "Babel-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytz];
     src = fetchurl {
       url = "https://pypi.python.org/packages/33/27/e3978243a03a76398c384c83f7ca879bc6e8f1511233a621fcada135606e/Babel-1.3.tar.gz";
       md5 = "5264ceb02717843cbc9ffce8e6e06bdb";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   Beaker = super.buildPythonPackage {
     name = "Beaker-1.7.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/97/8e/409d2e7c009b8aa803dc9e6f239f1db7c3cdf578249087a404e7c27a505d/Beaker-1.7.0.tar.gz";
       md5 = "386be3f7fe427358881eee4622b428b3";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   CProfileV = super.buildPythonPackage {
     name = "CProfileV-1.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [bottle];
     src = fetchurl {
       url = "https://pypi.python.org/packages/eb/df/983a0b6cfd3ac94abf023f5011cb04f33613ace196e33f53c86cf91850d5/CProfileV-1.0.6.tar.gz";
       md5 = "08c7c242b6e64237bc53c5d13537e03d";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   Fabric = super.buildPythonPackage {
     name = "Fabric-1.10.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [paramiko];
     src = fetchurl {
       url = "https://pypi.python.org/packages/e3/5f/b6ebdb5241d5ec9eab582a5c8a01255c1107da396f849e538801d2fe64a5/Fabric-1.10.0.tar.gz";
       md5 = "2cb96473387f0e7aa035210892352f4a";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   FormEncode = super.buildPythonPackage {
     name = "FormEncode-1.2.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8e/59/0174271a6f004512e0201188593e6d319db139d14cb7490e488bbb078015/FormEncode-1.2.4.tar.gz";
       md5 = "6bc17fb9aed8aea198975e888e2077f4";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.psfl ];
+    };
   };
   Jinja2 = super.buildPythonPackage {
     name = "Jinja2-2.7.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b0/73/eab0bca302d6d6a0b5c402f47ad1760dc9cb2dd14bbc1873ad48db258e4d/Jinja2-2.7.3.tar.gz";
       md5 = "b9dffd2f3b43d673802fe857c8445b1a";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   Mako = super.buildPythonPackage {
     name = "Mako-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8e/a4/aa56533ecaa5f22ca92428f74e074d0c9337282933c722391902c8f9e0f8/Mako-1.0.1.tar.gz";
       md5 = "9f0aafd177b039ef67b90ea350497a54";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   Markdown = super.buildPythonPackage {
     name = "Markdown-2.6.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/62/8b/83658b5f6c220d5fcde9f9852d46ea54765d734cfbc5a9f4c05bfc36db4d/Markdown-2.6.2.tar.gz";
       md5 = "256d19afcc564dc4ce4c229bb762f7ae";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   MarkupSafe = super.buildPythonPackage {
     name = "MarkupSafe-0.23";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c0/41/bae1254e0396c0cc8cf1751cb7d9afc90a602353695af5952530482c963f/MarkupSafe-0.23.tar.gz";
       md5 = "f5ab3deee4c37cd6a922fb81e730da6e";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   MySQL-python = super.buildPythonPackage {
     name = "MySQL-python-1.2.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a5/e9/51b544da85a36a68debe7a7091f068d802fc515a3a202652828c73453cad/MySQL-python-1.2.5.zip";
       md5 = "654f75b302db6ed8dc5a898c625e030c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.gpl1 ];
+    };
   };
   Paste = super.buildPythonPackage {
     name = "Paste-2.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d5/8d/0f8ac40687b97ff3e07ebd1369be20bdb3f93864d2dc3c2ff542edb4ce50/Paste-2.0.2.tar.gz";
       md5 = "4bfc8a7eaf858f6309d2ac0f40fc951c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   PasteDeploy = super.buildPythonPackage {
     name = "PasteDeploy-1.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0f/90/8e20cdae206c543ea10793cbf4136eb9a8b3f417e04e40a29d72d9922cbd/PasteDeploy-1.5.2.tar.gz";
       md5 = "352b7205c78c8de4987578d19431af3b";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   PasteScript = super.buildPythonPackage {
     name = "PasteScript-1.7.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Paste PasteDeploy];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a5/05/fc60efa7c2f17a1dbaeccb2a903a1e90902d92b9d00eebabe3095829d806/PasteScript-1.7.5.tar.gz";
       md5 = "4c72d78dcb6bb993f30536842c16af4d";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   Pygments = super.buildPythonPackage {
     name = "Pygments-2.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f4/c6/bdbc5a8a112256b2b6136af304dbae93d8b1ef8738ff2d12a51018800e46/Pygments-2.0.2.tar.gz";
       md5 = "238587a1370d62405edabd0794b3ec4a";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   Pylons = super.buildPythonPackage {
     name = "Pylons-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Routes WebHelpers Beaker Paste PasteDeploy PasteScript FormEncode simplejson decorator nose Mako WebError WebTest Tempita MarkupSafe WebOb];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a2/69/b835a6bad00acbfeed3f33c6e44fa3f936efc998c795bfb15c61a79ecf62/Pylons-1.0.1.tar.gz";
       md5 = "6cb880d75fa81213192142b07a6e4915";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   Pyro4 = super.buildPythonPackage {
     name = "Pyro4-4.41";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [serpent];
     src = fetchurl {
       url = "https://pypi.python.org/packages/56/2b/89b566b4bf3e7f8ba790db2d1223852f8cb454c52cab7693dd41f608ca2a/Pyro4-4.41.tar.gz";
       md5 = "ed69e9bfafa9c06c049a87cb0c4c2b6c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   Routes = super.buildPythonPackage {
     name = "Routes-1.13";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [repoze.lru];
     src = fetchurl {
       url = "https://pypi.python.org/packages/88/d3/259c3b3cde8837eb9441ab5f574a660e8a4acea8f54a078441d4d2acac1c/Routes-1.13.tar.gz";
       md5 = "d527b0ab7dd9172b1275a41f97448783";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   SQLAlchemy = super.buildPythonPackage {
     name = "SQLAlchemy-0.9.9";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/28/f7/1bbfd0d8597e8c358d5e15a166a486ad82fc5579b4e67b6ef7c05b1d182b/SQLAlchemy-0.9.9.tar.gz";
       md5 = "8a10a9bd13ed3336ef7333ac2cc679ff";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   Sphinx = super.buildPythonPackage {
     name = "Sphinx-1.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Pygments docutils Jinja2];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0a/50/34017e6efcd372893a416aba14b84a1a149fc7074537b0e9cb6ca7b7abe9/Sphinx-1.2.2.tar.gz";
       md5 = "3dc73ccaa8d0bfb2d62fb671b1f7e8a4";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   Tempita = super.buildPythonPackage {
     name = "Tempita-0.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/56/c8/8ed6eee83dbddf7b0fc64dd5d4454bc05e6ccaafff47991f73f2894d9ff4/Tempita-0.5.2.tar.gz";
       md5 = "4c2f17bb9d481821c41b6fbee904cea1";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   URLObject = super.buildPythonPackage {
     name = "URLObject-2.4.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/cb/b6/e25e58500f9caef85d664bec71ec67c116897bfebf8622c32cb75d1ca199/URLObject-2.4.0.tar.gz";
       md5 = "2ed819738a9f0a3051f31dc9924e3065";
     };
+    meta = {
+      license = [  ];
+    };
   };
   WebError = super.buildPythonPackage {
     name = "WebError-0.10.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb Tempita Pygments Paste];
     src = fetchurl {
       url = "https://pypi.python.org/packages/35/76/e7e5c2ce7e9c7f31b54c1ff295a495886d1279a002557d74dd8957346a79/WebError-0.10.3.tar.gz";
       md5 = "84b9990b0baae6fd440b1e60cdd06f9a";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   WebHelpers = super.buildPythonPackage {
     name = "WebHelpers-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ee/68/4d07672821d514184357f1552f2dad923324f597e722de3b016ca4f7844f/WebHelpers-1.3.tar.gz";
       md5 = "32749ffadfc40fea51075a7def32588b";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   WebHelpers2 = super.buildPythonPackage {
     name = "WebHelpers2-2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ff/30/56342c6ea522439e3662427c8d7b5e5b390dff4ff2dc92d8afcb8ab68b75/WebHelpers2-2.0.tar.gz";
       md5 = "0f6b68d70c12ee0aed48c00b24da13d3";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   WebOb = super.buildPythonPackage {
     name = "WebOb-1.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/16/78/adfc0380b8a0d75b2d543fa7085ba98a573b1ae486d9def88d172b81b9fa/WebOb-1.3.1.tar.gz";
       md5 = "20918251c5726956ba8fef22d1556177";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   WebTest = super.buildPythonPackage {
     name = "WebTest-1.4.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb];
     src = fetchurl {
       url = "https://pypi.python.org/packages/51/3d/84fd0f628df10b30c7db87895f56d0158e5411206b721ca903cb51bfd948/WebTest-1.4.3.zip";
       md5 = "631ce728bed92c681a4020a36adbc353";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   Whoosh = super.buildPythonPackage {
     name = "Whoosh-2.7.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/1c/dc/2f0231ff3875ded36df8c1ab851451e51a237dc0e5a86d3d96036158da94/Whoosh-2.7.0.zip";
       md5 = "7abfd970f16fadc7311960f3fa0bc7a9";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.bsd2 ];
+    };
   };
   alembic = super.buildPythonPackage {
     name = "alembic-0.8.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [SQLAlchemy Mako python-editor];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ca/7e/299b4499b5c75e5a38c5845145ad24755bebfb8eec07a2e1c366b7181eeb/alembic-0.8.4.tar.gz";
       md5 = "5f95d8ee62b443f9b37eb5bee76c582d";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   amqplib = super.buildPythonPackage {
     name = "amqplib-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/75/b7/8c2429bf8d92354a0118614f9a4d15e53bc69ebedce534284111de5a0102/amqplib-1.0.2.tgz";
       md5 = "5c92f17fbedd99b2b4a836d4352d1e2f";
     };
+    meta = {
+      license = [ { fullName = "LGPL"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
+    };
   };
   anyjson = super.buildPythonPackage {
     name = "anyjson-0.3.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz";
       md5 = "2ea28d6ec311aeeebaf993cb3008b27c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   appenlight-client = super.buildPythonPackage {
     name = "appenlight-client-0.6.14";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb requests];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4d/e0/23fee3ebada8143f707e65c06bcb82992040ee64ea8355e044ed55ebf0c1/appenlight_client-0.6.14.tar.gz";
       md5 = "578c69b09f4356d898fff1199b98a95c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal { fullName = "DFSG approved"; } ];
+    };
   };
   authomatic = super.buildPythonPackage {
     name = "authomatic-0.1.0.post1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/08/1a/8a930461e604c2d5a7a871e1ac59fa82ccf994c32e807230c8d2fb07815a/Authomatic-0.1.0.post1.tar.gz";
       md5 = "be3f3ce08747d776aae6d6cc8dcb49a9";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   backport-ipaddress = super.buildPythonPackage {
     name = "backport-ipaddress-0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d3/30/54c6dab05a4dec44db25ff309f1fbb6b7a8bde3f2bade38bb9da67bbab8f/backport_ipaddress-0.1.tar.gz";
       md5 = "9c1f45f4361f71b124d7293a60006c05";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.psfl ];
+    };
   };
   bottle = super.buildPythonPackage {
     name = "bottle-0.12.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/52/df/e4a408f3a7af396d186d4ecd3b389dd764f0f943b4fa8d257bfe7b49d343/bottle-0.12.8.tar.gz";
       md5 = "13132c0a8f607bf860810a6ee9064c5b";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   bumpversion = super.buildPythonPackage {
     name = "bumpversion-0.5.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/14/41/8c9da3549f8e00c84f0432c3a8cf8ed6898374714676aab91501d48760db/bumpversion-0.5.3.tar.gz";
       md5 = "c66a3492eafcf5ad4b024be9fca29820";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   celery = super.buildPythonPackage {
     name = "celery-2.2.10";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [python-dateutil anyjson kombu pyparsing];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b1/64/860fd50e45844c83442e7953effcddeff66b2851d90b2d784f7201c111b8/celery-2.2.10.tar.gz";
       md5 = "898bc87e54f278055b561316ba73e222";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   click = super.buildPythonPackage {
     name = "click-5.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b7/34/a496632c4fb6c1ee76efedf77bb8d28b29363d839953d95095b12defe791/click-5.1.tar.gz";
       md5 = "9c5323008cccfe232a8b161fc8196d41";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   colander = super.buildPythonPackage {
     name = "colander-1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [translationstring iso8601];
     src = fetchurl {
       url = "https://pypi.python.org/packages/14/23/c9ceba07a6a1dc0eefbb215fc0dc64aabc2b22ee756bc0f0c13278fa0887/colander-1.2.tar.gz";
       md5 = "83db21b07936a0726e588dae1914b9ed";
     };
+    meta = {
+      license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   configobj = super.buildPythonPackage {
     name = "configobj-5.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz";
       md5 = "e472a3a1c2a67bb0ec9b5d54c13a47d6";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   cov-core = super.buildPythonPackage {
     name = "cov-core-1.15.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [coverage];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4b/87/13e75a47b4ba1be06f29f6d807ca99638bedc6b57fa491cd3de891ca2923/cov-core-1.15.0.tar.gz";
       md5 = "f519d4cb4c4e52856afb14af52919fe6";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   coverage = super.buildPythonPackage {
     name = "coverage-3.7.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/09/4f/89b06c7fdc09687bca507dc411c342556ef9c5a3b26756137a4878ff19bf/coverage-3.7.1.tar.gz";
       md5 = "c47b36ceb17eaff3ecfab3bcd347d0df";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   cssselect = super.buildPythonPackage {
     name = "cssselect-0.9.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/aa/e5/9ee1460d485b94a6d55732eb7ad5b6c084caf73dd6f9cb0bb7d2a78fafe8/cssselect-0.9.1.tar.gz";
       md5 = "c74f45966277dc7a0f768b9b0f3522ac";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   decorator = super.buildPythonPackage {
     name = "decorator-3.4.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/35/3a/42566eb7a2cbac774399871af04e11d7ae3fc2579e7dae85213b8d1d1c57/decorator-3.4.2.tar.gz";
       md5 = "9e0536870d2b83ae27d58dbf22582f4d";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   docutils = super.buildPythonPackage {
     name = "docutils-0.12";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/37/38/ceda70135b9144d84884ae2fc5886c6baac4edea39550f28bcd144c1234d/docutils-0.12.tar.gz";
       md5 = "4622263b62c5c771c03502afa3157768";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.publicDomain pkgs.lib.licenses.gpl1 { fullName = "public domain, Python, 2-Clause BSD, GPL 3 (see COPYING.txt)"; } pkgs.lib.licenses.psfl ];
+    };
   };
   dogpile.cache = super.buildPythonPackage {
     name = "dogpile.cache-0.5.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [dogpile.core];
     src = fetchurl {
       url = "https://pypi.python.org/packages/07/74/2a83bedf758156d9c95d112691bbad870d3b77ccbcfb781b4ef836ea7d96/dogpile.cache-0.5.7.tar.gz";
       md5 = "3e58ce41af574aab41d78e9c4190f194";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   dogpile.core = super.buildPythonPackage {
     name = "dogpile.core-0.4.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0e/77/e72abc04c22aedf874301861e5c1e761231c288b5de369c18be8f4b5c9bb/dogpile.core-0.4.1.tar.gz";
       md5 = "01cb19f52bba3e95c9b560f39341f045";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   dulwich = super.buildPythonPackage {
     name = "dulwich-0.12.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6f/04/fbe561b6d45c0ec758330d5b7f5ba4b6cb4f1ca1ab49859d2fc16320da75/dulwich-0.12.0.tar.gz";
       md5 = "f3a8a12bd9f9dd8c233e18f3d49436fa";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.gpl2Plus ];
+    };
   };
   ecdsa = super.buildPythonPackage {
     name = "ecdsa-0.11";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6c/3f/92fe5dcdcaa7bd117be21e5520c9a54375112b66ec000d209e9e9519fad1/ecdsa-0.11.tar.gz";
       md5 = "8ef586fe4dbb156697d756900cb41d7c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   elasticsearch = super.buildPythonPackage {
     name = "elasticsearch-2.3.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [urllib3];
     src = fetchurl {
       url = "https://pypi.python.org/packages/10/35/5fd52c5f0b0ee405ed4b5195e8bce44c5e041787680dc7b94b8071cac600/elasticsearch-2.3.0.tar.gz";
       md5 = "2550f3b51629cf1ef9636608af92c340";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
   };
   elasticsearch-dsl = super.buildPythonPackage {
     name = "elasticsearch-dsl-2.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six python-dateutil elasticsearch];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4e/5d/e788ae8dbe2ff4d13426db0a027533386a5c276c77a2654dc0e2007ce04a/elasticsearch-dsl-2.0.0.tar.gz";
       md5 = "4cdfec81bb35383dd3b7d02d7dc5ee68";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
   };
   flake8 = super.buildPythonPackage {
     name = "flake8-2.4.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyflakes pep8 mccabe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8f/b5/9a73c66c7dba273bac8758398f060c008a25f3e84531063b42503b5d0a95/flake8-2.4.1.tar.gz";
       md5 = "ed45d3db81a3b7c88bd63c6e37ca1d65";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   future = super.buildPythonPackage {
     name = "future-0.14.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/83/80/8ef3a11a15f8eaafafa0937b20c1b3f73527e69ab6b3fa1cf94a5a96aabb/future-0.14.3.tar.gz";
       md5 = "e94079b0bd1fc054929e8769fc0f6083";
     };
+    meta = {
+      license = [ { fullName = "OSI Approved"; } pkgs.lib.licenses.mit ];
+    };
   };
   futures = super.buildPythonPackage {
     name = "futures-3.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f8/e7/fc0fcbeb9193ba2d4de00b065e7fd5aecd0679e93ce95a07322b2b1434f4/futures-3.0.2.tar.gz";
       md5 = "42aaf1e4de48d6e871d77dc1f9d96d5a";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   gnureadline = super.buildPythonPackage {
     name = "gnureadline-6.3.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3a/ee/2c3f568b0a74974791ac590ec742ef6133e2fbd287a074ba72a53fa5e97c/gnureadline-6.3.3.tar.gz";
       md5 = "c4af83c9a3fbeac8f2da9b5a7c60e51c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.gpl1 ];
+    };
   };
   gprof2dot = super.buildPythonPackage {
-    name = "gprof2dot-2015.12.01";
+    name = "gprof2dot-2015.12.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b9/34/7bf93c1952d40fa5c95ad963f4d8344b61ef58558632402eca18e6c14127/gprof2dot-2015.12.1.tar.gz";
       md5 = "e23bf4e2f94db032750c193384b4165b";
     };
+    meta = {
+      license = [ { fullName = "LGPL"; } ];
+    };
   };
   greenlet = super.buildPythonPackage {
     name = "greenlet-0.4.9";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4e/3d/9d421539b74e33608b245092870156b2e171fb49f2b51390aa4641eecb4a/greenlet-0.4.9.zip";
       md5 = "c6659cdb2a5e591723e629d2eef22e82";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   gunicorn = super.buildPythonPackage {
     name = "gunicorn-19.6.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/84/ce/7ea5396efad1cef682bbc4068e72a0276341d9d9d0f501da609fab9fcb80/gunicorn-19.6.0.tar.gz";
       md5 = "338e5e8a83ea0f0625f768dba4597530";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   infrae.cache = super.buildPythonPackage {
     name = "infrae.cache-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Beaker repoze.lru];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bb/f0/e7d5e984cf6592fd2807dc7bc44a93f9d18e04e6a61f87fdfb2622422d74/infrae.cache-1.0.1.tar.gz";
       md5 = "b09076a766747e6ed2a755cc62088e32";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 ];
+    };
   };
   invoke = super.buildPythonPackage {
-    name = "invoke-0.11.1";
+    name = "invoke-0.13.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/d3/bb/36a5558ea19882073def7b0edeef4a0e6282056fed96506dd10b1d532bd4/invoke-0.11.1.tar.gz";
-      md5 = "3d4ecbe26779ceef1046ecf702c9c4a8";
+      url = "https://pypi.python.org/packages/47/bf/d07ef52fa1ac645468858bbac7cb95b246a972a045e821493d17d89c81be/invoke-0.13.0.tar.gz";
+      md5 = "c0d1ed4bfb34eaab551662d8cfee6540";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipdb = super.buildPythonPackage {
     name = "ipdb-0.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [ipython];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f0/25/d7dd430ced6cd8dc242a933c8682b5dbf32eb4011d82f87e34209e5ec845/ipdb-0.8.zip";
       md5 = "96dca0712efa01aa5eaf6b22071dd3ed";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.gpl1 ];
+    };
   };
   ipython = super.buildPythonPackage {
     name = "ipython-3.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/06/91/120c0835254c120af89f066afaabf81289bc2726c1fc3ca0555df6882f58/ipython-3.1.0.tar.gz";
       md5 = "a749d90c16068687b0ec45a27e72ef8f";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   iso8601 = super.buildPythonPackage {
     name = "iso8601-0.1.11";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c0/75/c9209ee4d1b5975eb8c2cba4428bde6b61bd55664a98290dd015cdb18e98/iso8601-0.1.11.tar.gz";
       md5 = "b06d11cd14a64096f907086044f0fe38";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   itsdangerous = super.buildPythonPackage {
     name = "itsdangerous-0.24";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz";
       md5 = "a3d55aa79369aef5345c036a8a26307f";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   kombu = super.buildPythonPackage {
     name = "kombu-1.5.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [anyjson amqplib];
     src = fetchurl {
       url = "https://pypi.python.org/packages/19/53/74bf2a624644b45f0850a638752514fc10a8e1cbd738f10804951a6df3f5/kombu-1.5.1.tar.gz";
       md5 = "50662f3c7e9395b3d0721fb75d100b63";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   lxml = super.buildPythonPackage {
     name = "lxml-3.4.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz";
       md5 = "a9a65972afc173ec7a39c585f4eea69c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   mccabe = super.buildPythonPackage {
     name = "mccabe-0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c9/2e/75231479e11a906b64ac43bad9d0bb534d00080b18bdca8db9da46e1faf7/mccabe-0.3.tar.gz";
       md5 = "81640948ff226f8c12b3277059489157";
     };
+    meta = {
+      license = [ { fullName = "Expat license"; } pkgs.lib.licenses.mit ];
+    };
   };
   meld3 = super.buildPythonPackage {
     name = "meld3-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/45/a0/317c6422b26c12fe0161e936fc35f36552069ba8e6f7ecbd99bbffe32a5f/meld3-1.0.2.tar.gz";
       md5 = "3ccc78cd79cffd63a751ad7684c02c91";
     };
+    meta = {
+      license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   mock = super.buildPythonPackage {
     name = "mock-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/15/45/30273ee91feb60dabb8fbb2da7868520525f02cf910279b3047182feed80/mock-1.0.1.zip";
       md5 = "869f08d003c289a97c1a6610faf5e913";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   msgpack-python = super.buildPythonPackage {
     name = "msgpack-python-0.4.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/15/ce/ff2840885789ef8035f66cd506ea05bdb228340307d5e71a7b1e3f82224c/msgpack-python-0.4.6.tar.gz";
       md5 = "8b317669314cf1bc881716cccdaccb30";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
   };
   nose = super.buildPythonPackage {
     name = "nose-1.3.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/70/c7/469e68148d17a0d3db5ed49150242fd70a74a8147b8f3f8b87776e028d99/nose-1.3.6.tar.gz";
       md5 = "0ca546d81ca8309080fc80cb389e7a16";
     };
+    meta = {
+      license = [ { fullName = "GNU Library or Lesser General Public License (LGPL)"; } { fullName = "GNU LGPL"; } ];
+    };
   };
   objgraph = super.buildPythonPackage {
     name = "objgraph-2.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d7/33/ace750b59247496ed769b170586c5def7202683f3d98e737b75b767ff29e/objgraph-2.0.0.tar.gz";
       md5 = "25b0d5e5adc74aa63ead15699614159c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   packaging = super.buildPythonPackage {
     name = "packaging-15.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/24/c4/185da1304f07047dc9e0c46c31db75c0351bd73458ac3efad7da3dbcfbe1/packaging-15.2.tar.gz";
       md5 = "c16093476f6ced42128bf610e5db3784";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
   };
   paramiko = super.buildPythonPackage {
     name = "paramiko-1.15.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pycrypto ecdsa];
     src = fetchurl {
       url = "https://pypi.python.org/packages/04/2b/a22d2a560c1951abbbf95a0628e245945565f70dc082d9e784666887222c/paramiko-1.15.1.tar.gz";
       md5 = "48c274c3f9b1282932567b21f6acf3b5";
     };
+    meta = {
+      license = [ { fullName = "LGPL"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
+    };
   };
   pep8 = super.buildPythonPackage {
     name = "pep8-1.5.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8b/de/259f5e735897ada1683489dd514b2a1c91aaa74e5e6b68f80acf128a6368/pep8-1.5.7.tar.gz";
       md5 = "f6adbdd69365ecca20513c709f9b7c93";
     };
+    meta = {
+      license = [ { fullName = "Expat license"; } pkgs.lib.licenses.mit ];
+    };
   };
   psutil = super.buildPythonPackage {
     name = "psutil-2.2.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/df/47/ee54ef14dd40f8ce831a7581001a5096494dc99fe71586260ca6b531fe86/psutil-2.2.1.tar.gz";
       md5 = "1a2b58cd9e3a53528bb6148f0c4d5244";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   psycopg2 = super.buildPythonPackage {
     name = "psycopg2-2.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/dd/c7/9016ff8ff69da269b1848276eebfb264af5badf6b38caad805426771f04d/psycopg2-2.6.tar.gz";
       md5 = "fbbb039a8765d561a1c04969bbae7c74";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 { fullName = "GNU Library or Lesser General Public License (LGPL)"; } { fullName = "LGPL with exceptions or ZPL"; } ];
+    };
   };
   py = super.buildPythonPackage {
     name = "py-1.4.29";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2a/bc/a1a4a332ac10069b8e5e25136a35e08a03f01fd6ab03d819889d79a1fd65/py-1.4.29.tar.gz";
       md5 = "c28e0accba523a29b35a48bb703fb96c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   py-bcrypt = super.buildPythonPackage {
     name = "py-bcrypt-0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/68/b1/1c3068c5c4d2e35c48b38dcc865301ebfdf45f54507086ac65ced1fd3b3d/py-bcrypt-0.4.tar.gz";
       md5 = "dd8b367d6b716a2ea2e72392525f4e36";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   pycrypto = super.buildPythonPackage {
     name = "pycrypto-2.6.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/60/db/645aa9af249f059cc3a368b118de33889219e0362141e75d4eaf6f80f163/pycrypto-2.6.1.tar.gz";
       md5 = "55a61a054aa66812daf5161a0d5d7eda";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.publicDomain ];
+    };
   };
   pycurl = super.buildPythonPackage {
     name = "pycurl-7.19.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6c/48/13bad289ef6f4869b1d8fc11ae54de8cfb3cc4a2eb9f7419c506f763be46/pycurl-7.19.5.tar.gz";
       md5 = "47b4eac84118e2606658122104e62072";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit { fullName = "LGPL/MIT"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
+    };
   };
   pyflakes = super.buildPythonPackage {
     name = "pyflakes-0.8.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/75/22/a90ec0252f4f87f3ffb6336504de71fe16a49d69c4538dae2f12b9360a38/pyflakes-0.8.1.tar.gz";
       md5 = "905fe91ad14b912807e8fdc2ac2e2c23";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pyparsing = super.buildPythonPackage {
     name = "pyparsing-1.5.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2e/26/e8fb5b4256a5f5036be7ce115ef8db8d06bc537becfbdc46c6af008314ee/pyparsing-1.5.7.zip";
       md5 = "b86854857a368d6ccb4d5b6e76d0637f";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pyramid = super.buildPythonPackage {
     name = "pyramid-1.6.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools WebOb repoze.lru zope.interface zope.deprecation venusian translationstring PasteDeploy];
     src = fetchurl {
       url = "https://pypi.python.org/packages/30/b3/fcc4a2a4800cbf21989e00454b5828cf1f7fe35c63e0810b350e56d4c475/pyramid-1.6.1.tar.gz";
       md5 = "b18688ff3cc33efdbb098a35b45dd122";
     };
+    meta = {
+      license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   pyramid-beaker = super.buildPythonPackage {
     name = "pyramid-beaker-0.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid Beaker];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d9/6e/b85426e00fd3d57f4545f74e1c3828552d8700f13ededeef9233f7bca8be/pyramid_beaker-0.8.tar.gz";
       md5 = "22f14be31b06549f80890e2c63a93834";
     };
+    meta = {
+      license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   pyramid-debugtoolbar = super.buildPythonPackage {
     name = "pyramid-debugtoolbar-2.4.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid pyramid-mako repoze.lru Pygments];
     src = fetchurl {
       url = "https://pypi.python.org/packages/89/00/ed5426ee41ed747ba3ffd30e8230841a6878286ea67d480b1444d24f06a2/pyramid_debugtoolbar-2.4.2.tar.gz";
       md5 = "073ea67086cc4bd5decc3a000853642d";
     };
+    meta = {
+      license = [ { fullName = "Repoze Public License"; } pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   pyramid-jinja2 = super.buildPythonPackage {
     name = "pyramid-jinja2-2.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid zope.deprecation Jinja2 MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a1/80/595e26ffab7deba7208676b6936b7e5a721875710f982e59899013cae1ed/pyramid_jinja2-2.5.tar.gz";
       md5 = "07cb6547204ac5e6f0b22a954ccee928";
     };
+    meta = {
+      license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   pyramid-mako = super.buildPythonPackage {
     name = "pyramid-mako-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid Mako];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f1/92/7e69bcf09676d286a71cb3bbb887b16595b96f9ba7adbdc239ffdd4b1eb9/pyramid_mako-1.0.2.tar.gz";
       md5 = "ee25343a97eb76bd90abdc2a774eb48a";
     };
+    meta = {
+      license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   pysqlite = super.buildPythonPackage {
     name = "pysqlite-2.6.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5c/a6/1c429cd4c8069cf4bfbd0eb4d592b3f4042155a8202df83d7e9b93aa3dc2/pysqlite-2.6.3.tar.gz";
       md5 = "7ff1cedee74646b50117acff87aa1cfa";
     };
+    meta = {
+      license = [ { fullName = "zlib/libpng License"; } { fullName = "zlib/libpng license"; } ];
+    };
   };
   pytest = super.buildPythonPackage {
     name = "pytest-2.8.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [py];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b1/3d/d7ea9b0c51e0cacded856e49859f0a13452747491e842c236bbab3714afe/pytest-2.8.5.zip";
       md5 = "8493b06f700862f1294298d6c1b715a9";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pytest-catchlog = super.buildPythonPackage {
     name = "pytest-catchlog-1.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [py pytest];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f2/2b/2faccdb1a978fab9dd0bf31cca9f6847fbe9184a0bdcc3011ac41dd44191/pytest-catchlog-1.2.2.zip";
       md5 = "09d890c54c7456c818102b7ff8c182c8";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pytest-cov = super.buildPythonPackage {
     name = "pytest-cov-1.8.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [py pytest coverage cov-core];
     src = fetchurl {
       url = "https://pypi.python.org/packages/11/4b/b04646e97f1721878eb21e9f779102d84dd044d324382263b1770a3e4838/pytest-cov-1.8.1.tar.gz";
       md5 = "76c778afa2494088270348be42d759fc";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pytest-profiling = super.buildPythonPackage {
     name = "pytest-profiling-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six pytest gprof2dot];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d8/67/8ffab73406e22870e07fa4dc8dce1d7689b26dba8efd00161c9b6fc01ec0/pytest-profiling-1.0.1.tar.gz";
       md5 = "354404eb5b3fd4dc5eb7fffbb3d9b68b";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pytest-runner = super.buildPythonPackage {
     name = "pytest-runner-2.7.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/99/6b/c4ff4418d3424d4475b7af60724fd4a5cdd91ed8e489dc9443281f0052bc/pytest-runner-2.7.1.tar.gz";
       md5 = "e56f0bc8d79a6bd91772b44ef4215c7e";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pytest-timeout = super.buildPythonPackage {
     name = "pytest-timeout-0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytest];
     src = fetchurl {
       url = "https://pypi.python.org/packages/24/48/5f6bd4b8026a26e1dd427243d560a29a0f1b24a5c7cffca4bf049a7bb65b/pytest-timeout-0.4.tar.gz";
       md5 = "03b28aff69cbbfb959ed35ade5fde262";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit { fullName = "DFSG approved"; } ];
+    };
   };
   python-dateutil = super.buildPythonPackage {
     name = "python-dateutil-1.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b4/7c/df59c89a753eb33c7c44e1dd42de0e9bc2ccdd5a4d576e0bfad97cc280cb/python-dateutil-1.5.tar.gz";
       md5 = "0dcb1de5e5cad69490a3b6ab63f0cfa5";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.psfl ];
+    };
   };
   python-editor = super.buildPythonPackage {
     name = "python-editor-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2b/c0/df7b87d5cf016f82eab3b05cd35f53287c1178ad8c42bfb6fa61b89b22f6/python-editor-1.0.1.tar.gz";
       md5 = "e1fa63535b40e022fa4fd646fd8b511a";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
   };
   python-ldap = super.buildPythonPackage {
     name = "python-ldap-2.4.19";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/42/81/1b64838c82e64f14d4e246ff00b52e650a35c012551b891ada2b85d40737/python-ldap-2.4.19.tar.gz";
       md5 = "b941bf31d09739492aa19ef679e94ae3";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.psfl ];
+    };
   };
   python-memcached = super.buildPythonPackage {
     name = "python-memcached-1.57";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/52/9d/eebc0dcbc5c7c66840ad207dfc1baa376dadb74912484bff73819cce01e6/python-memcached-1.57.tar.gz";
       md5 = "de21f64b42b2d961f3d4ad7beb5468a1";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.psfl ];
+    };
   };
   python-pam = super.buildPythonPackage {
     name = "python-pam-1.8.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/de/8c/f8f5d38b4f26893af267ea0b39023d4951705ab0413a39e0cf7cf4900505/python-pam-1.8.2.tar.gz";
       md5 = "db71b6b999246fb05d78ecfbe166629d";
     };
+    meta = {
+      license = [ { fullName = "License :: OSI Approved :: MIT License"; } pkgs.lib.licenses.mit ];
+    };
   };
   pytz = super.buildPythonPackage {
     name = "pytz-2015.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/7e/1a/f43b5c92df7b156822030fed151327ea096bcf417e45acc23bd1df43472f/pytz-2015.4.zip";
       md5 = "233f2a2b370d03f9b5911700cc9ebf3c";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   pyzmq = super.buildPythonPackage {
     name = "pyzmq-14.6.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8a/3b/5463d5a9d712cd8bbdac335daece0d69f6a6792da4e3dd89956c0db4e4e6/pyzmq-14.6.0.tar.gz";
       md5 = "395b5de95a931afa5b14c9349a5b8024";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal { fullName = "LGPL+BSD"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
+    };
   };
   recaptcha-client = super.buildPythonPackage {
     name = "recaptcha-client-1.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0a/ea/5f2fbbfd894bdac1c68ef8d92019066cfcf9fbff5fe3d728d2b5c25c8db4/recaptcha-client-1.0.6.tar.gz";
       md5 = "74228180f7e1fb76c4d7089160b0d919";
     };
+    meta = {
+      license = [ { fullName = "MIT/X11"; } ];
+    };
   };
   repoze.lru = super.buildPythonPackage {
     name = "repoze.lru-0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6e/1e/aa15cc90217e086dc8769872c8778b409812ff036bf021b15795638939e4/repoze.lru-0.6.tar.gz";
       md5 = "2c3b64b17a8e18b405f55d46173e14dd";
     };
+    meta = {
+      license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   requests = super.buildPythonPackage {
     name = "requests-2.9.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f9/6d/07c44fb1ebe04d069459a189e7dab9e4abfe9432adcd4477367c25332748/requests-2.9.1.tar.gz";
       md5 = "0b7f480d19012ec52bab78292efd976d";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
   };
   rhodecode-enterprise-ce = super.buildPythonPackage {
-    name = "rhodecode-enterprise-ce-4.1.2";
+    name = "rhodecode-enterprise-ce-4.2.0";
     buildInputs = with self; [WebTest configobj cssselect flake8 lxml mock pytest pytest-cov pytest-runner];
     doCheck = true;
     propagatedBuildInputs = with self; [Babel Beaker FormEncode Mako Markdown MarkupSafe MySQL-python Paste PasteDeploy PasteScript Pygments Pylons Pyro4 Routes SQLAlchemy Tempita URLObject WebError WebHelpers WebHelpers2 WebOb WebTest Whoosh alembic amqplib anyjson appenlight-client authomatic backport-ipaddress celery colander decorator docutils gunicorn infrae.cache ipython iso8601 kombu msgpack-python packaging psycopg2 pycrypto pycurl pyparsing pyramid pyramid-debugtoolbar pyramid-mako pyramid-beaker pysqlite python-dateutil python-ldap python-memcached python-pam recaptcha-client repoze.lru requests simplejson waitress zope.cachedescriptors psutil py-bcrypt];
     src = ./.;
+    meta = {
+      license = [ { fullName = "AGPLv3, and Commercial License"; } ];
+    };
   };
   rhodecode-tools = super.buildPythonPackage {
     name = "rhodecode-tools-0.8.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [click future six Mako MarkupSafe requests Whoosh elasticsearch elasticsearch-dsl];
     src = fetchurl {
       url = "https://code.rhodecode.com/rhodecode-tools-ce/archive/v0.8.3.zip";
       md5 = "9acdfd71b8ddf4056057065f37ab9ccb";
     };
+    meta = {
+      license = [ { fullName = "AGPLv3 and Proprietary"; } ];
+    };
   };
   serpent = super.buildPythonPackage {
     name = "serpent-1.12";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3b/19/1e0e83b47c09edaef8398655088036e7e67386b5c48770218ebb339fbbd5/serpent-1.12.tar.gz";
       md5 = "05869ac7b062828b34f8f927f0457b65";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   setproctitle = super.buildPythonPackage {
     name = "setproctitle-1.1.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/33/c3/ad367a4f4f1ca90468863ae727ac62f6edb558fc09a003d344a02cfc6ea6/setproctitle-1.1.8.tar.gz";
       md5 = "728f4c8c6031bbe56083a48594027edd";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.bsdOriginal ];
+    };
   };
   setuptools = super.buildPythonPackage {
     name = "setuptools-20.8.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c4/19/c1bdc88b53da654df43770f941079dbab4e4788c2dcb5658fb86259894c7/setuptools-20.8.1.zip";
       md5 = "fe58a5cac0df20bb83942b252a4b0543";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   setuptools-scm = super.buildPythonPackage {
     name = "setuptools-scm-1.11.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/cd/5f/e3a038292358058d83d764a47d09114aa5a8003ed4529518f9e580f1a94f/setuptools_scm-1.11.0.tar.gz";
       md5 = "4c5c896ba52e134bbc3507bac6400087";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   simplejson = super.buildPythonPackage {
     name = "simplejson-3.7.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6d/89/7f13f099344eea9d6722779a1f165087cb559598107844b1ac5dbd831fb1/simplejson-3.7.2.tar.gz";
       md5 = "a5fc7d05d4cb38492285553def5d4b46";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit pkgs.lib.licenses.afl21 ];
+    };
   };
   six = super.buildPythonPackage {
     name = "six-1.9.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/16/64/1dc5e5976b17466fd7d712e59cbe9fb1e18bec153109e5ba3ed6c9102f1a/six-1.9.0.tar.gz";
       md5 = "476881ef4012262dfc8adc645ee786c4";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   subprocess32 = super.buildPythonPackage {
     name = "subprocess32-3.2.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/28/8d/33ccbff51053f59ae6c357310cac0e79246bbed1d345ecc6188b176d72c3/subprocess32-3.2.6.tar.gz";
       md5 = "754c5ab9f533e764f931136974b618f1";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.psfl ];
+    };
   };
   supervisor = super.buildPythonPackage {
     name = "supervisor-3.1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [meld3];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a6/41/65ad5bd66230b173eb4d0b8810230f3a9c59ef52ae066e540b6b99895db7/supervisor-3.1.3.tar.gz";
       md5 = "aad263c4fbc070de63dd354864d5e552";
     };
+    meta = {
+      license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   transifex-client = super.buildPythonPackage {
     name = "transifex-client-0.10";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f3/4e/7b925192aee656fb3e04fa6381c8b3dc40198047c3b4a356f6cfd642c809/transifex-client-0.10.tar.gz";
       md5 = "5549538d84b8eede6b254cd81ae024fa";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.gpl2 ];
+    };
   };
   translationstring = super.buildPythonPackage {
     name = "translationstring-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5e/eb/bee578cc150b44c653b63f5ebe258b5d0d812ddac12497e5f80fcad5d0b4/translationstring-1.3.tar.gz";
       md5 = "a4b62e0f3c189c783a1685b3027f7c90";
     };
+    meta = {
+      license = [ { fullName = "BSD-like (http://repoze.org/license.html)"; } ];
+    };
   };
   trollius = super.buildPythonPackage {
     name = "trollius-1.0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [futures];
     src = fetchurl {
       url = "https://pypi.python.org/packages/aa/e6/4141db437f55e6ee7a3fb69663239e3fde7841a811b4bef293145ad6c836/trollius-1.0.4.tar.gz";
       md5 = "3631a464d49d0cbfd30ab2918ef2b783";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.asl20 ];
+    };
   };
   uWSGI = super.buildPythonPackage {
     name = "uWSGI-2.0.11.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/9b/78/918db0cfab0546afa580c1e565209c49aaf1476bbfe491314eadbe47c556/uwsgi-2.0.11.2.tar.gz";
       md5 = "1f02dcbee7f6f61de4b1fd68350cf16f";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.gpl2 ];
+    };
   };
   urllib3 = super.buildPythonPackage {
     name = "urllib3-1.16";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3b/f0/e763169124e3f5db0926bc3dbfcd580a105f9ca44cf5d8e6c7a803c9f6b5/urllib3-1.16.tar.gz";
       md5 = "fcaab1c5385c57deeb7053d3d7d81d59";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
   };
   venusian = super.buildPythonPackage {
     name = "venusian-1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/86/20/1948e0dfc4930ddde3da8c33612f6a5717c0b4bc28f591a5c5cf014dd390/venusian-1.0.tar.gz";
       md5 = "dccf2eafb7113759d60c86faf5538756";
     };
+    meta = {
+      license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
+    };
   };
   waitress = super.buildPythonPackage {
     name = "waitress-0.8.9";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ee/65/fc9dee74a909a1187ca51e4f15ad9c4d35476e4ab5813f73421505c48053/waitress-0.8.9.tar.gz";
       md5 = "da3f2e62b3676be5dd630703a68e2a04";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 ];
+    };
   };
   wsgiref = super.buildPythonPackage {
     name = "wsgiref-0.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/41/9e/309259ce8dff8c596e8c26df86dbc4e848b9249fd36797fd60be456f03fc/wsgiref-0.1.2.zip";
       md5 = "29b146e6ebd0f9fb119fe321f7bcf6cb";
     };
+    meta = {
+      license = [ { fullName = "PSF or ZPL"; } ];
+    };
   };
   zope.cachedescriptors = super.buildPythonPackage {
     name = "zope.cachedescriptors-4.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/40/33/694b6644c37f28553f4b9f20b3c3a20fb709a22574dff20b5bdffb09ecd5/zope.cachedescriptors-4.0.0.tar.gz";
       md5 = "8d308de8c936792c8e758058fcb7d0f0";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 ];
+    };
   };
   zope.deprecation = super.buildPythonPackage {
     name = "zope.deprecation-4.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c1/d3/3919492d5e57d8dd01b36f30b34fc8404a30577392b1eb817c303499ad20/zope.deprecation-4.1.2.tar.gz";
       md5 = "e9a663ded58f4f9f7881beb56cae2782";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 ];
+    };
   };
   zope.event = super.buildPythonPackage {
     name = "zope.event-4.0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c1/29/91ba884d7d6d96691df592e9e9c2bfa57a47040ec1ff47eff18c85137152/zope.event-4.0.3.tar.gz";
       md5 = "9a3780916332b18b8b85f522bcc3e249";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 ];
+    };
   };
   zope.interface = super.buildPythonPackage {
     name = "zope.interface-4.1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/9d/81/2509ca3c6f59080123c1a8a97125eb48414022618cec0e64eb1313727bfe/zope.interface-4.1.3.tar.gz";
       md5 = "9ae3d24c0c7415deb249dd1a132f0f79";
     };
+    meta = {
+      license = [ pkgs.lib.licenses.zpt21 ];
+    };
   };
 
 ### Test requirements
 
   
 }

release.nix

@@ -1,231 +1,232 b''
 #
 # About
 # =====
 #
 # This file defines jobs for our CI system and the attribute "build" is used
 # as the input for packaging.
 #
 #
 # CI details
 # ==========
 #
 # This file defines an attribute set of derivations. Each of these attributes is
 # then used in our CI system as one job to run. This way we keep the
 # configuration for the CI jobs as well under version control.
 #
 # Run CI jobs locally
 # -------------------
 #
 # Since it is all based on normal Nix derivations, the jobs can be tested
 # locally with a run of "nix-build" like the following example:
 #
 #    nix-build release.nix -A test-api -I vcsserver=~/rhodecode-vcsserver
 #
 # Note: Replace "~/rhodecode-vcsserver" with a path where a clone of the
 # vcsserver resides.
 
 { pkgs ? import <nixpkgs> {}
+, doCheck ? true
 }:
 
 let
 
   inherit (pkgs)
     stdenv
     system;
 
   testing = import <nixpkgs/nixos/lib/testing.nix> {
     inherit system;
   };
 
   runInMachine = testing.runInMachine;
 
   sphinx = import ./docs/default.nix {};
 
   mkDocs = kind: stdenv.mkDerivation {
     name = kind;
     srcs = [
       (./. + (builtins.toPath "/${kind}"))
       (builtins.filterSource
         (path: type: baseNameOf path == "VERSION")
         ./rhodecode)
     ];
     sourceRoot = kind;
     buildInputs = [ sphinx ];
     configurePhase = null;
     buildPhase = ''
       make SPHINXBUILD=sphinx-build html
     '';
     installPhase = ''
       mkdir -p $out
       mv _build/html $out/
 
       mkdir -p $out/nix-support
       echo "doc manual $out/html index.html" >> \
         "$out/nix-support/hydra-build-products"
     '';
   };
 
   enterprise = import ./default.nix {
     inherit
       pkgs;
 
     # TODO: for quick local testing
     doCheck = false;
   };
 
   test-cfg = stdenv.mkDerivation {
     name = "test-cfg";
     unpackPhase = "true";
     buildInputs = [
       enterprise.src
     ];
     installPhase = ''
       mkdir -p $out/etc
       cp ${enterprise.src}/test.ini $out/etc/enterprise.ini
       # TODO: johbo: Needed, so that the login works, this causes
       # probably some side effects
       substituteInPlace $out/etc/enterprise.ini --replace "is_test = True" ""
 
       # Gevent configuration
       cp $out/etc/enterprise.ini $out/etc/enterprise-gevent.ini;
       cat >> $out/etc/enterprise-gevent.ini <<EOF
 
       [server:main]
       use = egg:gunicorn#main
       worker_class = gevent
       EOF
 
       cp ${enterprise.src}/vcsserver/test.ini $out/etc/vcsserver.ini
     '';
   };
 
   ac-test-drv = import ./acceptance_tests {
     withExternals = false;
   };
 
   # TODO: johbo: Currently abusing buildPythonPackage to make the
   # needed environment for the ac-test tools.
   mkAcTests = {
     # Path to an INI file which will be used to run Enterprise.
     #
     # Intended usage is to provide different configuration files to
     # run the tests against a different configuration.
     enterpriseCfg ? "${test-cfg}/etc/enterprise.ini"
 
     # Path to an INI file which will be used to run the VCSServer.
   , vcsserverCfg ? "${test-cfg}/etc/vcsserver.ini"
   }: pkgs.pythonPackages.buildPythonPackage {
     name = "enterprise-ac-tests";
     src = ./acceptance_tests;
 
     buildInputs = with pkgs; [
       curl
       enterprise
       ac-test-drv
     ];
 
     buildPhase = ''
       cp ${enterpriseCfg} enterprise.ini
 
       echo "Creating a fake home directory"
       mkdir fake-home
       export HOME=$PWD/fake-home
 
       echo "Creating a repository directory"
       mkdir repos
 
       echo "Preparing the database"
       paster setup-rhodecode \
         --user=admin \
         --email=[e-mail] \
         --password=secret \
         --api-key=9999999999999999999999999999999999999999 \
         --force-yes \
         --repos=$PWD/repos \
         enterprise.ini > /dev/null
 
       echo "Starting rcserver"
       vcsserver --config ${vcsserverCfg} >vcsserver.log 2>&1 &
       rcserver enterprise.ini >rcserver.log 2>&1 &
 
       while ! curl -f -s http://localhost:5000 > /dev/null
       do
           echo "Waiting for server to be ready..."
           sleep 3
       done
       echo "Webserver is ready."
 
       echo "Starting the test run"
       py.test -c example.ini -vs --maxfail=5 tests
 
       echo "Kill rcserver"
       kill %2
       kill %1
     '';
 
     # TODO: johbo: Use the install phase again once the normal mkDerivation
     # can be used again.
     postInstall = ''
       mkdir -p $out
       cp enterprise.ini $out
       cp ${vcsserverCfg} $out/vcsserver.ini
       cp rcserver.log $out
       cp vcsserver.log $out
 
       mkdir -p $out/nix-support
       echo "report config $out enterprise.ini" >> $out/nix-support/hydra-build-products
       echo "report config $out vcsserver.ini" >> $out/nix-support/hydra-build-products
       echo "report rcserver $out rcserver.log" >> $out/nix-support/hydra-build-products
       echo "report vcsserver $out vcsserver.log" >> $out/nix-support/hydra-build-products
     '';
   };
 
   vcsserver = import <vcsserver> {
     inherit pkgs;
 
     # TODO: johbo: Think of a more elegant solution to this problem
     pythonExternalOverrides = self: super: (enterprise.myPythonPackagesUnfix self);
   };
 
   runTests = optionString: (enterprise.override (attrs: {
     doCheck = true;
     name = "test-run";
     buildInputs = attrs.buildInputs ++ [
       vcsserver
     ];
     checkPhase = ''
       py.test ${optionString} -vv -ra
     '';
     buildPhase = attrs.shellHook;
     installPhase = ''
       echo "Intentionally not installing anything"
     '';
     meta.description = "Enterprise test run ${optionString}";
   }));
 
   jobs = {
 
     build = enterprise;
 
     # johbo: Currently this is simply running the tests against the sources. Nicer
     # would be to run xdist and against the installed application, so that we also
     # cover the impact of installing the application.
     test-api = runTests "rhodecode/api";
     test-functional = runTests "rhodecode/tests/functional";
     test-rest = runTests "rhodecode/tests --ignore=rhodecode/tests/functional";
     test-full = runTests "rhodecode";
 
     docs = mkDocs "docs";
 
     aggregate = pkgs.releaseTools.aggregate {
       name = "aggregated-jobs";
       constituents = [
         jobs.build
         jobs.test-api
         jobs.test-rest
         jobs.docs
       ];
     };
   };
 
 in jobs

requirements.txt

@@ -1,151 +1,150 b''
 Babel==1.3
 Beaker==1.7.0
 CProfileV==1.0.6
 Fabric==1.10.0
 FormEncode==1.2.4
 Jinja2==2.7.3
 Mako==1.0.1
 Markdown==2.6.2
 MarkupSafe==0.23
 MySQL-python==1.2.5
 Paste==2.0.2
 PasteDeploy==1.5.2
 PasteScript==1.7.5
 Pygments==2.0.2
 
 # TODO: This version is not available on PyPI
 # Pylons==1.0.2.dev20160108
 Pylons==1.0.1
 
 # TODO: This version is not available, but newer ones are
 # Pyro4==4.35
 Pyro4==4.41
 
 # TODO: This should probably not be in here
 # -e hg+https://[e-mail]/johbo/rhodecode-fork@3a454bd1f17c0b2b2a951cf2b111e0320d7942a9#egg=RhodeCodeEnterprise-dev
 
 # TODO: This is not really a dependency, we should add it only
 # into the development environment, since there it is useful.
 # RhodeCodeVCSServer==3.9.0
 
 Routes==1.13
 SQLAlchemy==0.9.9
 Sphinx==1.2.2
 Tempita==0.5.2
 URLObject==2.4.0
 WebError==0.10.3
 
 # TODO: This is modified by us, needs a better integration. For now
 # using the latest version before.
 # WebHelpers==1.3.dev20150807
 WebHelpers==1.3
 
 WebHelpers2==2.0
 WebOb==1.3.1
 WebTest==1.4.3
 Whoosh==2.7.0
 alembic==0.8.4
 amqplib==1.0.2
 anyjson==0.3.3
 appenlight-client==0.6.14
 authomatic==0.1.0.post1;
 backport-ipaddress==0.1
 bottle==0.12.8
 bumpversion==0.5.3
 celery==2.2.10
 click==5.1
 colander==1.2
 configobj==5.0.6
 cov-core==1.15.0
 coverage==3.7.1
 cssselect==0.9.1
 decorator==3.4.2
 docutils==0.12
 dogpile.cache==0.5.7
 dogpile.core==0.4.1
 dulwich==0.12.0
 ecdsa==0.11
 flake8==2.4.1
 future==0.14.3
 futures==3.0.2
 gprof2dot==2015.12.1
-greenlet==0.4.9
 gunicorn==19.6.0
 
 # TODO: Needs subvertpy and blows up without Subversion headers,
 # actually we should not need this for Enterprise at all.
 # hgsubversion==1.8.2
 
 gnureadline==6.3.3
 infrae.cache==1.0.1
-invoke==0.11.1
+invoke==0.13.0
 ipdb==0.8
 ipython==3.1.0
 iso8601==0.1.11
 itsdangerous==0.24
 kombu==1.5.1
 lxml==3.4.4
 mccabe==0.3
 meld3==1.0.2
 mock==1.0.1
 msgpack-python==0.4.6
 nose==1.3.6
 objgraph==2.0.0
 packaging==15.2
 paramiko==1.15.1
 pep8==1.5.7
 psutil==2.2.1
 psycopg2==2.6
 py==1.4.29
 py-bcrypt==0.4
 pycrypto==2.6.1
 pycurl==7.19.5
 pyflakes==0.8.1
 pyparsing==1.5.7
 pyramid==1.6.1
 pyramid-beaker==0.8
 pyramid-debugtoolbar==2.4.2
 pyramid-jinja2==2.5
 pyramid-mako==1.0.2
 pysqlite==2.6.3
 pytest==2.8.5
 pytest-runner==2.7.1
 pytest-catchlog==1.2.2
 pytest-cov==1.8.1
 pytest-profiling==1.0.1
 pytest-timeout==0.4
 python-dateutil==1.5
 python-ldap==2.4.19
 python-memcached==1.57
 python-pam==1.8.2
 pytz==2015.4
 pyzmq==14.6.0
 
 # TODO: This is not available in public
 # rc-testdata==0.2.0
 
 https://code.rhodecode.com/rhodecode-tools-ce/archive/v0.8.3.zip#md5=9acdfd71b8ddf4056057065f37ab9ccb
 
 
 recaptcha-client==1.0.6
 repoze.lru==0.6
 requests==2.9.1
 serpent==1.12
 setproctitle==1.1.8
 setuptools==20.8.1
 setuptools-scm==1.11.0
 simplejson==3.7.2
 six==1.9.0
 subprocess32==3.2.6
 supervisor==3.1.3
 transifex-client==0.10
 translationstring==1.3
 trollius==1.0.4
 uWSGI==2.0.11.2
 venusian==1.0
 waitress==0.8.9
 wsgiref==0.1.2
 zope.cachedescriptors==4.0.0
 zope.deprecation==4.1.2
 zope.event==4.0.3
 zope.interface==4.1.3

rhodecode/VERSION

@@ -1,1 +1,1 b''
-4.1.2
  No newline at end of file
+4.2.0
  No newline at end of file

rhodecode/authentication/base.py

@@ -1,609 +1,615 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 Authentication modules
 """
 
+import colander
 import logging
 import time
 import traceback
 import warnings
 
 from pyramid.threadlocal import get_current_registry
 from sqlalchemy.ext.hybrid import hybrid_property
 
 from rhodecode.authentication.interface import IAuthnPluginRegistry
 from rhodecode.authentication.schema import AuthnPluginSettingsSchemaBase
 from rhodecode.lib import caches
 from rhodecode.lib.auth import PasswordGenerator, _RhodeCodeCryptoBCrypt
 from rhodecode.lib.utils2 import md5_safe, safe_int
 from rhodecode.lib.utils2 import safe_str
 from rhodecode.model.db import User
 from rhodecode.model.meta import Session
 from rhodecode.model.settings import SettingsModel
 from rhodecode.model.user import UserModel
 from rhodecode.model.user_group import UserGroupModel
 
 
 log = logging.getLogger(__name__)
 
 # auth types that authenticate() function can receive
 VCS_TYPE = 'vcs'
 HTTP_TYPE = 'http'
 
 
 class LazyFormencode(object):
     def __init__(self, formencode_obj, *args, **kwargs):
         self.formencode_obj = formencode_obj
         self.args = args
         self.kwargs = kwargs
 
     def __call__(self, *args, **kwargs):
         from inspect import isfunction
         formencode_obj = self.formencode_obj
         if isfunction(formencode_obj):
             # case we wrap validators into functions
             formencode_obj = self.formencode_obj(*args, **kwargs)
         return formencode_obj(*self.args, **self.kwargs)
 
 
 class RhodeCodeAuthPluginBase(object):
     # cache the authentication request for N amount of seconds. Some kind
     # of authentication methods are very heavy and it's very efficient to cache
     # the result of a call. If it's set to None (default) cache is off
     AUTH_CACHE_TTL = None
     AUTH_CACHE = {}
 
     auth_func_attrs = {
         "username": "unique username",
         "firstname": "first name",
         "lastname": "last name",
         "email": "email address",
         "groups": '["list", "of", "groups"]',
         "extern_name": "name in external source of record",
         "extern_type": "type of external source of record",
         "admin": 'True|False defines if user should be RhodeCode super admin',
         "active":
             'True|False defines active state of user internally for RhodeCode',
         "active_from_extern":
             "True|False\None, active state from the external auth, "
             "None means use definition from RhodeCode extern_type active value"
     }
     # set on authenticate() method and via set_auth_type func.
     auth_type = None
 
     # List of setting names to store encrypted. Plugins may override this list
     # to store settings encrypted.
     _settings_encrypted = []
 
     # Mapping of python to DB settings model types. Plugins may override or
     # extend this mapping.
     _settings_type_map = {
-        str: 'str',
-        int: 'int',
-        unicode: 'unicode',
-        bool: 'bool',
-        list: 'list',
+        colander.String: 'unicode',
+        colander.Integer: 'int',
+        colander.Boolean: 'bool',
+        colander.List: 'list',
     }
 
     def __init__(self, plugin_id):
         self._plugin_id = plugin_id
 
+    def __str__(self):
+        return self.get_id()
+
     def _get_setting_full_name(self, name):
         """
         Return the full setting name used for storing values in the database.
         """
         # TODO: johbo: Using the name here is problematic. It would be good to
         # introduce either new models in the database to hold Plugin and
         # PluginSetting or to use the plugin id here.
         return 'auth_{}_{}'.format(self.name, name)
 
-    def _get_setting_type(self, name, value):
+    def _get_setting_type(self, name):
+        """
+        Return the type of a setting. This type is defined by the SettingsModel
+        and determines how the setting is stored in DB. Optionally the suffix
+        `.encrypted` is appended to instruct SettingsModel to store it
+        encrypted.
         """
-        Get the type as used by the SettingsModel accordingly to type of passed
-        value. Optionally the suffix `.encrypted` is appended to instruct
-        SettingsModel to store it encrypted.
-        """
-        type_ = self._settings_type_map.get(type(value), 'unicode')
+        schema_node = self.get_settings_schema().get(name)
+        db_type = self._settings_type_map.get(
+            type(schema_node.typ), 'unicode')
         if name in self._settings_encrypted:
-            type_ = '{}.encrypted'.format(type_)
-        return type_
+            db_type = '{}.encrypted'.format(db_type)
+        return db_type
 
     def is_enabled(self):
         """
         Returns true if this plugin is enabled. An enabled plugin can be
         configured in the admin interface but it is not consulted during
         authentication.
         """
         auth_plugins = SettingsModel().get_auth_plugins()
         return self.get_id() in auth_plugins
 
     def is_active(self):
         """
         Returns true if the plugin is activated. An activated plugin is
         consulted during authentication, assumed it is also enabled.
         """
         return self.get_setting_by_name('enabled')
 
     def get_id(self):
         """
         Returns the plugin id.
         """
         return self._plugin_id
 
     def get_display_name(self):
         """
         Returns a translation string for displaying purposes.
         """
         raise NotImplementedError('Not implemented in base class')
 
     def get_settings_schema(self):
         """
         Returns a colander schema, representing the plugin settings.
         """
         return AuthnPluginSettingsSchemaBase()
 
-    def get_setting_by_name(self, name):
+    def get_setting_by_name(self, name, default=None):
         """
         Returns a plugin setting by name.
         """
         full_name = self._get_setting_full_name(name)
         db_setting = SettingsModel().get_setting_by_name(full_name)
-        return db_setting.app_settings_value if db_setting else None
+        return db_setting.app_settings_value if db_setting else default
 
     def create_or_update_setting(self, name, value):
         """
         Create or update a setting for this plugin in the persistent storage.
         """
         full_name = self._get_setting_full_name(name)
-        type_ = self._get_setting_type(name, value)
+        type_ = self._get_setting_type(name)
         db_setting = SettingsModel().create_or_update_setting(
             full_name, value, type_)
         return db_setting.app_settings_value
 
     def get_settings(self):
         """
         Returns the plugin settings as dictionary.
         """
         settings = {}
         for node in self.get_settings_schema():
             settings[node.name] = self.get_setting_by_name(node.name)
         return settings
 
     @property
     def validators(self):
         """
         Exposes RhodeCode validators modules
         """
         # this is a hack to overcome issues with pylons threadlocals and
         # translator object _() not beein registered properly.
         class LazyCaller(object):
             def __init__(self, name):
                 self.validator_name = name
 
             def __call__(self, *args, **kwargs):
                 from rhodecode.model import validators as v
                 obj = getattr(v, self.validator_name)
                 # log.debug('Initializing lazy formencode object: %s', obj)
                 return LazyFormencode(obj, *args, **kwargs)
 
         class ProxyGet(object):
             def __getattribute__(self, name):
                 return LazyCaller(name)
 
         return ProxyGet()
 
     @hybrid_property
     def name(self):
         """
         Returns the name of this authentication plugin.
 
         :returns: string
         """
         raise NotImplementedError("Not implemented in base class")
 
     @property
     def is_headers_auth(self):
         """
         Returns True if this authentication plugin uses HTTP headers as
         authentication method.
         """
         return False
 
     @hybrid_property
     def is_container_auth(self):
         """
         Deprecated method that indicates if this authentication plugin uses
         HTTP headers as authentication method.
         """
         warnings.warn(
             'Use is_headers_auth instead.', category=DeprecationWarning)
         return self.is_headers_auth
 
     @hybrid_property
     def allows_creating_users(self):
         """
         Defines if Plugin allows users to be created on-the-fly when
         authentication is called. Controls how external plugins should behave
         in terms if they are allowed to create new users, or not. Base plugins
         should not be allowed to, but External ones should be !
 
         :return: bool
         """
         return False
 
     def set_auth_type(self, auth_type):
         self.auth_type = auth_type
 
     def allows_authentication_from(
             self, user, allows_non_existing_user=True,
             allowed_auth_plugins=None, allowed_auth_sources=None):
         """
         Checks if this authentication module should accept a request for
         the current user.
 
         :param user: user object fetched using plugin's get_user() method.
         :param allows_non_existing_user: if True, don't allow the
             user to be empty, meaning not existing in our database
         :param allowed_auth_plugins: if provided, users extern_type will be
             checked against a list of provided extern types, which are plugin
             auth_names in the end
         :param allowed_auth_sources: authentication type allowed,
             `http` or `vcs` default is both.
             defines if plugin will accept only http authentication vcs
             authentication(git/hg) or both
         :returns: boolean
         """
         if not user and not allows_non_existing_user:
             log.debug('User is empty but plugin does not allow empty users,'
                       'not allowed to authenticate')
             return False
 
         expected_auth_plugins = allowed_auth_plugins or [self.name]
         if user and (user.extern_type and
                      user.extern_type not in expected_auth_plugins):
             log.debug(
                 'User `%s` is bound to `%s` auth type. Plugin allows only '
                 '%s, skipping', user, user.extern_type, expected_auth_plugins)
 
             return False
 
         # by default accept both
         expected_auth_from = allowed_auth_sources or [HTTP_TYPE, VCS_TYPE]
         if self.auth_type not in expected_auth_from:
             log.debug('Current auth source is %s but plugin only allows %s',
                       self.auth_type, expected_auth_from)
             return False
 
         return True
 
     def get_user(self, username=None, **kwargs):
         """
         Helper method for user fetching in plugins, by default it's using
         simple fetch by username, but this method can be custimized in plugins
         eg. headers auth plugin to fetch user by environ params
 
         :param username: username if given to fetch from database
         :param kwargs: extra arguments needed for user fetching.
         """
         user = None
         log.debug(
             'Trying to fetch user `%s` from RhodeCode database', username)
         if username:
             user = User.get_by_username(username)
             if not user:
                 log.debug('User not found, fallback to fetch user in '
                           'case insensitive mode')
                 user = User.get_by_username(username, case_insensitive=True)
         else:
             log.debug('provided username:`%s` is empty skipping...', username)
         if not user:
             log.debug('User `%s` not found in database', username)
         return user
 
     def user_activation_state(self):
         """
         Defines user activation state when creating new users
 
         :returns: boolean
         """
         raise NotImplementedError("Not implemented in base class")
 
     def auth(self, userobj, username, passwd, settings, **kwargs):
         """
         Given a user object (which may be null), username, a plaintext
         password, and a settings object (containing all the keys needed as
         listed in settings()), authenticate this user's login attempt.
 
         Return None on failure. On success, return a dictionary of the form:
 
             see: RhodeCodeAuthPluginBase.auth_func_attrs
         This is later validated for correctness
         """
         raise NotImplementedError("not implemented in base class")
 
     def _authenticate(self, userobj, username, passwd, settings, **kwargs):
         """
         Wrapper to call self.auth() that validates call on it
 
         :param userobj: userobj
         :param username: username
         :param passwd: plaintext password
         :param settings: plugin settings
         """
         auth = self.auth(userobj, username, passwd, settings, **kwargs)
         if auth:
             # check if hash should be migrated ?
             new_hash = auth.get('_hash_migrate')
             if new_hash:
                 self._migrate_hash_to_bcrypt(username, passwd, new_hash)
             return self._validate_auth_return(auth)
         return auth
 
     def _migrate_hash_to_bcrypt(self, username, password, new_hash):
         new_hash_cypher = _RhodeCodeCryptoBCrypt()
         # extra checks, so make sure new hash is correct.
         password_encoded = safe_str(password)
         if new_hash and new_hash_cypher.hash_check(
                 password_encoded, new_hash):
             cur_user = User.get_by_username(username)
             cur_user.password = new_hash
             Session().add(cur_user)
             Session().flush()
             log.info('Migrated user %s hash to bcrypt', cur_user)
 
     def _validate_auth_return(self, ret):
         if not isinstance(ret, dict):
             raise Exception('returned value from auth must be a dict')
         for k in self.auth_func_attrs:
             if k not in ret:
                 raise Exception('Missing %s attribute from returned data' % k)
         return ret
 
 
 class RhodeCodeExternalAuthPlugin(RhodeCodeAuthPluginBase):
 
     @hybrid_property
     def allows_creating_users(self):
         return True
 
     def use_fake_password(self):
         """
         Return a boolean that indicates whether or not we should set the user's
         password to a random value when it is authenticated by this plugin.
         If your plugin provides authentication, then you will generally
         want this.
 
         :returns: boolean
         """
         raise NotImplementedError("Not implemented in base class")
 
     def _authenticate(self, userobj, username, passwd, settings, **kwargs):
         # at this point _authenticate calls plugin's `auth()` function
         auth = super(RhodeCodeExternalAuthPlugin, self)._authenticate(
             userobj, username, passwd, settings, **kwargs)
         if auth:
             # maybe plugin will clean the username ?
             # we should use the return value
             username = auth['username']
 
             # if external source tells us that user is not active, we should
             # skip rest of the process. This can prevent from creating users in
             # RhodeCode when using external authentication, but if it's
             # inactive user we shouldn't create that user anyway
             if auth['active_from_extern'] is False:
                 log.warning(
                     "User %s authenticated against %s, but is inactive",
                     username, self.__module__)
                 return None
 
             cur_user = User.get_by_username(username, case_insensitive=True)
             is_user_existing = cur_user is not None
 
             if is_user_existing:
                 log.debug('Syncing user `%s` from '
                           '`%s` plugin', username, self.name)
             else:
                 log.debug('Creating non existing user `%s` from '
                           '`%s` plugin', username, self.name)
 
             if self.allows_creating_users:
                 log.debug('Plugin `%s` allows to '
                           'create new users', self.name)
             else:
                 log.debug('Plugin `%s` does not allow to '
                           'create new users', self.name)
 
             user_parameters = {
                 'username': username,
                 'email': auth["email"],
                 'firstname': auth["firstname"],
                 'lastname': auth["lastname"],
                 'active': auth["active"],
                 'admin': auth["admin"],
                 'extern_name': auth["extern_name"],
                 'extern_type': self.name,
                 'plugin': self,
                 'allow_to_create_user': self.allows_creating_users,
             }
 
             if not is_user_existing:
                 if self.use_fake_password():
                     # Randomize the PW because we don't need it, but don't want
                     # them blank either
                     passwd = PasswordGenerator().gen_password(length=16)
                 user_parameters['password'] = passwd
             else:
                 # Since the password is required by create_or_update method of
                 # UserModel, we need to set it explicitly.
                 # The create_or_update method is smart and recognises the
                 # password hashes as well.
                 user_parameters['password'] = cur_user.password
 
             # we either create or update users, we also pass the flag
             # that controls if this method can actually do that.
             # raises NotAllowedToCreateUserError if it cannot, and we try to.
             user = UserModel().create_or_update(**user_parameters)
             Session().flush()
             # enforce user is just in given groups, all of them has to be ones
             # created from plugins. We store this info in _group_data JSON
             # field
             try:
                 groups = auth['groups'] or []
                 UserGroupModel().enforce_groups(user, groups, self.name)
             except Exception:
                 # for any reason group syncing fails, we should
                 # proceed with login
                 log.error(traceback.format_exc())
             Session().commit()
         return auth
 
 
 def loadplugin(plugin_id):
     """
     Loads and returns an instantiated authentication plugin.
     Returns the RhodeCodeAuthPluginBase subclass on success,
     or None on failure.
     """
     # TODO: Disusing pyramids thread locals to retrieve the registry.
     authn_registry = get_current_registry().getUtility(IAuthnPluginRegistry)
     plugin = authn_registry.get_plugin(plugin_id)
     if plugin is None:
         log.error('Authentication plugin not found: "%s"', plugin_id)
     return plugin
 
 
 def get_auth_cache_manager(custom_ttl=None):
     return caches.get_cache_manager(
         'auth_plugins', 'rhodecode.authentication', custom_ttl)
 
 
 def authenticate(username, password, environ=None, auth_type=None,
                  skip_missing=False):
     """
     Authentication function used for access control,
     It tries to authenticate based on enabled authentication modules.
 
     :param username: username can be empty for headers auth
     :param password: password can be empty for headers auth
     :param environ: environ headers passed for headers auth
     :param auth_type: type of authentication, either `HTTP_TYPE` or `VCS_TYPE`
     :param skip_missing: ignores plugins that are in db but not in environment
     :returns: None if auth failed, plugin_user dict if auth is correct
     """
     if not auth_type or auth_type not in [HTTP_TYPE, VCS_TYPE]:
         raise ValueError('auth type must be on of http, vcs got "%s" instead'
                          % auth_type)
     headers_only = environ and not (username and password)
 
     authn_registry = get_current_registry().getUtility(IAuthnPluginRegistry)
     for plugin in authn_registry.get_plugins_for_authentication():
         plugin.set_auth_type(auth_type)
         user = plugin.get_user(username)
         display_user = user.username if user else username
 
         if headers_only and not plugin.is_headers_auth:
             log.debug('Auth type is for headers only and plugin `%s` is not '
                       'headers plugin, skipping...', plugin.get_id())
             continue
 
         # load plugin settings from RhodeCode database
         plugin_settings = plugin.get_settings()
         log.debug('Plugin settings:%s', plugin_settings)
 
         log.debug('Trying authentication using ** %s **', plugin.get_id())
         # use plugin's method of user extraction.
         user = plugin.get_user(username, environ=environ,
                                settings=plugin_settings)
         display_user = user.username if user else username
         log.debug(
             'Plugin %s extracted user is `%s`', plugin.get_id(), display_user)
 
         if not plugin.allows_authentication_from(user):
             log.debug('Plugin %s does not accept user `%s` for authentication',
                       plugin.get_id(), display_user)
             continue
         else:
             log.debug('Plugin %s accepted user `%s` for authentication',
                       plugin.get_id(), display_user)
 
         log.info('Authenticating user `%s` using %s plugin',
                  display_user, plugin.get_id())
 
         _cache_ttl = 0
 
         if isinstance(plugin.AUTH_CACHE_TTL, (int, long)):
             # plugin cache set inside is more important than the settings value
             _cache_ttl = plugin.AUTH_CACHE_TTL
         elif plugin_settings.get('auth_cache_ttl'):
             _cache_ttl = safe_int(plugin_settings.get('auth_cache_ttl'), 0)
 
         plugin_cache_active = bool(_cache_ttl and _cache_ttl > 0)
 
         # get instance of cache manager configured for a namespace
         cache_manager = get_auth_cache_manager(custom_ttl=_cache_ttl)
 
         log.debug('Cache for plugin `%s` active: %s', plugin.get_id(),
                   plugin_cache_active)
 
         # for environ based password can be empty, but then the validation is
         # on the server that fills in the env data needed for authentication
         _password_hash = md5_safe(plugin.name + username + (password or ''))
 
         # _authenticate is a wrapper for .auth() method of plugin.
         # it checks if .auth() sends proper data.
         # For RhodeCodeExternalAuthPlugin it also maps users to
         # Database and maps the attributes returned from .auth()
         # to RhodeCode database. If this function returns data
         # then auth is correct.
         start = time.time()
         log.debug('Running plugin `%s` _authenticate method',
                   plugin.get_id())
 
         def auth_func():
             """
             This function is used internally in Cache of Beaker to calculate
             Results
             """
             return plugin._authenticate(
                 user, username, password, plugin_settings,
                 environ=environ or {})
 
         if plugin_cache_active:
             plugin_user = cache_manager.get(
                 _password_hash, createfunc=auth_func)
         else:
             plugin_user = auth_func()
 
         auth_time = time.time() - start
         log.debug('Authentication for plugin `%s` completed in %.3fs, '
                   'expiration time of fetched cache %.1fs.',
                   plugin.get_id(), auth_time, _cache_ttl)
 
         log.debug('PLUGIN USER DATA: %s', plugin_user)
 
         if plugin_user:
             log.debug('Plugin returned proper authentication data')
             return plugin_user
         # we failed to Auth because .auth() method didn't return proper user
         log.debug("User `%s` failed to authenticate against %s",
                   display_user, plugin.get_id())
     return None

rhodecode/authentication/views.py

@@ -1,188 +1,192 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2012-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import colander
 import formencode.htmlfill
 import logging
 
 from pyramid.httpexceptions import HTTPFound
 from pyramid.renderers import render
 from pyramid.response import Response
 
 from rhodecode.authentication.base import get_auth_cache_manager
 from rhodecode.authentication.interface import IAuthnPluginRegistry
 from rhodecode.lib import auth
 from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator
 from rhodecode.model.forms import AuthSettingsForm
 from rhodecode.model.meta import Session
 from rhodecode.model.settings import SettingsModel
 from rhodecode.translation import _
 
 log = logging.getLogger(__name__)
 
 
 class AuthnPluginViewBase(object):
 
     def __init__(self, context, request):
         self.request = request
         self.context = context
         self.plugin = context.plugin
         self._rhodecode_user = request.user
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_get(self, defaults=None, errors=None):
         """
         View that displays the plugin settings as a form.
         """
         defaults = defaults or {}
         errors = errors or {}
         schema = self.plugin.get_settings_schema()
 
-        # Get default values for the form.
+        # Compute default values for the form. Priority is:
+        # 1. Passed to this method 2. DB value 3. Schema default
         for node in schema:
-            db_value = self.plugin.get_setting_by_name(node.name)
-            defaults.setdefault(node.name, db_value)
+            if node.name not in defaults:
+                defaults[node.name] = self.plugin.get_setting_by_name(
+                    node.name, node.default)
 
         template_context = {
             'defaults': defaults,
             'errors': errors,
             'plugin': self.context.plugin,
             'resource': self.context,
         }
 
         return template_context
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_post(self):
         """
         View that validates and stores the plugin settings.
         """
         schema = self.plugin.get_settings_schema()
+        data = self.request.params
+
         try:
-            valid_data = schema.deserialize(self.request.params)
+            valid_data = schema.deserialize(data)
         except colander.Invalid, e:
             # Display error message and display form again.
             self.request.session.flash(
                 _('Errors exist when saving plugin settings. '
                   'Please check the form inputs.'),
                 queue='error')
-            defaults = schema.flatten(self.request.params)
+            defaults = {key: data[key] for key in data if key in schema}
             return self.settings_get(errors=e.asdict(), defaults=defaults)
 
         # Store validated data.
         for name, value in valid_data.items():
             self.plugin.create_or_update_setting(name, value)
         Session.commit()
 
         # Display success message and redirect.
         self.request.session.flash(
             _('Auth settings updated successfully.'),
             queue='success')
         redirect_to = self.request.resource_path(
             self.context, route_name='auth_home')
         return HTTPFound(redirect_to)
 
 
 # TODO: Ongoing migration in these views.
 # - Maybe we should also use a colander schema for these views.
 class AuthSettingsView(object):
     def __init__(self, context, request):
         self.context = context
         self.request = request
 
         # TODO: Move this into a utility function. It is needed in all view
         # classes during migration. Maybe a mixin?
 
         # Some of the decorators rely on this attribute to be present on the
         # class of the decorated method.
         self._rhodecode_user = request.user
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def index(self, defaults=None, errors=None, prefix_error=False):
         defaults = defaults or {}
         authn_registry = self.request.registry.getUtility(IAuthnPluginRegistry)
         enabled_plugins = SettingsModel().get_auth_plugins()
 
         # Create template context and render it.
         template_context = {
             'resource': self.context,
             'available_plugins': authn_registry.get_plugins(),
             'enabled_plugins': enabled_plugins,
         }
         html = render('rhodecode:templates/admin/auth/auth_settings.html',
                       template_context,
                       request=self.request)
 
         # Create form default values and fill the form.
         form_defaults = {
             'auth_plugins': ','.join(enabled_plugins)
         }
         form_defaults.update(defaults)
         html = formencode.htmlfill.render(
             html,
             defaults=form_defaults,
             errors=errors,
             prefix_error=prefix_error,
             encoding="UTF-8",
             force_defaults=False)
 
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def auth_settings(self):
         try:
             form = AuthSettingsForm()()
             form_result = form.to_python(self.request.params)
             plugins = ','.join(form_result['auth_plugins'])
             setting = SettingsModel().create_or_update_setting(
                 'auth_plugins', plugins)
             Session().add(setting)
             Session().commit()
 
             cache_manager = get_auth_cache_manager()
             cache_manager.clear()
             self.request.session.flash(
                 _('Auth settings updated successfully.'),
                 queue='success')
         except formencode.Invalid as errors:
             e = errors.error_dict or {}
             self.request.session.flash(
                 _('Errors exist when saving plugin setting. '
                   'Please check the form inputs.'),
                 queue='error')
             return self.index(
                 defaults=errors.value,
                 errors=e,
                 prefix_error=False)
         except Exception:
             log.exception('Exception in auth_settings')
             self.request.session.flash(
                 _('Error occurred during update of auth settings.'),
                 queue='error')
 
         redirect_to = self.request.resource_path(
             self.context, route_name='auth_home')
         return HTTPFound(redirect_to)

rhodecode/config/environment.py

@@ -1,192 +1,192 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 Pylons environment configuration
 """
 
 import os
 import logging
 import rhodecode
 import platform
 import re
 import io
 
 from mako.lookup import TemplateLookup
 from pylons.configuration import PylonsConfig
 from pylons.error import handle_mako_error
 from pyramid.settings import asbool
 
 # don't remove this import it does magic for celery
 from rhodecode.lib import celerypylons  # noqa
 
 import rhodecode.lib.app_globals as app_globals
 
 from rhodecode.config import utils
 from rhodecode.config.routing import make_map
 from rhodecode.config.jsroutes import generate_jsroutes_content
 
 from rhodecode.lib import helpers
 from rhodecode.lib.auth import set_available_permissions
 from rhodecode.lib.utils import (
     repo2db_mapper, make_db_config, set_rhodecode_config,
     load_rcextensions)
 from rhodecode.lib.utils2 import str2bool, aslist
 from rhodecode.lib.vcs import connect_vcs, start_vcs_server
 from rhodecode.model.scm import ScmModel
 
 log = logging.getLogger(__name__)
 
 def load_environment(global_conf, app_conf, initial=False,
                      test_env=None, test_index=None):
     """
     Configure the Pylons environment via the ``pylons.config``
     object
     """
     config = PylonsConfig()
 
 
     # Pylons paths
     root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
     paths = {
         'root': root,
         'controllers': os.path.join(root, 'controllers'),
         'static_files': os.path.join(root, 'public'),
         'templates': [os.path.join(root, 'templates')],
     }
 
     # Initialize config with the basic options
     config.init_app(global_conf, app_conf, package='rhodecode', paths=paths)
 
     # store some globals into rhodecode
     rhodecode.CELERY_ENABLED = str2bool(config['app_conf'].get('use_celery'))
     rhodecode.CELERY_EAGER = str2bool(
         config['app_conf'].get('celery.always.eager'))
 
     config['routes.map'] = make_map(config)
 
-    if asbool(config['debug']):
+    if asbool(config.get('generate_js_files', 'false')):
         jsroutes = config['routes.map'].jsroutes()
         jsroutes_file_content = generate_jsroutes_content(jsroutes)
         jsroutes_file_path = os.path.join(
             paths['static_files'], 'js', 'rhodecode', 'routes.js')
 
         with io.open(jsroutes_file_path, 'w', encoding='utf-8') as f:
             f.write(jsroutes_file_content)
 
     config['pylons.app_globals'] = app_globals.Globals(config)
     config['pylons.h'] = helpers
     rhodecode.CONFIG = config
 
     load_rcextensions(root_path=config['here'])
 
     # Setup cache object as early as possible
     import pylons
     pylons.cache._push_object(config['pylons.app_globals'].cache)
 
     # Create the Mako TemplateLookup, with the default auto-escaping
     config['pylons.app_globals'].mako_lookup = TemplateLookup(
         directories=paths['templates'],
         error_handler=handle_mako_error,
         module_directory=os.path.join(app_conf['cache_dir'], 'templates'),
         input_encoding='utf-8', default_filters=['escape'],
         imports=['from webhelpers.html import escape'])
 
     # sets the c attribute access when don't existing attribute are accessed
     config['pylons.strict_tmpl_context'] = True
 
     # Limit backends to "vcs.backends" from configuration
     backends = config['vcs.backends'] = aslist(
         config.get('vcs.backends', 'hg,git'), sep=',')
     for alias in rhodecode.BACKENDS.keys():
         if alias not in backends:
             del rhodecode.BACKENDS[alias]
     log.info("Enabled backends: %s", backends)
 
     # initialize vcs client and optionally run the server if enabled
     vcs_server_uri = config.get('vcs.server', '')
     vcs_server_enabled = str2bool(config.get('vcs.server.enable', 'true'))
     start_server = (
         str2bool(config.get('vcs.start_server', 'false')) and
         not int(os.environ.get('RC_VCSSERVER_TEST_DISABLE', '0')))
     if vcs_server_enabled and start_server:
         log.info("Starting vcsserver")
         start_vcs_server(server_and_port=vcs_server_uri,
                          protocol=utils.get_vcs_server_protocol(config),
                          log_level=config['vcs.server.log_level'])
 
     set_available_permissions(config)
     db_cfg = make_db_config(clear_session=True)
 
     repos_path = list(db_cfg.items('paths'))[0][1]
     config['base_path'] = repos_path
 
     config['vcs.hooks.direct_calls'] = _use_direct_hook_calls(config)
     config['vcs.hooks.protocol'] = _get_vcs_hooks_protocol(config)
 
     # store db config also in main global CONFIG
     set_rhodecode_config(config)
 
     # configure instance id
     utils.set_instance_id(config)
 
     # CONFIGURATION OPTIONS HERE (note: all config options will override
     # any Pylons config options)
 
     # store config reference into our module to skip import magic of pylons
     rhodecode.CONFIG.update(config)
 
     utils.configure_pyro4(config)
     utils.configure_vcs(config)
     if vcs_server_enabled:
         connect_vcs(vcs_server_uri, utils.get_vcs_server_protocol(config))
 
     import_on_startup = str2bool(config.get('startup.import_repos', False))
     if vcs_server_enabled and import_on_startup:
         repo2db_mapper(ScmModel().repo_scan(repos_path), remove_obsolete=False)
     return config
 
 
 def _use_direct_hook_calls(config):
     default_direct_hook_calls = 'false'
     direct_hook_calls = str2bool(
         config.get('vcs.hooks.direct_calls', default_direct_hook_calls))
     return direct_hook_calls
 
 
 def _get_vcs_hooks_protocol(config):
     protocol = config.get('vcs.hooks.protocol', 'pyro4').lower()
     return protocol
 
 
 def load_pyramid_environment(global_config, settings):
     # Some parts of the code expect a merge of global and app settings.
     settings_merged = global_config.copy()
     settings_merged.update(settings)
 
     # If this is a test run we prepare the test environment like
     # creating a test database, test search index and test repositories.
     # This has to be done before the database connection is initialized.
     if settings['is_test']:
         rhodecode.is_test = True
         utils.initialize_test_environment(settings_merged)
 
     # Initialize the database connection.
     utils.initialize_database(settings_merged)

rhodecode/config/jsroutes.py

@@ -1,42 +1,43 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 def generate_jsroutes_content(jsroutes):
     statements = []
     for url_name, url, fields in jsroutes:
         statements.append(
             "pyroutes.register('%s', '%s', %s);" % (url_name, url, fields))
     return u'''
 /******************************************************************************
  *                                                                            *
  *                      DO NOT CHANGE THIS FILE MANUALLY                      *
  *                                                                            *
  *                                                                            *
- *        This file is automatically generated when the app starts up.        *
+ *        This file is automatically generated when the app starts up with    *
+ *                         generate_js_files = true                           *
  *                                                                            *
  *  To add a route here pass jsroute=True to the route definition in the app  *
  *                                                                            *
  ******************************************************************************/
 function registerRCRoutes() {
     // routes registration
     %s
 }
 ''' % '\n    '.join(statements)
 

rhodecode/config/licenses.json

@@ -1,217 +1,256 b''
 {
-  "cyrus-sasl-2.1.26": {
-    "cyrus": "http://cyrusimap.web.cmu.edu/mediawiki/index.php/Downloads#Licensing"
+  "nodejs-4.3.1": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "openldap-2.4.41": {
-    "OLDAP-2.8": "http://spdx.org/licenses/OLDAP-2.8"
+  "postgresql-9.5.1": {
+    "PostgreSQL License": "http://spdx.org/licenses/PostgreSQL"
   }, 
-  "openssl-1.0.1p": {
-    "OpenSSL": "http://spdx.org/licenses/OpenSSL"
-  }, 
-  "python-2.7.10": {
-    "Python-2.0": "http://spdx.org/licenses/Python-2.0"
+  "python-2.7.11": {
+    "Python Software Foundation License version 2": "http://spdx.org/licenses/Python-2.0"
   }, 
   "python2.7-Babel-1.3": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-Beaker-1.7.0": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-FormEncode-1.2.4": {
-    "Python-2.0": "http://spdx.org/licenses/Python-2.0"
+    "Python Software Foundation License version 2": "http://spdx.org/licenses/Python-2.0"
   }, 
   "python2.7-Mako-1.0.1": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-Markdown-2.6.2": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-MarkupSafe-0.23": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-Paste-2.0.2": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-PasteDeploy-1.5.2": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-PasteScript-1.7.5": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-Pygments-2.0.2": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
-  "python2.7-Pylons-1.0.2-patch1": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+  "python2.7-Pylons-1.0.1-patch1": {
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-Pyro4-4.35": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-Routes-1.13": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-SQLAlchemy-0.9.9": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-Tempita-0.5.2": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-URLObject-2.4.0": {
-    "Unlicense": "http://spdx.org/licenses/Unlicense"
+    "The Unlicense": "http://unlicense.org/"
   }, 
   "python2.7-WebError-0.10.3": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-WebHelpers-1.3-cust1": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+  "python2.7-WebHelpers-1.3": {
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-WebHelpers2-2.0": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-WebOb-1.3.1": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-Whoosh-2.7.0-patch1": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+  "python2.7-Whoosh-2.7.0": {
+    "BSD 2-clause \"Simplified\" License": "http://spdx.org/licenses/BSD-2-Clause", 
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-alembic-0.8.4": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-amqplib-1.0.2": {
-    "LGPL-3.0": "http://spdx.org/licenses/LGPL-3.0"
+    "GNU Lesser General Public License v3.0 only": "http://spdx.org/licenses/LGPL-3.0"
   }, 
   "python2.7-anyjson-0.3.3": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
+  }, 
+  "python2.7-appenlight-client-0.6.14": {
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
-  "python2.7-appenlight_client-0.6.14": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+  "python2.7-authomatic-0.1.0.post1": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-backport_ipaddress-0.1": {
-    "Python-2.0": "http://spdx.org/licenses/Python-2.0"
+  "python2.7-backport-ipaddress-0.1": {
+    "Python Software Foundation License version 2": "http://spdx.org/licenses/Python-2.0"
   }, 
   "python2.7-celery-2.2.10": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
-  "python2.7-click-4.0": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+  "python2.7-click-5.1": {
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
+  }, 
+  "python2.7-colander-1.2": {
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
   }, 
   "python2.7-configobj-5.0.6": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-cssselect-0.9.1": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-decorator-3.4.2": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-docutils-0.12": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+    "BSD 2-clause \"Simplified\" License": "http://spdx.org/licenses/BSD-2-Clause"
+  }, 
+  "python2.7-elasticsearch-2.3.0": {
+    "Apache License 2.0": "http://spdx.org/licenses/Apache-2.0"
+  }, 
+  "python2.7-elasticsearch-dsl-2.0.0": {
+    "Apache License 2.0": "http://spdx.org/licenses/Apache-2.0"
   }, 
   "python2.7-future-0.14.3": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-futures-3.0.2": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
+  }, 
+  "python2.7-gnureadline-6.3.3": {
+    "GNU General Public License v1.0 only": "http://spdx.org/licenses/GPL-1.0"
   }, 
-  "python2.7-greenlet-0.4.7": {
-    "MIT": "http://spdx.org/licenses/MIT"
+  "python2.7-gunicorn-19.6.0": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-gunicorn-19.3.0": {
-    "MIT": "http://spdx.org/licenses/MIT"
+  "python2.7-infrae.cache-1.0.1": {
+    "Zope Public License 2.1": "http://spdx.org/licenses/ZPL-2.1"
   }, 
   "python2.7-ipython-3.1.0": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
-  }, 
-  "python2.7-kombu-1.5.1-patch1": {
-    "BSD-3-Clause": "http://spdx.org/licenses/BSD-3-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
-  "python2.7-mccabe-0.3": {
-    "expat": "http://directory.fsf.org/wiki/License:Expat"
+  "python2.7-iso8601-0.1.11": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-meld3-1.0.2": {
-    "repoze": "http://repoze.org/license.html"
+  "python2.7-kombu-1.5.1": {
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-msgpack-python-0.4.6": {
-    "Apache-2.0": "http://spdx.org/licenses/Apache-2.0"
-  }, 
-  "python2.7-objgraph-2.0.0": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "Apache License 2.0": "http://spdx.org/licenses/Apache-2.0"
   }, 
   "python2.7-packaging-15.2": {
-    "Apache-2.0": "http://spdx.org/licenses/Apache-2.0"
+    "Apache License 2.0": "http://spdx.org/licenses/Apache-2.0"
   }, 
   "python2.7-psutil-2.2.1": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-psycopg2-2.6": {
-    "LGPL-3.0+": "http://spdx.org/licenses/LGPL-3.0+"
+    "GNU Lesser General Public License v3.0 or later": "http://spdx.org/licenses/LGPL-3.0+"
   }, 
   "python2.7-py-1.4.29": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-py-bcrypt-0.4": {
-    "BSD-4-Clause": "http://spdx.org/licenses/BSD-4-Clause"
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause"
   }, 
   "python2.7-pycrypto-2.6.1": {
-    "publicDomain": null
+    "Public Domain": null
+  }, 
+  "python2.7-pycurl-7.19.5": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-pyparsing-1.5.7": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
+  }, 
+  "python2.7-pyramid-1.6.1": {
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
+  }, 
+  "python2.7-pyramid-beaker-0.8": {
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
+  }, 
+  "python2.7-pyramid-debugtoolbar-2.4.2": {
+    "BSD 4-clause \"Original\" or \"Old\" License": "http://spdx.org/licenses/BSD-4-Clause", 
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
+  }, 
+  "python2.7-pyramid-mako-1.0.2": {
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
   }, 
   "python2.7-pysqlite-2.6.3": {
-    "Libpng": "http://spdx.org/licenses/Libpng", 
-    "Zlib": "http://spdx.org/licenses/Zlib"
+    "libpng License": "http://spdx.org/licenses/Libpng", 
+    "zlib License": "http://spdx.org/licenses/Zlib"
   }, 
   "python2.7-pytest-2.8.5": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
+  }, 
+  "python2.7-pytest-runner-2.7.1": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-python-dateutil-1.5": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+    "Python Software Foundation License version 2": "http://spdx.org/licenses/Python-2.0"
+  }, 
+  "python2.7-python-editor-1.0.1": {
+    "Apache License 2.0": "http://spdx.org/licenses/Apache-2.0"
   }, 
   "python2.7-python-ldap-2.4.19": {
-    "Python-2.0": "http://spdx.org/licenses/Python-2.0"
+    "Python Software Foundation License version 2": "http://spdx.org/licenses/Python-2.0"
+  }, 
+  "python2.7-python-memcached-1.57": {
+    "Python Software Foundation License version 2": "http://spdx.org/licenses/Python-2.0"
   }, 
   "python2.7-pytz-2015.4": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-recaptcha-client-1.0.6": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-repoze.lru-0.6": {
-    "repoze": "http://repoze.org/license.html"
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
   }, 
-  "python2.7-requests-2.5.1": {
-    "APSL-2.0": "http://spdx.org/licenses/APSL-2.0"
+  "python2.7-requests-2.9.1": {
+    "Apache License 2.0": "http://spdx.org/licenses/Apache-2.0"
   }, 
-  "python2.7-serpent-1.11": {
-    "MIT": "http://spdx.org/licenses/MIT"
+  "python2.7-serpent-1.12": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-setproctitle-1.1.8": {
-    "BSD-2-Clause": "http://spdx.org/licenses/BSD-2-Clause"
+  "python2.7-setuptools-19.4": {
+    "Python Software Foundation License version 2": "http://spdx.org/licenses/Python-2.0", 
+    "Zope Public License 2.0": "http://spdx.org/licenses/ZPL-2.0"
   }, 
-  "python2.7-setuptools-18.0.1": {
-    "PSF": null, 
-    "ZPL": null
+  "python2.7-setuptools-scm-1.11.0": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-simplejson-3.7.2": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "Academic Free License": "http://spdx.org/licenses/AFL-2.1", 
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
   "python2.7-six-1.9.0": {
-    "MIT": "http://spdx.org/licenses/MIT"
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-subprocess32-3.2.6": {
-    "Python-2.0": "http://spdx.org/licenses/Python-2.0"
+  "python2.7-translationstring-1.3": {
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
   }, 
-  "python2.7-supervisor-3.1.3": {
-    "repoze": "http://repoze.org/license.html"
+  "python2.7-urllib3-1.16": {
+    "MIT License": "http://spdx.org/licenses/MIT"
   }, 
-  "python2.7-trollius-1.0.4": {
-    "APSL-2.0": "http://spdx.org/licenses/APSL-2.0"
+  "python2.7-venusian-1.0": {
+    "Repoze License": "http://www.repoze.org/LICENSE.txt"
   }, 
   "python2.7-waitress-0.8.9": {
-    "ZPL-2.1": "http://spdx.org/licenses/ZPL-2.1"
+    "Zope Public License 2.1": "http://spdx.org/licenses/ZPL-2.1"
   }, 
   "python2.7-zope.cachedescriptors-4.0.0": {
-    "ZPL-2.1": "http://spdx.org/licenses/ZPL-2.1"
+    "Zope Public License 2.1": "http://spdx.org/licenses/ZPL-2.1"
+  }, 
+  "python2.7-zope.deprecation-4.1.2": {
+    "Zope Public License 2.1": "http://spdx.org/licenses/ZPL-2.1"
+  }, 
+  "python2.7-zope.interface-4.1.3": {
+    "Zope Public License 2.1": "http://spdx.org/licenses/ZPL-2.1"
   }
-}
+}
  No newline at end of file

rhodecode/config/middleware.py

@@ -1,316 +1,387 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 Pylons middleware initialization
 """
 import logging
 
 from paste.registry import RegistryManager
 from paste.gzipper import make_gzip_middleware
-from pylons.middleware import ErrorHandler, StatusCodeRedirect
 from pylons.wsgiapp import PylonsApp
 from pyramid.authorization import ACLAuthorizationPolicy
 from pyramid.config import Configurator
 from pyramid.static import static_view
 from pyramid.settings import asbool, aslist
 from pyramid.wsgi import wsgiapp
+from pyramid.httpexceptions import HTTPError, HTTPInternalServerError
+import pyramid.httpexceptions as httpexceptions
+from pyramid.renderers import render_to_response, render
 from routes.middleware import RoutesMiddleware
 import routes.util
 
 import rhodecode
 from rhodecode.config import patches
 from rhodecode.config.environment import (
     load_environment, load_pyramid_environment)
 from rhodecode.lib.middleware import csrf
 from rhodecode.lib.middleware.appenlight import wrap_in_appenlight_if_enabled
 from rhodecode.lib.middleware.disable_vcs import DisableVCSPagesWrapper
 from rhodecode.lib.middleware.https_fixup import HttpsFixup
 from rhodecode.lib.middleware.vcs import VCSMiddleware
 from rhodecode.lib.plugins.utils import register_rhodecode_plugin
 
 
 log = logging.getLogger(__name__)
 
 
 def make_app(global_conf, full_stack=True, static_files=True, **app_conf):
     """Create a Pylons WSGI application and return it
 
     ``global_conf``
         The inherited configuration for this application. Normally from
         the [DEFAULT] section of the Paste ini file.
 
     ``full_stack``
         Whether or not this application provides a full WSGI stack (by
         default, meaning it handles its own exceptions and errors).
         Disable full_stack when this application is "managed" by
         another WSGI middleware.
 
     ``app_conf``
         The application's local configuration. Normally specified in
         the [app:<name>] section of the Paste ini file (where <name>
         defaults to main).
 
     """
     # Apply compatibility patches
     patches.kombu_1_5_1_python_2_7_11()
     patches.inspect_getargspec()
 
     # Configure the Pylons environment
     config = load_environment(global_conf, app_conf)
 
     # The Pylons WSGI app
     app = PylonsApp(config=config)
     if rhodecode.is_test:
         app = csrf.CSRFDetector(app)
 
     expected_origin = config.get('expected_origin')
     if expected_origin:
         # The API can be accessed from other Origins.
         app = csrf.OriginChecker(app, expected_origin,
                                  skip_urls=[routes.util.url_for('api')])
 
-    # Add RoutesMiddleware. Currently we have two instances in the stack. This
-    # is the lower one to make the StatusCodeRedirect middleware happy.
-    # TODO: johbo: This is not optimal, search for a better solution.
-    app = RoutesMiddleware(app, config['routes.map'])
-
-    # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)
-    if asbool(config['pdebug']):
-        from rhodecode.lib.profiler import ProfilingMiddleware
-        app = ProfilingMiddleware(app)
-
-    # Protect from VCS Server error related pages when server is not available
-    vcs_server_enabled = asbool(config.get('vcs.server.enable', 'true'))
-    if not vcs_server_enabled:
-        app = DisableVCSPagesWrapper(app)
 
     if asbool(full_stack):
 
         # Appenlight monitoring and error handler
         app, appenlight_client = wrap_in_appenlight_if_enabled(app, config)
 
-        # Handle Python exceptions
-        app = ErrorHandler(app, global_conf, **config['pylons.errorware'])
-
         # we want our low level middleware to get to the request ASAP. We don't
         # need any pylons stack middleware in them
         app = VCSMiddleware(app, config, appenlight_client)
-        # Display error documents for 401, 403, 404 status codes (and
-        # 500 when debug is disabled)
-        if asbool(config['debug']):
-            app = StatusCodeRedirect(app)
-        else:
-            app = StatusCodeRedirect(app, [400, 401, 403, 404, 500])
 
     # Establish the Registry for this application
     app = RegistryManager(app)
 
     app.config = config
 
     return app
 
 
 def make_pyramid_app(global_config, **settings):
     """
     Constructs the WSGI application based on Pyramid and wraps the Pylons based
     application.
 
     Specials:
 
     * We migrate from Pylons to Pyramid. While doing this, we keep both
       frameworks functional. This involves moving some WSGI middlewares around
       and providing access to some data internals, so that the old code is
       still functional.
 
     * The application can also be integrated like a plugin via the call to
       `includeme`. This is accompanied with the other utility functions which
       are called. Changing this should be done with great care to not break
       cases when these fragments are assembled from another place.
 
     """
     # The edition string should be available in pylons too, so we add it here
     # before copying the settings.
     settings.setdefault('rhodecode.edition', 'Community Edition')
 
     # As long as our Pylons application does expect "unprepared" settings, make
     # sure that we keep an unmodified copy. This avoids unintentional change of
     # behavior in the old application.
     settings_pylons = settings.copy()
 
     sanitize_settings_and_apply_defaults(settings)
     config = Configurator(settings=settings)
     add_pylons_compat_data(config.registry, global_config, settings_pylons)
 
     load_pyramid_environment(global_config, settings)
 
     includeme(config)
     includeme_last(config)
     pyramid_app = config.make_wsgi_app()
     pyramid_app = wrap_app_in_wsgi_middlewares(pyramid_app, config)
     return pyramid_app
 
 
 def add_pylons_compat_data(registry, global_config, settings):
     """
     Attach data to the registry to support the Pylons integration.
     """
     registry._pylons_compat_global_config = global_config
     registry._pylons_compat_settings = settings
 
 
+def webob_to_pyramid_http_response(webob_response):
+    ResponseClass = httpexceptions.status_map[webob_response.status_int]
+    pyramid_response = ResponseClass(webob_response.status)
+    pyramid_response.status = webob_response.status
+    pyramid_response.headers.update(webob_response.headers)
+    if pyramid_response.headers['content-type'] == 'text/html':
+        pyramid_response.headers['content-type'] = 'text/html; charset=UTF-8'
+    return pyramid_response
+
+
+def error_handler(exception, request):
+    # TODO: dan: replace the old pylons error controller with this
+    from rhodecode.model.settings import SettingsModel
+    from rhodecode.lib.utils2 import AttributeDict
+
+    try:
+        rc_config = SettingsModel().get_all_settings()
+    except Exception:
+        log.exception('failed to fetch settings')
+        rc_config = {}
+
+    base_response = HTTPInternalServerError()
+    # prefer original exception for the response since it may have headers set
+    if isinstance(exception, HTTPError):
+        base_response = exception
+
+    c = AttributeDict()
+    c.error_message = base_response.status
+    c.error_explanation = base_response.explanation or str(base_response)
+    c.visual = AttributeDict()
+
+    c.visual.rhodecode_support_url = (
+        request.registry.settings.get('rhodecode_support_url') or
+        request.route_url('rhodecode_support')
+    )
+    c.redirect_time = 0
+    c.rhodecode_name = rc_config.get('rhodecode_title', '')
+    if not c.rhodecode_name:
+        c.rhodecode_name = 'Rhodecode'
+
+    response = render_to_response(
+        '/errors/error_document.html', {'c': c}, request=request,
+        response=base_response)
+
+    return response
+
+
 def includeme(config):
     settings = config.registry.settings
 
+    if asbool(settings.get('appenlight', 'false')):
+        config.include('appenlight_client.ext.pyramid_tween')
+
     # Includes which are required. The application would fail without them.
     config.include('pyramid_mako')
     config.include('pyramid_beaker')
+    config.include('rhodecode.admin')
     config.include('rhodecode.authentication')
     config.include('rhodecode.login')
     config.include('rhodecode.tweens')
     config.include('rhodecode.api')
+    config.add_route(
+        'rhodecode_support', 'https://rhodecode.com/help/', static=True)
 
     # Set the authorization policy.
     authz_policy = ACLAuthorizationPolicy()
     config.set_authorization_policy(authz_policy)
 
     # Set the default renderer for HTML templates to mako.
     config.add_mako_renderer('.html')
 
     # plugin information
     config.registry.rhodecode_plugins = {}
 
     config.add_directive(
         'register_rhodecode_plugin', register_rhodecode_plugin)
     # include RhodeCode plugins
     includes = aslist(settings.get('rhodecode.includes', []))
     for inc in includes:
         config.include(inc)
 
+    pylons_app = make_app(
+        config.registry._pylons_compat_global_config,
+        **config.registry._pylons_compat_settings)
+    config.registry._pylons_compat_config = pylons_app.config
+
+    pylons_app_as_view = wsgiapp(pylons_app)
+
+    # Protect from VCS Server error related pages when server is not available
+    vcs_server_enabled = asbool(settings.get('vcs.server.enable', 'true'))
+    if not vcs_server_enabled:
+        pylons_app_as_view = DisableVCSPagesWrapper(pylons_app_as_view)
+
+
+    def pylons_app_with_error_handler(context, request):
+        """
+        Handle exceptions from rc pylons app:
+
+        - old webob type exceptions get converted to pyramid exceptions
+        - pyramid exceptions are passed to the error handler view
+        """
+        try:
+            response = pylons_app_as_view(context, request)
+            if 400 <= response.status_int <= 599: # webob type error responses
+                return error_handler(
+                    webob_to_pyramid_http_response(response), request)
+        except HTTPError as e: # pyramid type exceptions
+            return error_handler(e, request)
+        except Exception:
+            if settings.get('debugtoolbar.enabled', False):
+                raise
+            return error_handler(HTTPInternalServerError(), request)
+        return response
+
     # This is the glue which allows us to migrate in chunks. By registering the
     # pylons based application as the "Not Found" view in Pyramid, we will
     # fallback to the old application each time the new one does not yet know
     # how to handle a request.
-    pylons_app = make_app(
-        config.registry._pylons_compat_global_config,
-        **config.registry._pylons_compat_settings)
-    config.registry._pylons_compat_config = pylons_app.config
-    pylons_app_as_view = wsgiapp(pylons_app)
-    config.add_notfound_view(pylons_app_as_view)
+    config.add_notfound_view(pylons_app_with_error_handler)
+
+    if settings.get('debugtoolbar.enabled', False):
+        # if toolbar, then only http type exceptions get caught and rendered
+        ExcClass = HTTPError
+    else:
+        # if no toolbar, then any exception gets caught and rendered
+        ExcClass = Exception
+    config.add_view(error_handler, context=ExcClass)
 
 
 def includeme_last(config):
     """
     The static file catchall needs to be last in the view configuration.
     """
     settings = config.registry.settings
 
     # Note: johbo: I would prefer to register a prefix for static files at some
     # point, e.g. move them under '_static/'. This would fully avoid that we
     # can have name clashes with a repository name. Imaging someone calling his
     # repo "css" ;-) Also having an external web server to serve out the static
     # files seems to be easier to set up if they have a common prefix.
     #
     # Example: config.add_static_view('_static', path='rhodecode:public')
     #
     # It might be an option to register both paths for a while and then migrate
     # over to the new location.
 
     # Serving static files with a catchall.
     if settings['static_files']:
         config.add_route('catchall_static', '/*subpath')
         config.add_view(
             static_view('rhodecode:public'), route_name='catchall_static')
 
 
 def wrap_app_in_wsgi_middlewares(pyramid_app, config):
     """
     Apply outer WSGI middlewares around the application.
 
     Part of this has been moved up from the Pylons layer, so that the
     data is also available if old Pylons code is hit through an already ported
     view.
     """
     settings = config.registry.settings
 
     # enable https redirects based on HTTP_X_URL_SCHEME set by proxy
     pyramid_app = HttpsFixup(pyramid_app, settings)
 
-    # Add RoutesMiddleware. Currently we have two instances in the stack. This
-    # is the upper one to support the pylons compatibility tween during
+    # Add RoutesMiddleware to support the pylons compatibility tween during
 
     # migration to pyramid.
     pyramid_app = RoutesMiddleware(
         pyramid_app, config.registry._pylons_compat_config['routes.map'])
 
+    if asbool(settings.get('appenlight', 'false')):
+        pyramid_app, _ = wrap_in_appenlight_if_enabled(
+            pyramid_app, config.registry._pylons_compat_config)
+
     # TODO: johbo: Don't really see why we enable the gzip middleware when
     # serving static files, might be something that should have its own setting
     # as well?
     if settings['static_files']:
         pyramid_app = make_gzip_middleware(
             pyramid_app, settings, compress_level=1)
 
     return pyramid_app
 
 
 def sanitize_settings_and_apply_defaults(settings):
     """
     Applies settings defaults and does all type conversion.
 
     We would move all settings parsing and preparation into this place, so that
     we have only one place left which deals with this part. The remaining parts
     of the application would start to rely fully on well prepared settings.
 
     This piece would later be split up per topic to avoid a big fat monster
     function.
     """
 
     # Pyramid's mako renderer has to search in the templates folder so that the
     # old templates still work. Ported and new templates are expected to use
     # real asset specifications for the includes.
     mako_directories = settings.setdefault('mako.directories', [
         # Base templates of the original Pylons application
         'rhodecode:templates',
     ])
     log.debug(
         "Using the following Mako template directories: %s",
         mako_directories)
 
     # Default includes, possible to change as a user
     pyramid_includes = settings.setdefault('pyramid.includes', [
         'rhodecode.lib.middleware.request_wrapper',
     ])
     log.debug(
         "Using the following pyramid.includes: %s",
         pyramid_includes)
 
     # TODO: johbo: Re-think this, usually the call to config.include
     # should allow to pass in a prefix.
     settings.setdefault('rhodecode.api.url', '/_admin/api')
 
     _bool_setting(settings, 'vcs.server.enable', 'true')
     _bool_setting(settings, 'static_files', 'true')
     _bool_setting(settings, 'is_test', 'false')
 
     return settings
 
 
 def _bool_setting(settings, name, default):
     settings[name] = asbool(settings.get(name, default))

rhodecode/config/routing.py

@@ -1,1149 +1,1141 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 Routes configuration
 
 The more specific and detailed routes should be defined first so they
 may take precedent over the more generic routes. For more information
 refer to the routes manual at http://routes.groovie.org/docs/
 
 IMPORTANT: if you change any routing here, make sure to take a look at lib/base.py
 and _route_name variable which uses some of stored naming here to do redirects.
 """
 import os
 import re
 from routes import Mapper
 
 from rhodecode.config import routing_links
 
 # prefix for non repository related links needs to be prefixed with `/`
 ADMIN_PREFIX = '/_admin'
 
 # Default requirements for URL parts
 URL_NAME_REQUIREMENTS = {
     # group name can have a slash in them, but they must not end with a slash
     'group_name': r'.*?[^/]',
     # repo names can have a slash in them, but they must not end with a slash
     'repo_name': r'.*?[^/]',
     # file path eats up everything at the end
     'f_path': r'.*',
     # reference types
     'source_ref_type': '(branch|book|tag|rev|\%\(source_ref_type\)s)',
     'target_ref_type': '(branch|book|tag|rev|\%\(target_ref_type\)s)',
 }
 
 
 class JSRoutesMapper(Mapper):
     """
     Wrapper for routes.Mapper to make pyroutes compatible url definitions
     """
     _named_route_regex = re.compile(r'^[a-z-_0-9A-Z]+$')
     _argument_prog = re.compile('\{(.*?)\}|:\((.*)\)')
     def __init__(self, *args, **kw):
         super(JSRoutesMapper, self).__init__(*args, **kw)
         self._jsroutes = []
 
     def connect(self, *args, **kw):
         """
         Wrapper for connect to take an extra argument jsroute=True
 
         :param jsroute: boolean, if True will add the route to the pyroutes list
         """
         if kw.pop('jsroute', False):
             if not self._named_route_regex.match(args[0]):
                 raise Exception('only named routes can be added to pyroutes')
             self._jsroutes.append(args[0])
 
         super(JSRoutesMapper, self).connect(*args, **kw)
 
     def _extract_route_information(self, route):
         """
         Convert a route into tuple(name, path, args), eg:
             ('user_profile', '/profile/%(username)s', ['username'])
         """
         routepath = route.routepath
         def replace(matchobj):
             if matchobj.group(1):
                 return "%%(%s)s" % matchobj.group(1).split(':')[0]
             else:
                 return "%%(%s)s" % matchobj.group(2)
 
         routepath = self._argument_prog.sub(replace, routepath)
         return (
             route.name,
             routepath,
             [(arg[0].split(':')[0] if arg[0] != '' else arg[1])
               for arg in self._argument_prog.findall(route.routepath)]
         )
 
     def jsroutes(self):
         """
         Return a list of pyroutes.js compatible routes
         """
         for route_name in self._jsroutes:
             yield self._extract_route_information(self._routenames[route_name])
 
 
 def make_map(config):
     """Create, configure and return the routes Mapper"""
     rmap = JSRoutesMapper(directory=config['pylons.paths']['controllers'],
                   always_scan=config['debug'])
     rmap.minimization = False
     rmap.explicit = False
 
     from rhodecode.lib.utils2 import str2bool
     from rhodecode.model import repo, repo_group
 
     def check_repo(environ, match_dict):
         """
         check for valid repository for proper 404 handling
 
         :param environ:
         :param match_dict:
         """
         repo_name = match_dict.get('repo_name')
 
         if match_dict.get('f_path'):
             # fix for multiple initial slashes that causes errors
             match_dict['f_path'] = match_dict['f_path'].lstrip('/')
         repo_model = repo.RepoModel()
         by_name_match = repo_model.get_by_repo_name(repo_name)
         # if we match quickly from database, short circuit the operation,
         # and validate repo based on the type.
         if by_name_match:
             return True
 
         by_id_match = repo_model.get_repo_by_id(repo_name)
         if by_id_match:
             repo_name = by_id_match.repo_name
             match_dict['repo_name'] = repo_name
             return True
 
         return False
 
     def check_group(environ, match_dict):
         """
         check for valid repository group path for proper 404 handling
 
         :param environ:
         :param match_dict:
         """
         repo_group_name = match_dict.get('group_name')
         repo_group_model = repo_group.RepoGroupModel()
         by_name_match = repo_group_model.get_by_group_name(repo_group_name)
         if by_name_match:
             return True
 
         return False
 
     def check_user_group(environ, match_dict):
         """
         check for valid user group for proper 404 handling
 
         :param environ:
         :param match_dict:
         """
         return True
 
     def check_int(environ, match_dict):
         return match_dict.get('id').isdigit()
 
-    # The ErrorController route (handles 404/500 error pages); it should
-    # likely stay at the top, ensuring it can always be resolved
-    rmap.connect('/error/{action}', controller='error')
-    rmap.connect('/error/{action}/{id}', controller='error')
 
     #==========================================================================
     # CUSTOM ROUTES HERE
     #==========================================================================
 
     # MAIN PAGE
     rmap.connect('home', '/', controller='home', action='index', jsroute=True)
     rmap.connect('goto_switcher_data', '/_goto_data', controller='home',
                  action='goto_switcher_data')
     rmap.connect('repo_list_data', '/_repos', controller='home',
                  action='repo_list_data')
 
     rmap.connect('user_autocomplete_data', '/_users', controller='home',
                  action='user_autocomplete_data', jsroute=True)
     rmap.connect('user_group_autocomplete_data', '/_user_groups', controller='home',
                  action='user_group_autocomplete_data')
 
     rmap.connect(
         'user_profile', '/_profiles/{username}', controller='users',
         action='user_profile')
 
     # TODO: johbo: Static links, to be replaced by our redirection mechanism
     rmap.connect('rst_help',
                  'http://docutils.sourceforge.net/docs/user/rst/quickref.html',
                  _static=True)
     rmap.connect('markdown_help',
                  'http://daringfireball.net/projects/markdown/syntax',
                  _static=True)
     rmap.connect('rhodecode_official', 'https://rhodecode.com', _static=True)
     rmap.connect('rhodecode_support', 'https://rhodecode.com/help/', _static=True)
     rmap.connect('rhodecode_translations', 'https://rhodecode.com/translate/enterprise', _static=True)
     # TODO: anderson - making this a static link since redirect won't play
     # nice with POST requests
     rmap.connect('enterprise_license_convert_from_old',
                  'https://rhodecode.com/u/license-upgrade',
                  _static=True)
 
     routing_links.connect_redirection_links(rmap)
 
     rmap.connect('ping', '%s/ping' % (ADMIN_PREFIX,), controller='home', action='ping')
     rmap.connect('error_test', '%s/error_test' % (ADMIN_PREFIX,), controller='home', action='error_test')
 
     # ADMIN REPOSITORY ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/repos') as m:
         m.connect('repos', '/repos',
                   action='create', conditions={'method': ['POST']})
         m.connect('repos', '/repos',
                   action='index', conditions={'method': ['GET']})
         m.connect('new_repo', '/create_repository', jsroute=True,
                   action='create_repository', conditions={'method': ['GET']})
         m.connect('/repos/{repo_name}',
                   action='update', conditions={'method': ['PUT'],
                                                'function': check_repo},
                   requirements=URL_NAME_REQUIREMENTS)
         m.connect('delete_repo', '/repos/{repo_name}',
                   action='delete', conditions={'method': ['DELETE']},
                   requirements=URL_NAME_REQUIREMENTS)
         m.connect('repo', '/repos/{repo_name}',
                   action='show', conditions={'method': ['GET'],
                                              'function': check_repo},
                   requirements=URL_NAME_REQUIREMENTS)
 
     # ADMIN REPOSITORY GROUPS ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/repo_groups') as m:
         m.connect('repo_groups', '/repo_groups',
                   action='create', conditions={'method': ['POST']})
         m.connect('repo_groups', '/repo_groups',
                   action='index', conditions={'method': ['GET']})
         m.connect('new_repo_group', '/repo_groups/new',
                   action='new', conditions={'method': ['GET']})
         m.connect('update_repo_group', '/repo_groups/{group_name}',
                   action='update', conditions={'method': ['PUT'],
                                                'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
 
         # EXTRAS REPO GROUP ROUTES
         m.connect('edit_repo_group', '/repo_groups/{group_name}/edit',
                   action='edit',
                   conditions={'method': ['GET'], 'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
         m.connect('edit_repo_group', '/repo_groups/{group_name}/edit',
                   action='edit',
                   conditions={'method': ['PUT'], 'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
 
         m.connect('edit_repo_group_advanced', '/repo_groups/{group_name}/edit/advanced',
                   action='edit_repo_group_advanced',
                   conditions={'method': ['GET'], 'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
         m.connect('edit_repo_group_advanced', '/repo_groups/{group_name}/edit/advanced',
                   action='edit_repo_group_advanced',
                   conditions={'method': ['PUT'], 'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
 
         m.connect('edit_repo_group_perms', '/repo_groups/{group_name}/edit/permissions',
                   action='edit_repo_group_perms',
                   conditions={'method': ['GET'], 'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
         m.connect('edit_repo_group_perms', '/repo_groups/{group_name}/edit/permissions',
                   action='update_perms',
                   conditions={'method': ['PUT'], 'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
 
         m.connect('delete_repo_group', '/repo_groups/{group_name}',
                   action='delete', conditions={'method': ['DELETE'],
                                                'function': check_group},
                   requirements=URL_NAME_REQUIREMENTS)
 
     # ADMIN USER ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/users') as m:
         m.connect('users', '/users',
                   action='create', conditions={'method': ['POST']})
         m.connect('users', '/users',
                   action='index', conditions={'method': ['GET']})
         m.connect('new_user', '/users/new',
                   action='new', conditions={'method': ['GET']})
         m.connect('update_user', '/users/{user_id}',
                   action='update', conditions={'method': ['PUT']})
         m.connect('delete_user', '/users/{user_id}',
                   action='delete', conditions={'method': ['DELETE']})
         m.connect('edit_user', '/users/{user_id}/edit',
                   action='edit', conditions={'method': ['GET']})
         m.connect('user', '/users/{user_id}',
                   action='show', conditions={'method': ['GET']})
         m.connect('force_password_reset_user', '/users/{user_id}/password_reset',
                   action='reset_password', conditions={'method': ['POST']})
         m.connect('create_personal_repo_group', '/users/{user_id}/create_repo_group',
                   action='create_personal_repo_group', conditions={'method': ['POST']})
 
         # EXTRAS USER ROUTES
         m.connect('edit_user_advanced', '/users/{user_id}/edit/advanced',
                   action='edit_advanced', conditions={'method': ['GET']})
         m.connect('edit_user_advanced', '/users/{user_id}/edit/advanced',
                   action='update_advanced', conditions={'method': ['PUT']})
 
         m.connect('edit_user_auth_tokens', '/users/{user_id}/edit/auth_tokens',
                   action='edit_auth_tokens', conditions={'method': ['GET']})
         m.connect('edit_user_auth_tokens', '/users/{user_id}/edit/auth_tokens',
                   action='add_auth_token', conditions={'method': ['PUT']})
         m.connect('edit_user_auth_tokens', '/users/{user_id}/edit/auth_tokens',
                   action='delete_auth_token', conditions={'method': ['DELETE']})
 
         m.connect('edit_user_global_perms', '/users/{user_id}/edit/global_permissions',
                   action='edit_global_perms', conditions={'method': ['GET']})
         m.connect('edit_user_global_perms', '/users/{user_id}/edit/global_permissions',
                   action='update_global_perms', conditions={'method': ['PUT']})
 
         m.connect('edit_user_perms_summary', '/users/{user_id}/edit/permissions_summary',
                   action='edit_perms_summary', conditions={'method': ['GET']})
 
         m.connect('edit_user_emails', '/users/{user_id}/edit/emails',
                   action='edit_emails', conditions={'method': ['GET']})
         m.connect('edit_user_emails', '/users/{user_id}/edit/emails',
                   action='add_email', conditions={'method': ['PUT']})
         m.connect('edit_user_emails', '/users/{user_id}/edit/emails',
                   action='delete_email', conditions={'method': ['DELETE']})
 
         m.connect('edit_user_ips', '/users/{user_id}/edit/ips',
                   action='edit_ips', conditions={'method': ['GET']})
         m.connect('edit_user_ips', '/users/{user_id}/edit/ips',
                   action='add_ip', conditions={'method': ['PUT']})
         m.connect('edit_user_ips', '/users/{user_id}/edit/ips',
                   action='delete_ip', conditions={'method': ['DELETE']})
 
     # ADMIN USER GROUPS REST ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/user_groups') as m:
         m.connect('users_groups', '/user_groups',
                   action='create', conditions={'method': ['POST']})
         m.connect('users_groups', '/user_groups',
                   action='index', conditions={'method': ['GET']})
         m.connect('new_users_group', '/user_groups/new',
                   action='new', conditions={'method': ['GET']})
         m.connect('update_users_group', '/user_groups/{user_group_id}',
                   action='update', conditions={'method': ['PUT']})
         m.connect('delete_users_group', '/user_groups/{user_group_id}',
                   action='delete', conditions={'method': ['DELETE']})
         m.connect('edit_users_group', '/user_groups/{user_group_id}/edit',
                   action='edit', conditions={'method': ['GET']},
                   function=check_user_group)
 
         # EXTRAS USER GROUP ROUTES
         m.connect('edit_user_group_global_perms',
                   '/user_groups/{user_group_id}/edit/global_permissions',
                   action='edit_global_perms', conditions={'method': ['GET']})
         m.connect('edit_user_group_global_perms',
                   '/user_groups/{user_group_id}/edit/global_permissions',
                   action='update_global_perms', conditions={'method': ['PUT']})
         m.connect('edit_user_group_perms_summary',
                   '/user_groups/{user_group_id}/edit/permissions_summary',
                   action='edit_perms_summary', conditions={'method': ['GET']})
 
         m.connect('edit_user_group_perms',
                   '/user_groups/{user_group_id}/edit/permissions',
                   action='edit_perms', conditions={'method': ['GET']})
         m.connect('edit_user_group_perms',
                   '/user_groups/{user_group_id}/edit/permissions',
                   action='update_perms', conditions={'method': ['PUT']})
 
         m.connect('edit_user_group_advanced',
                   '/user_groups/{user_group_id}/edit/advanced',
                   action='edit_advanced', conditions={'method': ['GET']})
 
         m.connect('edit_user_group_members',
                   '/user_groups/{user_group_id}/edit/members', jsroute=True,
                   action='edit_members', conditions={'method': ['GET']})
 
     # ADMIN PERMISSIONS ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/permissions') as m:
         m.connect('admin_permissions_application', '/permissions/application',
                   action='permission_application_update', conditions={'method': ['POST']})
         m.connect('admin_permissions_application', '/permissions/application',
                   action='permission_application', conditions={'method': ['GET']})
 
         m.connect('admin_permissions_global', '/permissions/global',
                   action='permission_global_update', conditions={'method': ['POST']})
         m.connect('admin_permissions_global', '/permissions/global',
                   action='permission_global', conditions={'method': ['GET']})
 
         m.connect('admin_permissions_object', '/permissions/object',
                   action='permission_objects_update', conditions={'method': ['POST']})
         m.connect('admin_permissions_object', '/permissions/object',
                   action='permission_objects', conditions={'method': ['GET']})
 
         m.connect('admin_permissions_ips', '/permissions/ips',
                   action='permission_ips', conditions={'method': ['POST']})
         m.connect('admin_permissions_ips', '/permissions/ips',
                   action='permission_ips', conditions={'method': ['GET']})
 
         m.connect('admin_permissions_overview', '/permissions/overview',
                   action='permission_perms', conditions={'method': ['GET']})
 
     # ADMIN DEFAULTS REST ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/defaults') as m:
         m.connect('admin_defaults_repositories', '/defaults/repositories',
                   action='update_repository_defaults', conditions={'method': ['POST']})
         m.connect('admin_defaults_repositories', '/defaults/repositories',
                   action='index', conditions={'method': ['GET']})
 
     # ADMIN DEBUG STYLE ROUTES
     if str2bool(config.get('debug_style')):
         with rmap.submapper(path_prefix=ADMIN_PREFIX + '/debug_style',
                             controller='debug_style') as m:
             m.connect('debug_style_home', '',
                       action='index', conditions={'method': ['GET']})
             m.connect('debug_style_template', '/t/{t_path}',
                       action='template', conditions={'method': ['GET']})
 
     # ADMIN SETTINGS ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/settings') as m:
 
         # default
         m.connect('admin_settings', '/settings',
                   action='settings_global_update',
                   conditions={'method': ['POST']})
         m.connect('admin_settings', '/settings',
                   action='settings_global', conditions={'method': ['GET']})
 
         m.connect('admin_settings_vcs', '/settings/vcs',
                   action='settings_vcs_update',
                   conditions={'method': ['POST']})
         m.connect('admin_settings_vcs', '/settings/vcs',
                   action='settings_vcs',
                   conditions={'method': ['GET']})
         m.connect('admin_settings_vcs', '/settings/vcs',
                   action='delete_svn_pattern',
                   conditions={'method': ['DELETE']})
 
         m.connect('admin_settings_mapping', '/settings/mapping',
                   action='settings_mapping_update',
                   conditions={'method': ['POST']})
         m.connect('admin_settings_mapping', '/settings/mapping',
                   action='settings_mapping', conditions={'method': ['GET']})
 
         m.connect('admin_settings_global', '/settings/global',
                   action='settings_global_update',
                   conditions={'method': ['POST']})
         m.connect('admin_settings_global', '/settings/global',
                   action='settings_global', conditions={'method': ['GET']})
 
         m.connect('admin_settings_visual', '/settings/visual',
                   action='settings_visual_update',
                   conditions={'method': ['POST']})
         m.connect('admin_settings_visual', '/settings/visual',
                   action='settings_visual', conditions={'method': ['GET']})
 
         m.connect('admin_settings_issuetracker',
                   '/settings/issue-tracker', action='settings_issuetracker',
                   conditions={'method': ['GET']})
         m.connect('admin_settings_issuetracker_save',
                   '/settings/issue-tracker/save',
                   action='settings_issuetracker_save',
                   conditions={'method': ['POST']})
         m.connect('admin_issuetracker_test', '/settings/issue-tracker/test',
                   action='settings_issuetracker_test',
                   conditions={'method': ['POST']})
         m.connect('admin_issuetracker_delete',
                   '/settings/issue-tracker/delete',
                   action='settings_issuetracker_delete',
                   conditions={'method': ['DELETE']})
 
         m.connect('admin_settings_email', '/settings/email',
                   action='settings_email_update',
                   conditions={'method': ['POST']})
         m.connect('admin_settings_email', '/settings/email',
                   action='settings_email', conditions={'method': ['GET']})
 
         m.connect('admin_settings_hooks', '/settings/hooks',
                   action='settings_hooks_update',
                   conditions={'method': ['POST', 'DELETE']})
         m.connect('admin_settings_hooks', '/settings/hooks',
                   action='settings_hooks', conditions={'method': ['GET']})
 
         m.connect('admin_settings_search', '/settings/search',
                   action='settings_search', conditions={'method': ['GET']})
 
         m.connect('admin_settings_system', '/settings/system',
                   action='settings_system', conditions={'method': ['GET']})
 
         m.connect('admin_settings_system_update', '/settings/system/updates',
                   action='settings_system_update', conditions={'method': ['GET']})
 
         m.connect('admin_settings_supervisor', '/settings/supervisor',
                   action='settings_supervisor', conditions={'method': ['GET']})
         m.connect('admin_settings_supervisor_log', '/settings/supervisor/{procid}/log',
                   action='settings_supervisor_log', conditions={'method': ['GET']})
 
         m.connect('admin_settings_labs', '/settings/labs',
                   action='settings_labs_update',
                   conditions={'method': ['POST']})
         m.connect('admin_settings_labs', '/settings/labs',
                   action='settings_labs', conditions={'method': ['GET']})
 
-        m.connect('admin_settings_open_source', '/settings/open_source',
-                  action='settings_open_source',
-                  conditions={'method': ['GET']})
-
     # ADMIN MY ACCOUNT
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/my_account') as m:
 
         m.connect('my_account', '/my_account',
                   action='my_account', conditions={'method': ['GET']})
         m.connect('my_account_edit', '/my_account/edit',
                   action='my_account_edit', conditions={'method': ['GET']})
         m.connect('my_account', '/my_account',
                   action='my_account_update', conditions={'method': ['POST']})
 
         m.connect('my_account_password', '/my_account/password',
                   action='my_account_password', conditions={'method': ['GET']})
         m.connect('my_account_password', '/my_account/password',
                   action='my_account_password_update', conditions={'method': ['POST']})
 
         m.connect('my_account_repos', '/my_account/repos',
                   action='my_account_repos', conditions={'method': ['GET']})
 
         m.connect('my_account_watched', '/my_account/watched',
                   action='my_account_watched', conditions={'method': ['GET']})
 
         m.connect('my_account_pullrequests', '/my_account/pull_requests',
                   action='my_account_pullrequests', conditions={'method': ['GET']})
 
         m.connect('my_account_perms', '/my_account/perms',
                   action='my_account_perms', conditions={'method': ['GET']})
 
         m.connect('my_account_emails', '/my_account/emails',
                   action='my_account_emails', conditions={'method': ['GET']})
         m.connect('my_account_emails', '/my_account/emails',
                   action='my_account_emails_add', conditions={'method': ['POST']})
         m.connect('my_account_emails', '/my_account/emails',
                   action='my_account_emails_delete', conditions={'method': ['DELETE']})
 
         m.connect('my_account_auth_tokens', '/my_account/auth_tokens',
                   action='my_account_auth_tokens', conditions={'method': ['GET']})
         m.connect('my_account_auth_tokens', '/my_account/auth_tokens',
                   action='my_account_auth_tokens_add', conditions={'method': ['POST']})
         m.connect('my_account_auth_tokens', '/my_account/auth_tokens',
                   action='my_account_auth_tokens_delete', conditions={'method': ['DELETE']})
 
     # NOTIFICATION REST ROUTES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/notifications') as m:
         m.connect('notifications', '/notifications',
                   action='index', conditions={'method': ['GET']})
         m.connect('notifications_mark_all_read', '/notifications/mark_all_read',
                   action='mark_all_read', conditions={'method': ['POST']})
 
         m.connect('/notifications/{notification_id}',
                   action='update', conditions={'method': ['PUT']})
         m.connect('/notifications/{notification_id}',
                   action='delete', conditions={'method': ['DELETE']})
         m.connect('notification', '/notifications/{notification_id}',
                   action='show', conditions={'method': ['GET']})
 
     # ADMIN GIST
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/gists') as m:
         m.connect('gists', '/gists',
                   action='create', conditions={'method': ['POST']})
         m.connect('gists', '/gists', jsroute=True,
                   action='index', conditions={'method': ['GET']})
         m.connect('new_gist', '/gists/new', jsroute=True,
                   action='new', conditions={'method': ['GET']})
 
         m.connect('/gists/{gist_id}',
                   action='delete', conditions={'method': ['DELETE']})
         m.connect('edit_gist', '/gists/{gist_id}/edit',
                   action='edit_form', conditions={'method': ['GET']})
         m.connect('edit_gist', '/gists/{gist_id}/edit',
                   action='edit', conditions={'method': ['POST']})
         m.connect(
             'edit_gist_check_revision', '/gists/{gist_id}/edit/check_revision',
             action='check_revision', conditions={'method': ['GET']})
 
         m.connect('gist', '/gists/{gist_id}',
                   action='show', conditions={'method': ['GET']})
         m.connect('gist_rev', '/gists/{gist_id}/{revision}',
                   revision='tip',
                   action='show', conditions={'method': ['GET']})
         m.connect('formatted_gist', '/gists/{gist_id}/{revision}/{format}',
                   revision='tip',
                   action='show', conditions={'method': ['GET']})
         m.connect('formatted_gist_file', '/gists/{gist_id}/{revision}/{format}/{f_path}',
                   revision='tip',
                   action='show', conditions={'method': ['GET']},
                   requirements=URL_NAME_REQUIREMENTS)
 
     # ADMIN MAIN PAGES
     with rmap.submapper(path_prefix=ADMIN_PREFIX,
                         controller='admin/admin') as m:
         m.connect('admin_home', '', action='index')
         m.connect('admin_add_repo', '/add_repo/{new_repo:[a-z0-9\. _-]*}',
                   action='add_repo')
         m.connect(
             'pull_requests_global_0', '/pull_requests/{pull_request_id:[0-9]+}',
             action='pull_requests')
         m.connect(
             'pull_requests_global', '/pull-requests/{pull_request_id:[0-9]+}',
             action='pull_requests')
 
 
     # USER JOURNAL
     rmap.connect('journal', '%s/journal' % (ADMIN_PREFIX,),
                  controller='journal', action='index')
     rmap.connect('journal_rss', '%s/journal/rss' % (ADMIN_PREFIX,),
                  controller='journal', action='journal_rss')
     rmap.connect('journal_atom', '%s/journal/atom' % (ADMIN_PREFIX,),
                  controller='journal', action='journal_atom')
 
     rmap.connect('public_journal', '%s/public_journal' % (ADMIN_PREFIX,),
                  controller='journal', action='public_journal')
 
     rmap.connect('public_journal_rss', '%s/public_journal/rss' % (ADMIN_PREFIX,),
                  controller='journal', action='public_journal_rss')
 
     rmap.connect('public_journal_rss_old', '%s/public_journal_rss' % (ADMIN_PREFIX,),
                  controller='journal', action='public_journal_rss')
 
     rmap.connect('public_journal_atom',
                  '%s/public_journal/atom' % (ADMIN_PREFIX,), controller='journal',
                  action='public_journal_atom')
 
     rmap.connect('public_journal_atom_old',
                  '%s/public_journal_atom' % (ADMIN_PREFIX,), controller='journal',
                  action='public_journal_atom')
 
     rmap.connect('toggle_following', '%s/toggle_following' % (ADMIN_PREFIX,),
                  controller='journal', action='toggle_following', jsroute=True,
                  conditions={'method': ['POST']})
 
     # FULL TEXT SEARCH
     rmap.connect('search', '%s/search' % (ADMIN_PREFIX,),
                  controller='search')
     rmap.connect('search_repo_home', '/{repo_name}/search',
                  controller='search',
                  action='index',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     # FEEDS
     rmap.connect('rss_feed_home', '/{repo_name}/feed/rss',
                  controller='feed', action='rss',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('atom_feed_home', '/{repo_name}/feed/atom',
                  controller='feed', action='atom',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     #==========================================================================
     # REPOSITORY ROUTES
     #==========================================================================
 
     rmap.connect('repo_creating_home', '/{repo_name}/repo_creating',
                  controller='admin/repos', action='repo_creating',
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_check_home', '/{repo_name}/crepo_check',
                  controller='admin/repos', action='repo_check',
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('repo_stats', '/{repo_name}/repo_stats/{commit_id}',
                  controller='summary', action='repo_stats',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('repo_refs_data', '/{repo_name}/refs-data',
                  controller='summary', action='repo_refs_data', jsroute=True,
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_refs_changelog_data', '/{repo_name}/refs-data-changelog',
                  controller='summary', action='repo_refs_changelog_data',
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('changeset_home', '/{repo_name}/changeset/{revision}',
                  controller='changeset', revision='tip', jsroute=True,
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('changeset_children', '/{repo_name}/changeset_children/{revision}',
                  controller='changeset', revision='tip', action='changeset_children',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('changeset_parents', '/{repo_name}/changeset_parents/{revision}',
                  controller='changeset', revision='tip', action='changeset_parents',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     # repo edit options
     rmap.connect('edit_repo', '/{repo_name}/settings', jsroute=True,
                  controller='admin/repos', action='edit',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_perms', '/{repo_name}/settings/permissions',
                  jsroute=True,
                  controller='admin/repos', action='edit_permissions',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('edit_repo_perms_update', '/{repo_name}/settings/permissions',
                  controller='admin/repos', action='edit_permissions_update',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_fields', '/{repo_name}/settings/fields',
                  controller='admin/repos', action='edit_fields',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('create_repo_fields', '/{repo_name}/settings/fields/new',
                  controller='admin/repos', action='create_repo_field',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('delete_repo_fields', '/{repo_name}/settings/fields/{field_id}',
                  controller='admin/repos', action='delete_repo_field',
                  conditions={'method': ['DELETE'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_advanced', '/{repo_name}/settings/advanced',
                  controller='admin/repos', action='edit_advanced',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_advanced_locking', '/{repo_name}/settings/advanced/locking',
                  controller='admin/repos', action='edit_advanced_locking',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('toggle_locking', '/{repo_name}/settings/advanced/locking_toggle',
                  controller='admin/repos', action='toggle_locking',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_advanced_journal', '/{repo_name}/settings/advanced/journal',
                  controller='admin/repos', action='edit_advanced_journal',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_advanced_fork', '/{repo_name}/settings/advanced/fork',
                  controller='admin/repos', action='edit_advanced_fork',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_caches', '/{repo_name}/settings/caches',
                  controller='admin/repos', action='edit_caches_form',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('edit_repo_caches', '/{repo_name}/settings/caches',
                  controller='admin/repos', action='edit_caches',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_remote', '/{repo_name}/settings/remote',
                  controller='admin/repos', action='edit_remote_form',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('edit_repo_remote', '/{repo_name}/settings/remote',
                  controller='admin/repos', action='edit_remote',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('edit_repo_statistics', '/{repo_name}/settings/statistics',
                  controller='admin/repos', action='edit_statistics_form',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('edit_repo_statistics', '/{repo_name}/settings/statistics',
                  controller='admin/repos', action='edit_statistics',
                  conditions={'method': ['PUT'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_settings_issuetracker',
                  '/{repo_name}/settings/issue-tracker',
                  controller='admin/repos', action='repo_issuetracker',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_issuetracker_test',
                  '/{repo_name}/settings/issue-tracker/test',
                  controller='admin/repos', action='repo_issuetracker_test',
                  conditions={'method': ['POST'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_issuetracker_delete',
                  '/{repo_name}/settings/issue-tracker/delete',
                  controller='admin/repos', action='repo_issuetracker_delete',
                  conditions={'method': ['DELETE'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_issuetracker_save',
                  '/{repo_name}/settings/issue-tracker/save',
                  controller='admin/repos', action='repo_issuetracker_save',
                  conditions={'method': ['POST'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_vcs_settings', '/{repo_name}/settings/vcs',
                  controller='admin/repos', action='repo_settings_vcs_update',
                  conditions={'method': ['POST'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_vcs_settings', '/{repo_name}/settings/vcs',
                  controller='admin/repos', action='repo_settings_vcs',
                  conditions={'method': ['GET'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
     rmap.connect('repo_vcs_settings', '/{repo_name}/settings/vcs',
                  controller='admin/repos', action='repo_delete_svn_pattern',
                  conditions={'method': ['DELETE'], 'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     # still working url for backward compat.
     rmap.connect('raw_changeset_home_depraced',
                  '/{repo_name}/raw-changeset/{revision}',
                  controller='changeset', action='changeset_raw',
                  revision='tip', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     # new URLs
     rmap.connect('changeset_raw_home',
                  '/{repo_name}/changeset-diff/{revision}',
                  controller='changeset', action='changeset_raw',
                  revision='tip', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changeset_patch_home',
                  '/{repo_name}/changeset-patch/{revision}',
                  controller='changeset', action='changeset_patch',
                  revision='tip', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changeset_download_home',
                  '/{repo_name}/changeset-download/{revision}',
                  controller='changeset', action='changeset_download',
                  revision='tip', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changeset_comment',
                  '/{repo_name}/changeset/{revision}/comment', jsroute=True,
                  controller='changeset', revision='tip', action='comment',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changeset_comment_preview',
                  '/{repo_name}/changeset/comment/preview', jsroute=True,
                  controller='changeset', action='preview_comment',
                  conditions={'function': check_repo, 'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changeset_comment_delete',
                  '/{repo_name}/changeset/comment/{comment_id}/delete',
                  controller='changeset', action='delete_comment',
                  conditions={'function': check_repo, 'method': ['DELETE']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('changeset_info', '/changeset_info/{repo_name}/{revision}',
                  controller='changeset', action='changeset_info',
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('compare_home',
                  '/{repo_name}/compare',
                  controller='compare', action='index',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('compare_url',
                  '/{repo_name}/compare/{source_ref_type}@{source_ref:.*?}...{target_ref_type}@{target_ref:.*?}',
                  controller='compare', action='compare',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest_home',
                  '/{repo_name}/pull-request/new', controller='pullrequests',
                  action='index', conditions={'function': check_repo,
                                              'method': ['GET']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest',
                  '/{repo_name}/pull-request/new', controller='pullrequests',
                  action='create', conditions={'function': check_repo,
                                               'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest_repo_refs',
                  '/{repo_name}/pull-request/refs/{target_repo_name:.*?[^/]}',
                  controller='pullrequests',
                  action='get_repo_refs',
                  conditions={'function': check_repo, 'method': ['GET']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest_repo_destinations',
                  '/{repo_name}/pull-request/repo-destinations',
                  controller='pullrequests',
                  action='get_repo_destinations',
                  conditions={'function': check_repo, 'method': ['GET']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest_show',
                  '/{repo_name}/pull-request/{pull_request_id}',
                  controller='pullrequests',
                  action='show', conditions={'function': check_repo,
                                             'method': ['GET']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('pullrequest_update',
                  '/{repo_name}/pull-request/{pull_request_id}',
                  controller='pullrequests',
                  action='update', conditions={'function': check_repo,
                                               'method': ['PUT']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest_merge',
                  '/{repo_name}/pull-request/{pull_request_id}',
                  controller='pullrequests',
                  action='merge', conditions={'function': check_repo,
                                              'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('pullrequest_delete',
                  '/{repo_name}/pull-request/{pull_request_id}',
                  controller='pullrequests',
                  action='delete', conditions={'function': check_repo,
                                               'method': ['DELETE']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('pullrequest_show_all',
                  '/{repo_name}/pull-request',
                  controller='pullrequests',
                  action='show_all', conditions={'function': check_repo,
                                                 'method': ['GET']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest_comment',
                  '/{repo_name}/pull-request-comment/{pull_request_id}',
                  controller='pullrequests',
                  action='comment', conditions={'function': check_repo,
                                                'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('pullrequest_comment_delete',
                  '/{repo_name}/pull-request-comment/{comment_id}/delete',
                  controller='pullrequests', action='delete_comment',
                  conditions={'function': check_repo, 'method': ['DELETE']},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('summary_home_explicit', '/{repo_name}/summary',
                  controller='summary', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('branches_home', '/{repo_name}/branches',
                  controller='branches', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('tags_home', '/{repo_name}/tags',
                  controller='tags', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('bookmarks_home', '/{repo_name}/bookmarks',
                  controller='bookmarks', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changelog_home', '/{repo_name}/changelog', jsroute=True,
                  controller='changelog', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changelog_summary_home', '/{repo_name}/changelog_summary',
                  controller='changelog', action='changelog_summary',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('changelog_file_home',
                  '/{repo_name}/changelog/{revision}/{f_path}',
                  controller='changelog', f_path=None,
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('changelog_details', '/{repo_name}/changelog_details/{cs}',
                  controller='changelog', action='changelog_details',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_home',  '/{repo_name}/files/{revision}/{f_path}',
                  controller='files', revision='tip', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('files_home_simple_catchrev',
                  '/{repo_name}/files/{revision}',
                  controller='files', revision='tip', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_home_simple_catchall',
                  '/{repo_name}/files',
                  controller='files', revision='tip', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_history_home',
                  '/{repo_name}/history/{revision}/{f_path}',
                  controller='files', action='history', revision='tip', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('files_authors_home',
                  '/{repo_name}/authors/{revision}/{f_path}',
                  controller='files', action='authors', revision='tip', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('files_diff_home', '/{repo_name}/diff/{f_path}',
                  controller='files', action='diff', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_diff_2way_home',
                  '/{repo_name}/diff-2way/{f_path}',
                  controller='files', action='diff_2way', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_rawfile_home',
                  '/{repo_name}/rawfile/{revision}/{f_path}',
                  controller='files', action='rawfile', revision='tip',
                  f_path='', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_raw_home',
                  '/{repo_name}/raw/{revision}/{f_path}',
                  controller='files', action='raw', revision='tip', f_path='',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_render_home',
                  '/{repo_name}/render/{revision}/{f_path}',
                  controller='files', action='index', revision='tip', f_path='',
                  rendered=True, conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_annotate_home',
                  '/{repo_name}/annotate/{revision}/{f_path}',
                  controller='files', action='index', revision='tip',
                  f_path='', annotate=True, conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_edit',
                  '/{repo_name}/edit/{revision}/{f_path}',
                  controller='files', action='edit', revision='tip',
                  f_path='',
                  conditions={'function': check_repo, 'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_edit_home',
                  '/{repo_name}/edit/{revision}/{f_path}',
                  controller='files', action='edit_home', revision='tip',
                  f_path='', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_add',
                  '/{repo_name}/add/{revision}/{f_path}',
                  controller='files', action='add', revision='tip',
                  f_path='',
                  conditions={'function': check_repo, 'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_add_home',
                  '/{repo_name}/add/{revision}/{f_path}',
                  controller='files', action='add_home', revision='tip',
                  f_path='', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_delete',
                  '/{repo_name}/delete/{revision}/{f_path}',
                  controller='files', action='delete', revision='tip',
                  f_path='',
                  conditions={'function': check_repo, 'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_delete_home',
                  '/{repo_name}/delete/{revision}/{f_path}',
                  controller='files', action='delete_home', revision='tip',
                  f_path='', conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('files_archive_home', '/{repo_name}/archive/{fname}',
                  controller='files', action='archivefile',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('files_nodelist_home',
                  '/{repo_name}/nodelist/{revision}/{f_path}',
                  controller='files', action='nodelist',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('files_metadata_list_home',
                  '/{repo_name}/metadata_list/{revision}/{f_path}',
                  controller='files', action='metadata_list',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS, jsroute=True)
 
     rmap.connect('repo_fork_create_home', '/{repo_name}/fork',
                  controller='forks', action='fork_create',
                  conditions={'function': check_repo, 'method': ['POST']},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('repo_fork_home', '/{repo_name}/fork',
                  controller='forks', action='fork',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('repo_forks_home', '/{repo_name}/forks',
                  controller='forks', action='forks',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     rmap.connect('repo_followers_home', '/{repo_name}/followers',
                  controller='followers', action='followers',
                  conditions={'function': check_repo},
                  requirements=URL_NAME_REQUIREMENTS)
 
     # must be here for proper group/repo catching pattern
     _connect_with_slash(
         rmap, 'repo_group_home', '/{group_name}',
         controller='home', action='index_repo_group',
         conditions={'function': check_group},
         requirements=URL_NAME_REQUIREMENTS)
 
     # catch all, at the end
     _connect_with_slash(
         rmap, 'summary_home', '/{repo_name}', jsroute=True,
         controller='summary', action='index',
         conditions={'function': check_repo},
         requirements=URL_NAME_REQUIREMENTS)
 
     return rmap
 
 
 def _connect_with_slash(mapper, name, path, *args, **kwargs):
     """
     Connect a route with an optional trailing slash in `path`.
     """
     mapper.connect(name + '_slash', path + '/', *args, **kwargs)
     mapper.connect(name, path, *args, **kwargs)

rhodecode/config/utils.py

@@ -1,99 +1,103 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import os
 import shlex
 import Pyro4
 import platform
 
 from rhodecode.model import init_model
 
 
 def configure_pyro4(config):
     """
     Configure Pyro4 based on `config`.
 
     This will mainly set the different configuration parameters of the Pyro4
     library based on the settings in our INI files. The Pyro4 documentation
     lists more details about the specific settings and their meaning.
     """
     Pyro4.config.COMMTIMEOUT = float(config['vcs.connection_timeout'])
     Pyro4.config.SERIALIZER = 'pickle'
     Pyro4.config.SERIALIZERS_ACCEPTED.add('pickle')
 
     # Note: We need server configuration in the WSGI processes
     # because we provide a callback server in certain vcs operations.
     Pyro4.config.SERVERTYPE = "multiplex"
     Pyro4.config.POLLTIMEOUT = 0.01
 
 
 def configure_vcs(config):
     """
     Patch VCS config with some RhodeCode specific stuff
     """
     from rhodecode.lib.vcs import conf
     from rhodecode.lib.utils2 import aslist
     conf.settings.BACKENDS = {
         'hg': 'rhodecode.lib.vcs.backends.hg.MercurialRepository',
         'git': 'rhodecode.lib.vcs.backends.git.GitRepository',
         'svn': 'rhodecode.lib.vcs.backends.svn.SubversionRepository',
     }
 
     conf.settings.HG_USE_REBASE_FOR_MERGING = config.get(
         'rhodecode_hg_use_rebase_for_merging', False)
     conf.settings.GIT_REV_FILTER = shlex.split(
         config.get('git_rev_filter', '--all').strip())
     conf.settings.DEFAULT_ENCODINGS = aslist(config.get('default_encoding',
                                                         'UTF-8'), sep=',')
     conf.settings.ALIASES[:] = config.get('vcs.backends')
     conf.settings.SVN_COMPATIBLE_VERSION = config.get(
         'vcs.svn.compatible_version')
 
 
 def initialize_database(config):
-    from rhodecode.lib.utils2 import engine_from_config
+    from rhodecode.lib.utils2 import engine_from_config, get_encryption_key
     engine = engine_from_config(config, 'sqlalchemy.db1.')
-    init_model(engine, encryption_key=config['beaker.session.secret'])
+    init_model(engine, encryption_key=get_encryption_key(config))
 
 
 def initialize_test_environment(settings, test_env=None):
     if test_env is None:
         test_env = not int(os.environ.get('RC_NO_TMP_PATH', 0))
 
-    from rhodecode.lib.utils import create_test_env, create_test_index
+    from rhodecode.lib.utils import (
+        create_test_directory, create_test_database, create_test_repositories,
+        create_test_index)
     from rhodecode.tests import TESTS_TMP_PATH
     # test repos
     if test_env:
-        create_test_env(TESTS_TMP_PATH, settings)
-        create_test_index(TESTS_TMP_PATH, settings, True)
+        create_test_directory(TESTS_TMP_PATH)
+        create_test_database(TESTS_TMP_PATH, settings)
+        create_test_repositories(TESTS_TMP_PATH, settings)
+        create_test_index(TESTS_TMP_PATH, settings)
 
 
 def get_vcs_server_protocol(config):
     protocol = config.get('vcs.server.protocol', 'pyro4')
     return protocol
 
 
 def set_instance_id(config):
     """ Sets a dynamic generated config['instance_id'] if missing or '*' """
 
     config['instance_id'] = config.get('instance_id') or ''
     if config['instance_id'] == '*' or not config['instance_id']:
         _platform_id = platform.uname()[1] or 'instance'
         config['instance_id'] = '%s-%s' % (_platform_id, os.getpid())

rhodecode/controllers/admin/repo_groups.py

@@ -1,407 +1,406 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 """
 Repository groups controller for RhodeCode
 """
 
 import logging
 import formencode
 
 from formencode import htmlfill
 
 from pylons import request, tmpl_context as c, url
 from pylons.controllers.util import abort, redirect
 from pylons.i18n.translation import _, ungettext
 
 from rhodecode.lib import auth
 from rhodecode.lib import helpers as h
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.auth import (
     LoginRequired, NotAnonymous, HasPermissionAll,
     HasRepoGroupPermissionAll, HasRepoGroupPermissionAnyDecorator)
 from rhodecode.lib.base import BaseController, render
 from rhodecode.model.db import RepoGroup, User
 from rhodecode.model.scm import RepoGroupList
 from rhodecode.model.repo_group import RepoGroupModel
 from rhodecode.model.forms import RepoGroupForm, RepoGroupPermsForm
 from rhodecode.model.meta import Session
 from rhodecode.lib.utils2 import safe_int
 
 
 log = logging.getLogger(__name__)
 
 
 class RepoGroupsController(BaseController):
     """REST Controller styled on the Atom Publishing Protocol"""
 
     @LoginRequired()
     def __before__(self):
         super(RepoGroupsController, self).__before__()
 
     def __load_defaults(self, allow_empty_group=False, repo_group=None):
         if self._can_create_repo_group():
             # we're global admin, we're ok and we can create TOP level groups
             allow_empty_group = True
 
         # override the choices for this form, we need to filter choices
         # and display only those we have ADMIN right
         groups_with_admin_rights = RepoGroupList(
             RepoGroup.query().all(),
             perm_set=['group.admin'])
         c.repo_groups = RepoGroup.groups_choices(
             groups=groups_with_admin_rights,
             show_empty_group=allow_empty_group)
 
         if repo_group:
             # exclude filtered ids
             exclude_group_ids = [repo_group.group_id]
             c.repo_groups = filter(lambda x: x[0] not in exclude_group_ids,
                                    c.repo_groups)
             c.repo_groups_choices = map(lambda k: unicode(k[0]), c.repo_groups)
             parent_group = repo_group.parent_group
 
             add_parent_group = (parent_group and (
                 unicode(parent_group.group_id) not in c.repo_groups_choices))
             if add_parent_group:
                 c.repo_groups_choices.append(unicode(parent_group.group_id))
                 c.repo_groups.append(RepoGroup._generate_choice(parent_group))
 
     def __load_data(self, group_id):
         """
         Load defaults settings for edit, and update
 
         :param group_id:
         """
         repo_group = RepoGroup.get_or_404(group_id)
         data = repo_group.get_dict()
         data['group_name'] = repo_group.name
 
         # fill owner
         if repo_group.user:
             data.update({'user': repo_group.user.username})
         else:
-            replacement_user = User.get_first_admin().username
+            replacement_user = User.get_first_super_admin().username
             data.update({'user': replacement_user})
 
         # fill repository group users
         for p in repo_group.repo_group_to_perm:
             data.update({
                 'u_perm_%s' % p.user.user_id: p.permission.permission_name})
 
         # fill repository group user groups
         for p in repo_group.users_group_to_perm:
             data.update({
                 'g_perm_%s' % p.users_group.users_group_id:
                 p.permission.permission_name})
         # html and form expects -1 as empty parent group
         data['group_parent_id'] = data['group_parent_id'] or -1
         return data
 
     def _revoke_perms_on_yourself(self, form_result):
         _updates = filter(lambda u: c.rhodecode_user.user_id == int(u[0]),
                           form_result['perm_updates'])
         _additions = filter(lambda u: c.rhodecode_user.user_id == int(u[0]),
                             form_result['perm_additions'])
         _deletions = filter(lambda u: c.rhodecode_user.user_id == int(u[0]),
                             form_result['perm_deletions'])
         admin_perm = 'group.admin'
         if _updates and _updates[0][1] != admin_perm or \
            _additions and _additions[0][1] != admin_perm or \
            _deletions and _deletions[0][1] != admin_perm:
             return True
         return False
 
     def _can_create_repo_group(self, parent_group_id=None):
         is_admin = HasPermissionAll('hg.admin')('group create controller')
         create_repo_group = HasPermissionAll(
             'hg.repogroup.create.true')('group create controller')
         if is_admin or (create_repo_group and not parent_group_id):
             # we're global admin, or we have global repo group create
             # permission
             # we're ok and we can create TOP level groups
             return True
         elif parent_group_id:
             # we check the permission if we can write to parent group
             group = RepoGroup.get(parent_group_id)
             group_name = group.group_name if group else None
             if HasRepoGroupPermissionAll('group.admin')(
                     group_name, 'check if user is an admin of group'):
                 # we're an admin of passed in group, we're ok.
                 return True
             else:
                 return False
         return False
 
     @NotAnonymous()
     def index(self):
         """GET /repo_groups: All items in the collection"""
         # url('repo_groups')
 
         repo_group_list = RepoGroup.get_all_repo_groups()
         _perms = ['group.admin']
         repo_group_list_acl = RepoGroupList(repo_group_list, perm_set=_perms)
         repo_group_data = RepoGroupModel().get_repo_groups_as_dict(
             repo_group_list=repo_group_list_acl, admin=True)
         c.data = json.dumps(repo_group_data)
         return render('admin/repo_groups/repo_groups.html')
 
     # perm checks inside
     @NotAnonymous()
     @auth.CSRFRequired()
     def create(self):
         """POST /repo_groups: Create a new item"""
         # url('repo_groups')
 
         parent_group_id = safe_int(request.POST.get('group_parent_id'))
         can_create = self._can_create_repo_group(parent_group_id)
 
         self.__load_defaults()
         # permissions for can create group based on parent_id are checked
         # here in the Form
         available_groups = map(lambda k: unicode(k[0]), c.repo_groups)
         repo_group_form = RepoGroupForm(available_groups=available_groups,
                                         can_create_in_root=can_create)()
         try:
             owner = c.rhodecode_user
             form_result = repo_group_form.to_python(dict(request.POST))
             RepoGroupModel().create(
                 group_name=form_result['group_name_full'],
                 group_description=form_result['group_description'],
                 owner=owner.user_id,
                 copy_permissions=form_result['group_copy_permissions']
             )
             Session().commit()
             _new_group_name = form_result['group_name_full']
             repo_group_url = h.link_to(
                 _new_group_name,
                 h.url('repo_group_home', group_name=_new_group_name))
             h.flash(h.literal(_('Created repository group %s')
                     % repo_group_url), category='success')
             # TODO: in futureaction_logger(, '', '', '', self.sa)
         except formencode.Invalid as errors:
             return htmlfill.render(
                 render('admin/repo_groups/repo_group_add.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
         except Exception:
             log.exception("Exception during creation of repository group")
             h.flash(_('Error occurred during creation of repository group %s')
                     % request.POST.get('group_name'), category='error')
 
         # TODO: maybe we should get back to the main view, not the admin one
         return redirect(url('repo_groups', parent_group=parent_group_id))
 
     # perm checks inside
     @NotAnonymous()
     def new(self):
         """GET /repo_groups/new: Form to create a new item"""
         # url('new_repo_group')
         # perm check for admin, create_group perm or admin of parent_group
         parent_group_id = safe_int(request.GET.get('parent_group'))
         if not self._can_create_repo_group(parent_group_id):
             return abort(403)
 
         self.__load_defaults()
         return render('admin/repo_groups/repo_group_add.html')
 
     @HasRepoGroupPermissionAnyDecorator('group.admin')
     @auth.CSRFRequired()
     def update(self, group_name):
         """PUT /repo_groups/group_name: Update an existing item"""
         # Forms posted to this method should contain a hidden field:
         #    <input type="hidden" name="_method" value="PUT" />
         # Or using helpers:
         #    h.form(url('repos_group', group_name=GROUP_NAME), method='put')
         # url('repo_group_home', group_name=GROUP_NAME)
 
         c.repo_group = RepoGroupModel()._get_repo_group(group_name)
         can_create_in_root = self._can_create_repo_group()
         show_root_location = can_create_in_root
         if not c.repo_group.parent_group:
             # this group don't have a parrent so we should show empty value
             show_root_location = True
         self.__load_defaults(allow_empty_group=show_root_location,
                              repo_group=c.repo_group)
 
         repo_group_form = RepoGroupForm(
-            edit=True,
-            old_data=c.repo_group.get_dict(),
+            edit=True, old_data=c.repo_group.get_dict(),
             available_groups=c.repo_groups_choices,
-            can_create_in_root=can_create_in_root,
-        )()
+            can_create_in_root=can_create_in_root, allow_disabled=True)()
+
         try:
             form_result = repo_group_form.to_python(dict(request.POST))
             gr_name = form_result['group_name']
             new_gr = RepoGroupModel().update(group_name, form_result)
             Session().commit()
             h.flash(_('Updated repository group %s') % (gr_name,),
                     category='success')
             # we now have new name !
             group_name = new_gr.group_name
             # TODO: in future action_logger(, '', '', '', self.sa)
         except formencode.Invalid as errors:
             c.active = 'settings'
             return htmlfill.render(
                 render('admin/repo_groups/repo_group_edit.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
         except Exception:
             log.exception("Exception during update or repository group")
             h.flash(_('Error occurred during update of repository group %s')
                     % request.POST.get('group_name'), category='error')
 
         return redirect(url('edit_repo_group', group_name=group_name))
 
     @HasRepoGroupPermissionAnyDecorator('group.admin')
     @auth.CSRFRequired()
     def delete(self, group_name):
         """DELETE /repo_groups/group_name: Delete an existing item"""
         # Forms posted to this method should contain a hidden field:
         #    <input type="hidden" name="_method" value="DELETE" />
         # Or using helpers:
         #    h.form(url('repos_group', group_name=GROUP_NAME), method='delete')
         # url('repo_group_home', group_name=GROUP_NAME)
 
         gr = c.repo_group = RepoGroupModel()._get_repo_group(group_name)
         repos = gr.repositories.all()
         if repos:
             msg = ungettext(
                 'This group contains %(num)d repository and cannot be deleted',
                 'This group contains %(num)d repositories and cannot be'
                 ' deleted',
                 len(repos)) % {'num': len(repos)}
             h.flash(msg, category='warning')
             return redirect(url('repo_groups'))
 
         children = gr.children.all()
         if children:
             msg = ungettext(
                 'This group contains %(num)d subgroup and cannot be deleted',
                 'This group contains %(num)d subgroups and cannot be deleted',
                 len(children)) % {'num': len(children)}
             h.flash(msg, category='warning')
             return redirect(url('repo_groups'))
 
         try:
             RepoGroupModel().delete(group_name)
             Session().commit()
             h.flash(_('Removed repository group %s') % group_name,
                     category='success')
             # TODO: in future action_logger(, '', '', '', self.sa)
         except Exception:
             log.exception("Exception during deletion of repository group")
             h.flash(_('Error occurred during deletion of repository group %s')
                     % group_name, category='error')
 
         return redirect(url('repo_groups'))
 
     @HasRepoGroupPermissionAnyDecorator('group.admin')
     def edit(self, group_name):
         """GET /repo_groups/group_name/edit: Form to edit an existing item"""
         # url('edit_repo_group', group_name=GROUP_NAME)
         c.active = 'settings'
 
         c.repo_group = RepoGroupModel()._get_repo_group(group_name)
         # we can only allow moving empty group if it's already a top-level
         # group, ie has no parents, or we're admin
         can_create_in_root = self._can_create_repo_group()
         show_root_location = can_create_in_root
         if not c.repo_group.parent_group:
             # this group don't have a parrent so we should show empty value
             show_root_location = True
         self.__load_defaults(allow_empty_group=show_root_location,
                              repo_group=c.repo_group)
         defaults = self.__load_data(c.repo_group.group_id)
 
         return htmlfill.render(
             render('admin/repo_groups/repo_group_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False
         )
 
     @HasRepoGroupPermissionAnyDecorator('group.admin')
     def edit_repo_group_advanced(self, group_name):
         """GET /repo_groups/group_name/edit: Form to edit an existing item"""
         # url('edit_repo_group', group_name=GROUP_NAME)
         c.active = 'advanced'
         c.repo_group = RepoGroupModel()._get_repo_group(group_name)
 
         return render('admin/repo_groups/repo_group_edit.html')
 
     @HasRepoGroupPermissionAnyDecorator('group.admin')
     def edit_repo_group_perms(self, group_name):
         """GET /repo_groups/group_name/edit: Form to edit an existing item"""
         # url('edit_repo_group', group_name=GROUP_NAME)
         c.active = 'perms'
         c.repo_group = RepoGroupModel()._get_repo_group(group_name)
         self.__load_defaults()
         defaults = self.__load_data(c.repo_group.group_id)
 
         return htmlfill.render(
             render('admin/repo_groups/repo_group_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False
         )
 
     @HasRepoGroupPermissionAnyDecorator('group.admin')
     @auth.CSRFRequired()
     def update_perms(self, group_name):
         """
         Update permissions for given repository group
 
         :param group_name:
         """
 
         c.repo_group = RepoGroupModel()._get_repo_group(group_name)
         valid_recursive_choices = ['none', 'repos', 'groups', 'all']
         form = RepoGroupPermsForm(valid_recursive_choices)().to_python(
             request.POST)
 
         if not c.rhodecode_user.is_admin:
             if self._revoke_perms_on_yourself(form):
                 msg = _('Cannot change permission for yourself as admin')
                 h.flash(msg, category='warning')
                 return redirect(
                     url('edit_repo_group_perms', group_name=group_name))
 
         # iterate over all members(if in recursive mode) of this groups and
         # set the permissions !
         # this can be potentially heavy operation
         RepoGroupModel().update_permissions(
             c.repo_group,
             form['perm_additions'], form['perm_updates'],
             form['perm_deletions'], form['recursive'])
 
         # TODO: implement this
         # action_logger(c.rhodecode_user, 'admin_changed_repo_permissions',
         #               repo_name, self.ip_addr, self.sa)
         Session().commit()
         h.flash(_('Repository Group permissions updated'), category='success')
         return redirect(url('edit_repo_group_perms', group_name=group_name))

rhodecode/controllers/admin/repos.py

@@ -1,878 +1,878 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2013-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 """
 Repositories controller for RhodeCode
 """
 
 import logging
 import traceback
 
 import formencode
 from formencode import htmlfill
 from pylons import request, tmpl_context as c, url
 from pylons.controllers.util import redirect
 from pylons.i18n.translation import _
 from webob.exc import HTTPForbidden, HTTPNotFound, HTTPBadRequest
 
 from rhodecode.lib import auth, helpers as h
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator,
     HasRepoPermissionAllDecorator, NotAnonymous, HasPermissionAny,
     HasRepoGroupPermissionAny, HasRepoPermissionAnyDecorator)
 from rhodecode.lib.base import BaseRepoController, render
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.exceptions import AttachedForksError
 from rhodecode.lib.utils import action_logger, repo_name_slug, jsonify
 from rhodecode.lib.utils2 import safe_int
 from rhodecode.lib.vcs import RepositoryError
 from rhodecode.model.db import (
     User, Repository, UserFollowing, RepoGroup, RepositoryField)
 from rhodecode.model.forms import (
     RepoForm, RepoFieldForm, RepoPermsForm, RepoVcsSettingsForm,
     IssueTrackerPatternsForm)
 from rhodecode.model.meta import Session
 from rhodecode.model.repo import RepoModel
 from rhodecode.model.scm import ScmModel, RepoGroupList, RepoList
 from rhodecode.model.settings import (
     SettingsModel, IssueTrackerSettingsModel, VcsSettingsModel,
     SettingNotFound)
 
 log = logging.getLogger(__name__)
 
 
 class ReposController(BaseRepoController):
     """
     REST Controller styled on the Atom Publishing Protocol"""
     # To properly map this controller, ensure your config/routing.py
     # file has a resource setup:
     #     map.resource('repo', 'repos')
 
     @LoginRequired()
     def __before__(self):
         super(ReposController, self).__before__()
 
     def _load_repo(self, repo_name):
         repo_obj = Repository.get_by_repo_name(repo_name)
 
         if repo_obj is None:
             h.not_mapped_error(repo_name)
             return redirect(url('repos'))
 
         return repo_obj
 
     def __load_defaults(self, repo=None):
         acl_groups = RepoGroupList(RepoGroup.query().all(),
                                perm_set=['group.write', 'group.admin'])
         c.repo_groups = RepoGroup.groups_choices(groups=acl_groups)
         c.repo_groups_choices = map(lambda k: unicode(k[0]), c.repo_groups)
 
         # in case someone no longer have a group.write access to a repository
         # pre fill the list with this entry, we don't care if this is the same
         # but it will allow saving repo data properly.
 
         repo_group = None
         if repo:
             repo_group = repo.group
         if repo_group and unicode(repo_group.group_id) not in c.repo_groups_choices:
             c.repo_groups_choices.append(unicode(repo_group.group_id))
             c.repo_groups.append(RepoGroup._generate_choice(repo_group))
 
         choices, c.landing_revs = ScmModel().get_repo_landing_revs()
         c.landing_revs_choices = choices
 
     def __load_data(self, repo_name=None):
         """
         Load defaults settings for edit, and update
 
         :param repo_name:
         """
         c.repo_info = self._load_repo(repo_name)
         self.__load_defaults(c.repo_info)
 
         # override defaults for exact repo info here git/hg etc
         if not c.repository_requirements_missing:
             choices, c.landing_revs = ScmModel().get_repo_landing_revs(
                 c.repo_info)
             c.landing_revs_choices = choices
         defaults = RepoModel()._get_defaults(repo_name)
 
         return defaults
 
     def _log_creation_exception(self, e, repo_name):
             reason = None
             if len(e.args) == 2:
                 reason = e.args[1]
 
             if reason == 'INVALID_CERTIFICATE':
                 log.exception(
                     'Exception creating a repository: invalid certificate')
                 msg = (_('Error creating repository %s: invalid certificate')
                        % repo_name)
             else:
                 log.exception("Exception creating a repository")
                 msg = (_('Error creating repository %s')
                        % repo_name)
 
             return msg
 
     @NotAnonymous()
     def index(self, format='html'):
         """GET /repos: All items in the collection"""
         # url('repos')
 
         repo_list = Repository.get_all_repos()
         c.repo_list = RepoList(repo_list, perm_set=['repository.admin'])
         repos_data = RepoModel().get_repos_as_dict(
             repo_list=c.repo_list, admin=True, super_user_actions=True)
         # json used to render the grid
         c.data = json.dumps(repos_data)
 
         return render('admin/repos/repos.html')
 
     # perms check inside
     @NotAnonymous()
     @auth.CSRFRequired()
     def create(self):
         """
         POST /repos: Create a new item"""
         # url('repos')
 
         self.__load_defaults()
         form_result = {}
         task_id = None
         try:
             # CanWriteToGroup validators checks permissions of this POST
             form_result = RepoForm(repo_groups=c.repo_groups_choices,
                                    landing_revs=c.landing_revs_choices)()\
                             .to_python(dict(request.POST))
 
             # create is done sometimes async on celery, db transaction
             # management is handled there.
             task = RepoModel().create(form_result, c.rhodecode_user.user_id)
             from celery.result import BaseAsyncResult
             if isinstance(task, BaseAsyncResult):
                 task_id = task.task_id
         except formencode.Invalid as errors:
             c.personal_repo_group = RepoGroup.get_by_group_name(
                 c.rhodecode_user.username)
             return htmlfill.render(
                 render('admin/repos/repo_add.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
 
         except Exception as e:
             msg = self._log_creation_exception(e, form_result.get('repo_name'))
             h.flash(msg, category='error')
             return redirect(url('home'))
 
         return redirect(h.url('repo_creating_home',
                               repo_name=form_result['repo_name_full'],
                               task_id=task_id))
 
     # perms check inside
     @NotAnonymous()
     def create_repository(self):
         """GET /_admin/create_repository: Form to create a new item"""
         new_repo = request.GET.get('repo', '')
         parent_group = request.GET.get('parent_group')
         if not HasPermissionAny('hg.admin', 'hg.create.repository')():
             # you're not super admin nor have global create permissions,
             # but maybe you have at least write permission to a parent group ?
             _gr = RepoGroup.get(parent_group)
             gr_name = _gr.group_name if _gr else None
             # create repositories with write permission on group is set to true
             create_on_write = HasPermissionAny('hg.create.write_on_repogroup.true')()
             group_admin = HasRepoGroupPermissionAny('group.admin')(group_name=gr_name)
             group_write = HasRepoGroupPermissionAny('group.write')(group_name=gr_name)
             if not (group_admin or (group_write and create_on_write)):
                 raise HTTPForbidden
 
         acl_groups = RepoGroupList(RepoGroup.query().all(),
                                perm_set=['group.write', 'group.admin'])
         c.repo_groups = RepoGroup.groups_choices(groups=acl_groups)
         c.repo_groups_choices = map(lambda k: unicode(k[0]), c.repo_groups)
         choices, c.landing_revs = ScmModel().get_repo_landing_revs()
         c.personal_repo_group = RepoGroup.get_by_group_name(c.rhodecode_user.username)
         c.new_repo = repo_name_slug(new_repo)
 
         ## apply the defaults from defaults page
         defaults = SettingsModel().get_default_repo_settings(strip_prefix=True)
         # set checkbox to autochecked
         defaults['repo_copy_permissions'] = True
         if parent_group:
             defaults.update({'repo_group': parent_group})
 
         return htmlfill.render(
             render('admin/repos/repo_add.html'),
             defaults=defaults,
             errors={},
             prefix_error=False,
             encoding="UTF-8",
             force_defaults=False
         )
 
     @NotAnonymous()
     def repo_creating(self, repo_name):
         c.repo = repo_name
         c.task_id = request.GET.get('task_id')
         if not c.repo:
             raise HTTPNotFound()
         return render('admin/repos/repo_creating.html')
 
     @NotAnonymous()
     @jsonify
     def repo_check(self, repo_name):
         c.repo = repo_name
         task_id = request.GET.get('task_id')
 
         if task_id and task_id not in ['None']:
-            from rhodecode import CELERY_ENABLED
+            import rhodecode
             from celery.result import AsyncResult
-            if CELERY_ENABLED:
+            if rhodecode.CELERY_ENABLED:
                 task = AsyncResult(task_id)
                 if task.failed():
                     msg = self._log_creation_exception(task.result, c.repo)
                     h.flash(msg, category='error')
                     return redirect(url('home'), code=501)
 
         repo = Repository.get_by_repo_name(repo_name)
         if repo and repo.repo_state == Repository.STATE_CREATED:
             if repo.clone_uri:
                 clone_uri = repo.clone_uri_hidden
                 h.flash(_('Created repository %s from %s')
                         % (repo.repo_name, clone_uri), category='success')
             else:
                 repo_url = h.link_to(repo.repo_name,
                                      h.url('summary_home',
                                            repo_name=repo.repo_name))
                 fork = repo.fork
                 if fork:
                     fork_name = fork.repo_name
                     h.flash(h.literal(_('Forked repository %s as %s')
                             % (fork_name, repo_url)), category='success')
                 else:
                     h.flash(h.literal(_('Created repository %s') % repo_url),
                             category='success')
             return {'result': True}
         return {'result': False}
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def update(self, repo_name):
         """
         PUT /repos/repo_name: Update an existing item"""
         # Forms posted to this method should contain a hidden field:
         #    <input type="hidden" name="_method" value="PUT" />
         # Or using helpers:
         #    h.form(url('repo', repo_name=ID),
         #           method='put')
         # url('repo', repo_name=ID)
 
         self.__load_data(repo_name)
         c.active = 'settings'
         c.repo_fields = RepositoryField.query()\
             .filter(RepositoryField.repository == c.repo_info).all()
 
         repo_model = RepoModel()
         changed_name = repo_name
 
         # override the choices with extracted revisions !
         c.personal_repo_group = RepoGroup.get_by_group_name(
             c.rhodecode_user.username)
         repo = Repository.get_by_repo_name(repo_name)
         old_data = {
             'repo_name': repo_name,
             'repo_group': repo.group.get_dict() if repo.group else {},
             'repo_type': repo.repo_type,
         }
-        _form = RepoForm(edit=True, old_data=old_data,
-                         repo_groups=c.repo_groups_choices,
-                         landing_revs=c.landing_revs_choices)()
+        _form = RepoForm(
+            edit=True, old_data=old_data, repo_groups=c.repo_groups_choices,
+            landing_revs=c.landing_revs_choices, allow_disabled=True)()
 
         try:
             form_result = _form.to_python(dict(request.POST))
             repo = repo_model.update(repo_name, **form_result)
             ScmModel().mark_for_invalidation(repo_name)
             h.flash(_('Repository %s updated successfully') % repo_name,
                     category='success')
             changed_name = repo.repo_name
             action_logger(c.rhodecode_user, 'admin_updated_repo',
                               changed_name, self.ip_addr, self.sa)
             Session().commit()
         except formencode.Invalid as errors:
             defaults = self.__load_data(repo_name)
             defaults.update(errors.value)
             return htmlfill.render(
                 render('admin/repos/repo_edit.html'),
                 defaults=defaults,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
 
         except Exception:
             log.exception("Exception during update of repository")
             h.flash(_('Error occurred during update of repository %s') \
                     % repo_name, category='error')
         return redirect(url('edit_repo', repo_name=changed_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def delete(self, repo_name):
         """
         DELETE /repos/repo_name: Delete an existing item"""
         # Forms posted to this method should contain a hidden field:
         #    <input type="hidden" name="_method" value="DELETE" />
         # Or using helpers:
         #    h.form(url('repo', repo_name=ID),
         #           method='delete')
         # url('repo', repo_name=ID)
 
         repo_model = RepoModel()
         repo = repo_model.get_by_repo_name(repo_name)
         if not repo:
             h.not_mapped_error(repo_name)
             return redirect(url('repos'))
         try:
             _forks = repo.forks.count()
             handle_forks = None
             if _forks and request.POST.get('forks'):
                 do = request.POST['forks']
                 if do == 'detach_forks':
                     handle_forks = 'detach'
                     h.flash(_('Detached %s forks') % _forks, category='success')
                 elif do == 'delete_forks':
                     handle_forks = 'delete'
                     h.flash(_('Deleted %s forks') % _forks, category='success')
             repo_model.delete(repo, forks=handle_forks)
             action_logger(c.rhodecode_user, 'admin_deleted_repo',
                   repo_name, self.ip_addr, self.sa)
             ScmModel().mark_for_invalidation(repo_name)
             h.flash(_('Deleted repository %s') % repo_name, category='success')
             Session().commit()
         except AttachedForksError:
             h.flash(_('Cannot delete %s it still contains attached forks')
                         % repo_name, category='warning')
 
         except Exception:
             log.exception("Exception during deletion of repository")
             h.flash(_('An error occurred during deletion of %s') % repo_name,
                     category='error')
 
         return redirect(url('repos'))
 
     @HasPermissionAllDecorator('hg.admin')
     def show(self, repo_name, format='html'):
         """GET /repos/repo_name: Show a specific item"""
         # url('repo', repo_name=ID)
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def edit(self, repo_name):
         """GET /repo_name/settings: Form to edit an existing item"""
         # url('edit_repo', repo_name=ID)
         defaults = self.__load_data(repo_name)
         if 'clone_uri' in defaults:
             del defaults['clone_uri']
 
         c.repo_fields = RepositoryField.query()\
             .filter(RepositoryField.repository == c.repo_info).all()
         c.personal_repo_group = RepoGroup.get_by_group_name(
             c.rhodecode_user.username)
         c.active = 'settings'
         return htmlfill.render(
             render('admin/repos/repo_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def edit_permissions(self, repo_name):
         """GET /repo_name/settings: Form to edit an existing item"""
         # url('edit_repo', repo_name=ID)
         c.repo_info = self._load_repo(repo_name)
         c.active = 'permissions'
         defaults = RepoModel()._get_defaults(repo_name)
 
         return htmlfill.render(
             render('admin/repos/repo_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def edit_permissions_update(self, repo_name):
         form = RepoPermsForm()().to_python(request.POST)
         RepoModel().update_permissions(repo_name,
             form['perm_additions'], form['perm_updates'], form['perm_deletions'])
 
         #TODO: implement this
         #action_logger(c.rhodecode_user, 'admin_changed_repo_permissions',
         #              repo_name, self.ip_addr, self.sa)
         Session().commit()
         h.flash(_('Repository permissions updated'), category='success')
         return redirect(url('edit_repo_perms', repo_name=repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def edit_fields(self, repo_name):
         """GET /repo_name/settings: Form to edit an existing item"""
         # url('edit_repo', repo_name=ID)
         c.repo_info = self._load_repo(repo_name)
         c.repo_fields = RepositoryField.query()\
             .filter(RepositoryField.repository == c.repo_info).all()
         c.active = 'fields'
         if request.POST:
 
             return redirect(url('repo_edit_fields'))
         return render('admin/repos/repo_edit.html')
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def create_repo_field(self, repo_name):
         try:
             form_result = RepoFieldForm()().to_python(dict(request.POST))
             RepoModel().add_repo_field(
                 repo_name, form_result['new_field_key'],
                 field_type=form_result['new_field_type'],
                 field_value=form_result['new_field_value'],
                 field_label=form_result['new_field_label'],
                 field_desc=form_result['new_field_desc'])
 
             Session().commit()
         except Exception as e:
             log.exception("Exception creating field")
             msg = _('An error occurred during creation of field')
             if isinstance(e, formencode.Invalid):
                 msg += ". " + e.msg
             h.flash(msg, category='error')
         return redirect(url('edit_repo_fields', repo_name=repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def delete_repo_field(self, repo_name, field_id):
         field = RepositoryField.get_or_404(field_id)
         try:
             RepoModel().delete_repo_field(repo_name, field.field_key)
             Session().commit()
         except Exception as e:
             log.exception("Exception during removal of field")
             msg = _('An error occurred during removal of field')
             h.flash(msg, category='error')
         return redirect(url('edit_repo_fields', repo_name=repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def edit_advanced(self, repo_name):
         """GET /repo_name/settings: Form to edit an existing item"""
         # url('edit_repo', repo_name=ID)
         c.repo_info = self._load_repo(repo_name)
         c.default_user_id = User.get_default_user().user_id
         c.in_public_journal = UserFollowing.query()\
             .filter(UserFollowing.user_id == c.default_user_id)\
             .filter(UserFollowing.follows_repository == c.repo_info).scalar()
 
         c.active = 'advanced'
         c.has_origin_repo_read_perm = False
         if c.repo_info.fork:
             c.has_origin_repo_read_perm = h.HasRepoPermissionAny(
                 'repository.write', 'repository.read', 'repository.admin')(
                 c.repo_info.fork.repo_name, 'repo set as fork page')
 
         if request.POST:
             return redirect(url('repo_edit_advanced'))
         return render('admin/repos/repo_edit.html')
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def edit_advanced_journal(self, repo_name):
         """
         Set's this repository to be visible in public journal,
         in other words assing default user to follow this repo
 
         :param repo_name:
         """
 
         try:
             repo_id = Repository.get_by_repo_name(repo_name).repo_id
             user_id = User.get_default_user().user_id
             self.scm_model.toggle_following_repo(repo_id, user_id)
             h.flash(_('Updated repository visibility in public journal'),
                     category='success')
             Session().commit()
         except Exception:
             h.flash(_('An error occurred during setting this'
                       ' repository in public journal'),
                     category='error')
 
         return redirect(url('edit_repo_advanced', repo_name=repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def edit_advanced_fork(self, repo_name):
         """
         Mark given repository as a fork of another
 
         :param repo_name:
         """
 
         new_fork_id = request.POST.get('id_fork_of')
         try:
 
             if new_fork_id and not new_fork_id.isdigit():
                 log.error('Given fork id %s is not an INT', new_fork_id)
 
             fork_id = safe_int(new_fork_id)
             repo = ScmModel().mark_as_fork(repo_name, fork_id,
                                            c.rhodecode_user.username)
             fork = repo.fork.repo_name if repo.fork else _('Nothing')
             Session().commit()
             h.flash(_('Marked repo %s as fork of %s') % (repo_name, fork),
                     category='success')
         except RepositoryError as e:
             log.exception("Repository Error occurred")
             h.flash(str(e), category='error')
         except Exception as e:
             log.exception("Exception while editing fork")
             h.flash(_('An error occurred during this operation'),
                     category='error')
 
         return redirect(url('edit_repo_advanced', repo_name=repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def edit_advanced_locking(self, repo_name):
         """
         Unlock repository when it is locked !
 
         :param repo_name:
         """
         try:
             repo = Repository.get_by_repo_name(repo_name)
             if request.POST.get('set_lock'):
                 Repository.lock(repo, c.rhodecode_user.user_id,
                                 lock_reason=Repository.LOCK_WEB)
                 h.flash(_('Locked repository'), category='success')
             elif request.POST.get('set_unlock'):
                 Repository.unlock(repo)
                 h.flash(_('Unlocked repository'), category='success')
         except Exception as e:
             log.exception("Exception during unlocking")
             h.flash(_('An error occurred during unlocking'),
                     category='error')
         return redirect(url('edit_repo_advanced', repo_name=repo_name))
 
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     @auth.CSRFRequired()
     def toggle_locking(self, repo_name):
         """
         Toggle locking of repository by simple GET call to url
 
         :param repo_name:
         """
 
         try:
             repo = Repository.get_by_repo_name(repo_name)
 
             if repo.enable_locking:
                 if repo.locked[0]:
                     Repository.unlock(repo)
                     action = _('Unlocked')
                 else:
                     Repository.lock(repo, c.rhodecode_user.user_id,
                                     lock_reason=Repository.LOCK_WEB)
                     action = _('Locked')
 
                 h.flash(_('Repository has been %s') % action,
                         category='success')
         except Exception:
             log.exception("Exception during unlocking")
             h.flash(_('An error occurred during unlocking'),
                     category='error')
         return redirect(url('summary_home', repo_name=repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def edit_caches(self, repo_name):
         """PUT /{repo_name}/settings/caches: invalidate the repo caches."""
         try:
             ScmModel().mark_for_invalidation(repo_name, delete=True)
             Session().commit()
             h.flash(_('Cache invalidation successful'),
                     category='success')
         except Exception:
             log.exception("Exception during cache invalidation")
             h.flash(_('An error occurred during cache invalidation'),
                     category='error')
 
         return redirect(url('edit_repo_caches', repo_name=c.repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def edit_caches_form(self, repo_name):
         """GET /repo_name/settings: Form to edit an existing item"""
         # url('edit_repo', repo_name=ID)
         c.repo_info = self._load_repo(repo_name)
         c.active = 'caches'
 
         return render('admin/repos/repo_edit.html')
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def edit_remote(self, repo_name):
         """PUT /{repo_name}/settings/remote: edit the repo remote."""
         try:
             ScmModel().pull_changes(repo_name, c.rhodecode_user.username)
             h.flash(_('Pulled from remote location'), category='success')
         except Exception:
             log.exception("Exception during pull from remote")
             h.flash(_('An error occurred during pull from remote location'),
                     category='error')
         return redirect(url('edit_repo_remote', repo_name=c.repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def edit_remote_form(self, repo_name):
         """GET /repo_name/settings: Form to edit an existing item"""
         # url('edit_repo', repo_name=ID)
         c.repo_info = self._load_repo(repo_name)
         c.active = 'remote'
 
         return render('admin/repos/repo_edit.html')
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def edit_statistics(self, repo_name):
         """PUT /{repo_name}/settings/statistics: reset the repo statistics."""
         try:
             RepoModel().delete_stats(repo_name)
             Session().commit()
         except Exception as e:
             log.error(traceback.format_exc())
             h.flash(_('An error occurred during deletion of repository stats'),
                     category='error')
         return redirect(url('edit_repo_statistics', repo_name=c.repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def edit_statistics_form(self, repo_name):
         """GET /repo_name/settings: Form to edit an existing item"""
         # url('edit_repo', repo_name=ID)
         c.repo_info = self._load_repo(repo_name)
         repo = c.repo_info.scm_instance()
 
         if c.repo_info.stats:
             # this is on what revision we ended up so we add +1 for count
             last_rev = c.repo_info.stats.stat_on_revision + 1
         else:
             last_rev = 0
         c.stats_revision = last_rev
 
         c.repo_last_rev = repo.count()
 
         if last_rev == 0 or c.repo_last_rev == 0:
             c.stats_percentage = 0
         else:
             c.stats_percentage = '%.2f' % ((float((last_rev)) / c.repo_last_rev) * 100)
 
         c.active = 'statistics'
 
         return render('admin/repos/repo_edit.html')
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def repo_issuetracker_test(self, repo_name):
         if request.is_xhr:
             return h.urlify_commit_message(
                 request.POST.get('test_text', ''),
                 repo_name)
         else:
             raise HTTPBadRequest()
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def repo_issuetracker_delete(self, repo_name):
         uid = request.POST.get('uid')
         repo_settings = IssueTrackerSettingsModel(repo=repo_name)
         try:
             repo_settings.delete_entries(uid)
         except Exception:
             h.flash(_('Error occurred during deleting issue tracker entry'),
                     category='error')
         else:
             h.flash(_('Removed issue tracker entry'), category='success')
         return redirect(url('repo_settings_issuetracker',
                         repo_name=repo_name))
 
     def _update_patterns(self, form, repo_settings):
         for uid in form['delete_patterns']:
             repo_settings.delete_entries(uid)
 
         for pattern in form['patterns']:
             for setting, value, type_ in pattern:
                 sett = repo_settings.create_or_update_setting(
                     setting, value, type_)
                 Session().add(sett)
 
             Session().commit()
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def repo_issuetracker_save(self, repo_name):
         # Save inheritance
         repo_settings = IssueTrackerSettingsModel(repo=repo_name)
         inherited = (request.POST.get('inherit_global_issuetracker')
                      == "inherited")
         repo_settings.inherit_global_settings = inherited
         Session().commit()
 
         form = IssueTrackerPatternsForm()().to_python(request.POST)
         if form:
             self._update_patterns(form, repo_settings)
 
         h.flash(_('Updated issue tracker entries'), category='success')
         return redirect(url('repo_settings_issuetracker',
                         repo_name=repo_name))
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def repo_issuetracker(self, repo_name):
         """GET /admin/settings/issue-tracker: All items in the collection"""
         c.active = 'issuetracker'
         c.data = 'data'
         c.repo_info = self._load_repo(repo_name)
 
         repo = Repository.get_by_repo_name(repo_name)
         c.settings_model = IssueTrackerSettingsModel(repo=repo)
         c.global_patterns = c.settings_model.get_global_settings()
         c.repo_patterns = c.settings_model.get_repo_settings()
 
         return render('admin/repos/repo_edit.html')
 
     @HasRepoPermissionAllDecorator('repository.admin')
     def repo_settings_vcs(self, repo_name):
         """GET /{repo_name}/settings/vcs/: All items in the collection"""
 
         model = VcsSettingsModel(repo=repo_name)
 
         c.active = 'vcs'
         c.global_svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.global_svn_tag_patterns = model.get_global_svn_tag_patterns()
         c.svn_branch_patterns = model.get_repo_svn_branch_patterns()
         c.svn_tag_patterns = model.get_repo_svn_tag_patterns()
         c.repo_info = self._load_repo(repo_name)
         defaults = self._vcs_form_defaults(repo_name)
         c.inherit_global_settings = defaults['inherit_global_settings']
 
         return htmlfill.render(
             render('admin/repos/repo_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     def repo_settings_vcs_update(self, repo_name):
         """POST /{repo_name}/settings/vcs/: All items in the collection"""
         c.active = 'vcs'
 
         model = VcsSettingsModel(repo=repo_name)
         c.global_svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.global_svn_tag_patterns = model.get_global_svn_tag_patterns()
         c.svn_branch_patterns = model.get_repo_svn_branch_patterns()
         c.svn_tag_patterns = model.get_repo_svn_tag_patterns()
         c.repo_info = self._load_repo(repo_name)
         defaults = self._vcs_form_defaults(repo_name)
         c.inherit_global_settings = defaults['inherit_global_settings']
 
         application_form = RepoVcsSettingsForm(repo_name)()
         try:
             form_result = application_form.to_python(dict(request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _("Some form inputs contain invalid data."),
                 category='error')
             return htmlfill.render(
                 render('admin/repos/repo_edit.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
 
         try:
             inherit_global_settings = form_result['inherit_global_settings']
             model.create_or_update_repo_settings(
                 form_result, inherit_global_settings=inherit_global_settings)
         except Exception:
             log.exception("Exception while updating settings")
             h.flash(
                 _('Error occurred during updating repository VCS settings'),
                 category='error')
         else:
             Session().commit()
             h.flash(_('Updated VCS settings'), category='success')
             return redirect(url('repo_vcs_settings', repo_name=repo_name))
 
         return htmlfill.render(
             render('admin/repos/repo_edit.html'),
             defaults=self._vcs_form_defaults(repo_name),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasRepoPermissionAllDecorator('repository.admin')
     @auth.CSRFRequired()
     @jsonify
     def repo_delete_svn_pattern(self, repo_name):
         if not request.is_xhr:
             return False
 
         delete_pattern_id = request.POST.get('delete_svn_pattern')
         model = VcsSettingsModel(repo=repo_name)
         try:
             model.delete_repo_svn_pattern(delete_pattern_id)
         except SettingNotFound:
             raise HTTPBadRequest()
 
         Session().commit()
         return True
 
     def _vcs_form_defaults(self, repo_name):
         model = VcsSettingsModel(repo=repo_name)
         global_defaults = model.get_global_settings()
 
         repo_defaults = {}
         repo_defaults.update(global_defaults)
         repo_defaults.update(model.get_repo_settings())
 
         global_defaults = {
             '{}_inherited'.format(k): global_defaults[k]
             for k in global_defaults}
 
         defaults = {
             'inherit_global_settings': model.inherit_global_settings
         }
         defaults.update(global_defaults)
         defaults.update(repo_defaults)
         defaults.update({
             'new_svn_branch': '',
             'new_svn_tag': '',
         })
         return defaults

rhodecode/controllers/admin/settings.py

@@ -1,866 +1,813 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 """
 settings controller for rhodecode admin
 """
 
 import collections
 import logging
 import urllib2
 
 import datetime
 import formencode
 from formencode import htmlfill
 import packaging.version
 from pylons import request, tmpl_context as c, url, config
 from pylons.controllers.util import redirect
 from pylons.i18n.translation import _, lazy_ugettext
 from webob.exc import HTTPBadRequest
 
 import rhodecode
+from rhodecode.admin.navigation import navigation_list
 from rhodecode.lib import auth
 from rhodecode.lib import helpers as h
 from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator
 from rhodecode.lib.base import BaseController, render
 from rhodecode.lib.celerylib import tasks, run_task
 from rhodecode.lib.utils import repo2db_mapper
 from rhodecode.lib.utils2 import (
     str2bool, safe_unicode, AttributeDict, safe_int)
 from rhodecode.lib.compat import OrderedDict
 from rhodecode.lib.ext_json import json
-from rhodecode.lib.utils import jsonify, read_opensource_licenses
+from rhodecode.lib.utils import jsonify
 
 from rhodecode.model.db import RhodeCodeUi, Repository
 from rhodecode.model.forms import ApplicationSettingsForm, \
     ApplicationUiSettingsForm, ApplicationVisualisationForm, \
     LabsSettingsForm, IssueTrackerPatternsForm
 
 from rhodecode.model.scm import ScmModel
 from rhodecode.model.notification import EmailNotificationModel
 from rhodecode.model.meta import Session
 from rhodecode.model.settings import (
     IssueTrackerSettingsModel, VcsSettingsModel, SettingNotFound,
     SettingsModel)
+
 from rhodecode.model.supervisor import SupervisorModel, SUPERVISOR_MASTER
-from rhodecode.model.user import UserModel
+
 
 log = logging.getLogger(__name__)
 
 
 class SettingsController(BaseController):
     """REST Controller styled on the Atom Publishing Protocol"""
     # To properly map this controller, ensure your config/routing.py
     # file has a resource setup:
     #     map.resource('setting', 'settings', controller='admin/settings',
     #         path_prefix='/admin', name_prefix='admin_')
 
     @LoginRequired()
     def __before__(self):
         super(SettingsController, self).__before__()
         c.labs_active = str2bool(
             rhodecode.CONFIG.get('labs_settings_active', 'false'))
-        c.navlist = navigation.get_navlist(request)
+        c.navlist = navigation_list(request)
 
     def _get_hg_ui_settings(self):
         ret = RhodeCodeUi.query().all()
 
         if not ret:
             raise Exception('Could not get application ui settings !')
         settings = {}
         for each in ret:
             k = each.ui_key
             v = each.ui_value
             if k == '/':
                 k = 'root_path'
 
             if k in ['push_ssl', 'publish']:
                 v = str2bool(v)
 
             if k.find('.') != -1:
                 k = k.replace('.', '_')
 
             if each.ui_section in ['hooks', 'extensions']:
                 v = each.ui_active
 
             settings[each.ui_section + '_' + k] = v
         return settings
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     @jsonify
     def delete_svn_pattern(self):
         if not request.is_xhr:
             raise HTTPBadRequest()
 
         delete_pattern_id = request.POST.get('delete_svn_pattern')
         model = VcsSettingsModel()
         try:
             model.delete_global_svn_pattern(delete_pattern_id)
         except SettingNotFound:
             raise HTTPBadRequest()
 
         Session().commit()
         return True
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_vcs_update(self):
         """POST /admin/settings: All items in the collection"""
         # url('admin_settings_vcs')
         c.active = 'vcs'
 
         model = VcsSettingsModel()
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
 
         application_form = ApplicationUiSettingsForm()()
         try:
             form_result = application_form.to_python(dict(request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _("Some form inputs contain invalid data."),
                 category='error')
             return htmlfill.render(
                 render('admin/settings/settings.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
 
         try:
             model.update_global_ssl_setting(form_result['web_push_ssl'])
             if c.visual.allow_repo_location_change:
                 model.update_global_path_setting(
                     form_result['paths_root_path'])
             model.update_global_hook_settings(form_result)
             model.create_global_svn_settings(form_result)
             model.create_or_update_global_hg_settings(form_result)
             model.create_or_update_global_pr_settings(form_result)
         except Exception:
             log.exception("Exception while updating settings")
             h.flash(_('Error occurred during updating '
                       'application settings'), category='error')
         else:
             Session().commit()
             h.flash(_('Updated VCS settings'), category='success')
             return redirect(url('admin_settings_vcs'))
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_vcs(self):
         """GET /admin/settings: All items in the collection"""
         # url('admin_settings_vcs')
         c.active = 'vcs'
         model = VcsSettingsModel()
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_mapping_update(self):
         """POST /admin/settings/mapping: All items in the collection"""
         # url('admin_settings_mapping')
         c.active = 'mapping'
         rm_obsolete = request.POST.get('destroy', False)
         invalidate_cache = request.POST.get('invalidate', False)
         log.debug(
             'rescanning repo location with destroy obsolete=%s', rm_obsolete)
 
         if invalidate_cache:
             log.debug('invalidating all repositories cache')
             for repo in Repository.get_all():
                 ScmModel().mark_for_invalidation(repo.repo_name, delete=True)
 
         filesystem_repos = ScmModel().repo_scan()
         added, removed = repo2db_mapper(filesystem_repos, rm_obsolete)
         _repr = lambda l: ', '.join(map(safe_unicode, l)) or '-'
         h.flash(_('Repositories successfully '
                   'rescanned added: %s ; removed: %s') %
                 (_repr(added), _repr(removed)),
                 category='success')
         return redirect(url('admin_settings_mapping'))
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_mapping(self):
         """GET /admin/settings/mapping: All items in the collection"""
         # url('admin_settings_mapping')
         c.active = 'mapping'
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_global_update(self):
         """POST /admin/settings/global: All items in the collection"""
         # url('admin_settings_global')
         c.active = 'global'
         application_form = ApplicationSettingsForm()()
         try:
             form_result = application_form.to_python(dict(request.POST))
         except formencode.Invalid as errors:
             return htmlfill.render(
                 render('admin/settings/settings.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
 
         try:
             settings = [
                 ('title', 'rhodecode_title'),
                 ('realm', 'rhodecode_realm'),
                 ('pre_code', 'rhodecode_pre_code'),
                 ('post_code', 'rhodecode_post_code'),
                 ('captcha_public_key', 'rhodecode_captcha_public_key'),
                 ('captcha_private_key', 'rhodecode_captcha_private_key'),
             ]
             for setting, form_key in settings:
                 sett = SettingsModel().create_or_update_setting(
                     setting, form_result[form_key])
                 Session().add(sett)
 
             Session().commit()
+            SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated application settings'), category='success')
-
         except Exception:
             log.exception("Exception while updating application settings")
             h.flash(
                 _('Error occurred during updating application settings'),
                 category='error')
 
         return redirect(url('admin_settings_global'))
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_global(self):
         """GET /admin/settings/global: All items in the collection"""
         # url('admin_settings_global')
         c.active = 'global'
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_visual_update(self):
         """POST /admin/settings/visual: All items in the collection"""
         # url('admin_settings_visual')
         c.active = 'visual'
         application_form = ApplicationVisualisationForm()()
         try:
             form_result = application_form.to_python(dict(request.POST))
         except formencode.Invalid as errors:
             return htmlfill.render(
                 render('admin/settings/settings.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
 
         try:
             settings = [
                 ('show_public_icon', 'rhodecode_show_public_icon', 'bool'),
                 ('show_private_icon', 'rhodecode_show_private_icon', 'bool'),
                 ('stylify_metatags', 'rhodecode_stylify_metatags', 'bool'),
                 ('repository_fields', 'rhodecode_repository_fields', 'bool'),
                 ('dashboard_items', 'rhodecode_dashboard_items', 'int'),
                 ('admin_grid_items', 'rhodecode_admin_grid_items', 'int'),
                 ('show_version', 'rhodecode_show_version', 'bool'),
                 ('use_gravatar', 'rhodecode_use_gravatar', 'bool'),
                 ('markup_renderer', 'rhodecode_markup_renderer', 'unicode'),
                 ('gravatar_url', 'rhodecode_gravatar_url', 'unicode'),
                 ('clone_uri_tmpl', 'rhodecode_clone_uri_tmpl', 'unicode'),
                 ('support_url', 'rhodecode_support_url', 'unicode'),
                 ('show_revision_number', 'rhodecode_show_revision_number', 'bool'),
                 ('show_sha_length', 'rhodecode_show_sha_length', 'int'),
             ]
             for setting, form_key, type_ in settings:
                 sett = SettingsModel().create_or_update_setting(
                     setting, form_result[form_key], type_)
                 Session().add(sett)
 
             Session().commit()
-
+            SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated visualisation settings'), category='success')
         except Exception:
             log.exception("Exception updating visualization settings")
             h.flash(_('Error occurred during updating '
                       'visualisation settings'),
                     category='error')
 
         return redirect(url('admin_settings_visual'))
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_visual(self):
         """GET /admin/settings/visual: All items in the collection"""
         # url('admin_settings_visual')
         c.active = 'visual'
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_issuetracker_test(self):
         if request.is_xhr:
             return h.urlify_commit_message(
                 request.POST.get('test_text', ''),
                 'repo_group/test_repo1')
         else:
             raise HTTPBadRequest()
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_issuetracker_delete(self):
         uid = request.POST.get('uid')
         IssueTrackerSettingsModel().delete_entries(uid)
         h.flash(_('Removed issue tracker entry'), category='success')
         return redirect(url('admin_settings_issuetracker'))
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_issuetracker(self):
         """GET /admin/settings/issue-tracker: All items in the collection"""
         # url('admin_settings_issuetracker')
         c.active = 'issuetracker'
         defaults = SettingsModel().get_all_settings()
 
         entry_key = 'rhodecode_issuetracker_pat_'
 
         c.issuetracker_entries = {}
         for k, v in defaults.items():
             if k.startswith(entry_key):
                 uid = k[len(entry_key):]
                 c.issuetracker_entries[uid] = None
 
         for uid in c.issuetracker_entries:
             c.issuetracker_entries[uid] = AttributeDict({
                 'pat': defaults.get('rhodecode_issuetracker_pat_' + uid),
                 'url': defaults.get('rhodecode_issuetracker_url_' + uid),
                 'pref': defaults.get('rhodecode_issuetracker_pref_' + uid),
                 'desc': defaults.get('rhodecode_issuetracker_desc_' + uid),
             })
 
         return render('admin/settings/settings.html')
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_issuetracker_save(self):
         settings_model = IssueTrackerSettingsModel()
 
         form = IssueTrackerPatternsForm()().to_python(request.POST)
         for uid in form['delete_patterns']:
             settings_model.delete_entries(uid)
 
         for pattern in form['patterns']:
             for setting, value, type_ in pattern:
                 sett = settings_model.create_or_update_setting(
                     setting, value, type_)
                 Session().add(sett)
 
             Session().commit()
 
+        SettingsModel().invalidate_settings_cache()
         h.flash(_('Updated issue tracker entries'), category='success')
         return redirect(url('admin_settings_issuetracker'))
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_email_update(self):
         """POST /admin/settings/email: All items in the collection"""
         # url('admin_settings_email')
         c.active = 'email'
 
         test_email = request.POST.get('test_email')
 
         if not test_email:
             h.flash(_('Please enter email address'), category='error')
             return redirect(url('admin_settings_email'))
 
         email_kwargs = {
             'date': datetime.datetime.now(),
             'user': c.rhodecode_user,
             'rhodecode_version': c.rhodecode_version
         }
 
         (subject, headers, email_body,
          email_body_plaintext) = EmailNotificationModel().render_email(
             EmailNotificationModel.TYPE_EMAIL_TEST, **email_kwargs)
 
         recipients = [test_email] if test_email else None
 
         run_task(tasks.send_email, recipients, subject,
                  email_body_plaintext, email_body)
 
         h.flash(_('Send email task created'), category='success')
         return redirect(url('admin_settings_email'))
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_email(self):
         """GET /admin/settings/email: All items in the collection"""
         # url('admin_settings_email')
         c.active = 'email'
         c.rhodecode_ini = rhodecode.CONFIG
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_hooks_update(self):
         """POST or DELETE /admin/settings/hooks: All items in the collection"""
         # url('admin_settings_hooks')
         c.active = 'hooks'
         if c.visual.allow_custom_hooks_settings:
             ui_key = request.POST.get('new_hook_ui_key')
             ui_value = request.POST.get('new_hook_ui_value')
 
             hook_id = request.POST.get('hook_id')
             new_hook = False
 
             model = SettingsModel()
             try:
                 if ui_value and ui_key:
                     model.create_or_update_hook(ui_key, ui_value)
                     h.flash(_('Added new hook'), category='success')
                     new_hook = True
                 elif hook_id:
                     RhodeCodeUi.delete(hook_id)
                     Session().commit()
 
                 # check for edits
                 update = False
                 _d = request.POST.dict_of_lists()
                 for k, v in zip(_d.get('hook_ui_key', []),
                                 _d.get('hook_ui_value_new', [])):
                     model.create_or_update_hook(k, v)
                     update = True
 
                 if update and not new_hook:
                     h.flash(_('Updated hooks'), category='success')
                 Session().commit()
             except Exception:
                 log.exception("Exception during hook creation")
                 h.flash(_('Error occurred during hook creation'),
                         category='error')
 
         return redirect(url('admin_settings_hooks'))
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_hooks(self):
         """GET /admin/settings/hooks: All items in the collection"""
         # url('admin_settings_hooks')
         c.active = 'hooks'
 
         model = SettingsModel()
         c.hooks = model.get_builtin_hooks()
         c.custom_hooks = model.get_custom_hooks()
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_search(self):
         """GET /admin/settings/search: All items in the collection"""
         # url('admin_settings_search')
         c.active = 'search'
 
         from rhodecode.lib.index import searcher_from_config
         searcher = searcher_from_config(config)
         c.statistics = searcher.statistics()
 
         return render('admin/settings/settings.html')
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_system(self):
         """GET /admin/settings/system: All items in the collection"""
         # url('admin_settings_system')
+        snapshot = str2bool(request.GET.get('snapshot'))
         c.active = 'system'
 
         defaults = self._form_defaults()
         c.rhodecode_ini = rhodecode.CONFIG
         c.rhodecode_update_url = defaults.get('rhodecode_update_url')
         server_info = ScmModel().get_server_info(request.environ)
         for key, val in server_info.iteritems():
             setattr(c, key, val)
 
         if c.disk['percent'] > 90:
             h.flash(h.literal(_(
                 'Critical: your disk space is very low <b>%s%%</b> used' %
                 c.disk['percent'])), 'error')
         elif c.disk['percent'] > 70:
             h.flash(h.literal(_(
                 'Warning: your disk space is running low <b>%s%%</b> used' %
                 c.disk['percent'])), 'warning')
 
         try:
             c.uptime_age = h._age(
                 h.time_to_datetime(c.boot_time), False, show_suffix=False)
         except TypeError:
             c.uptime_age = c.boot_time
 
         try:
             c.system_memory = '%s/%s, %s%% (%s%%) used%s' % (
                 h.format_byte_size_binary(c.memory['used']),
                 h.format_byte_size_binary(c.memory['total']),
                 c.memory['percent2'],
                 c.memory['percent'],
                 ' %s' % c.memory['error'] if 'error' in c.memory else '')
         except TypeError:
             c.system_memory = 'NOT AVAILABLE'
 
+        rhodecode_ini_safe = rhodecode.CONFIG.copy()
+        blacklist = [
+            'rhodecode_license_key',
+            'routes.map',
+            'pylons.h',
+            'pylons.app_globals',
+            'pylons.environ_config',
+            'sqlalchemy.db1.url',
+            ('app_conf', 'sqlalchemy.db1.url')
+        ]
+        for k in blacklist:
+            if isinstance(k, tuple):
+                section, key = k
+                if section in rhodecode_ini_safe:
+                    rhodecode_ini_safe[section].pop(key, None)
+            else:
+                rhodecode_ini_safe.pop(k, None)
+
+        c.rhodecode_ini_safe = rhodecode_ini_safe
+
+        # TODO: marcink, figure out how to allow only selected users to do this
+        c.allowed_to_snapshot = False
+
+        if snapshot:
+            if c.allowed_to_snapshot:
+                return render('admin/settings/settings_system_snapshot.html')
+            else:
+                h.flash('You are not allowed to do this', category='warning')
+
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @staticmethod
     def get_update_data(update_url):
         """Return the JSON update data."""
         ver = rhodecode.__version__
         log.debug('Checking for upgrade on `%s` server', update_url)
         opener = urllib2.build_opener()
         opener.addheaders = [('User-agent', 'RhodeCode-SCM/%s' % ver)]
         response = opener.open(update_url)
         response_data = response.read()
         data = json.loads(response_data)
 
         return data
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_system_update(self):
         """GET /admin/settings/system/updates: All items in the collection"""
         # url('admin_settings_system_update')
         defaults = self._form_defaults()
         update_url = defaults.get('rhodecode_update_url', '')
 
         _err = lambda s: '<div style="color:#ff8888; padding:4px 0px">%s</div>' % (s)
         try:
             data = self.get_update_data(update_url)
         except urllib2.URLError as e:
             log.exception("Exception contacting upgrade server")
             return _err('Failed to contact upgrade server: %r' % e)
         except ValueError as e:
             log.exception("Bad data sent from update server")
             return _err('Bad data sent from update server')
 
         latest = data['versions'][0]
 
         c.update_url = update_url
         c.latest_data = latest
         c.latest_ver = latest['version']
         c.cur_ver = rhodecode.__version__
         c.should_upgrade = False
 
         if (packaging.version.Version(c.latest_ver) >
                 packaging.version.Version(c.cur_ver)):
             c.should_upgrade = True
         c.important_notices = latest['general']
 
         return render('admin/settings/settings_system_update.html')
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_supervisor(self):
         c.rhodecode_ini = rhodecode.CONFIG
         c.active = 'supervisor'
 
         c.supervisor_procs = OrderedDict([
             (SUPERVISOR_MASTER, {}),
         ])
 
         c.log_size = 10240
         supervisor = SupervisorModel()
 
         _connection = supervisor.get_connection(
             c.rhodecode_ini.get('supervisor.uri'))
         c.connection_error = None
         try:
             _connection.supervisor.getAllProcessInfo()
         except Exception as e:
             c.connection_error = str(e)
             log.exception("Exception reading supervisor data")
             return render('admin/settings/settings.html')
 
         groupid = c.rhodecode_ini.get('supervisor.group_id')
 
         # feed our group processes to the main
         for proc in supervisor.get_group_processes(_connection, groupid):
             c.supervisor_procs[proc['name']] = {}
 
         for k in c.supervisor_procs.keys():
             try:
                 # master process info
                 if k == SUPERVISOR_MASTER:
                     _data = supervisor.get_master_state(_connection)
                     _data['name'] = 'supervisor master'
                     _data['description'] = 'pid %s, id: %s, ver: %s' % (
                         _data['pid'], _data['id'], _data['ver'])
                     c.supervisor_procs[k] = _data
                 else:
                     procid = groupid + ":" + k
                     c.supervisor_procs[k] = supervisor.get_process_info(_connection, procid)
             except Exception as e:
                 log.exception("Exception reading supervisor data")
                 c.supervisor_procs[k] = {'_rhodecode_error': str(e)}
 
         return render('admin/settings/settings.html')
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_supervisor_log(self, procid):
         import rhodecode
         c.rhodecode_ini = rhodecode.CONFIG
         c.active = 'supervisor_tail'
 
         supervisor = SupervisorModel()
         _connection = supervisor.get_connection(c.rhodecode_ini.get('supervisor.uri'))
         groupid = c.rhodecode_ini.get('supervisor.group_id')
         procid = groupid + ":" + procid if procid != SUPERVISOR_MASTER else procid
 
         c.log_size = 10240
         offset = abs(safe_int(request.GET.get('offset', c.log_size))) * -1
         c.log = supervisor.read_process_log(_connection, procid, offset, 0)
 
         return render('admin/settings/settings.html')
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def settings_labs_update(self):
         """POST /admin/settings/labs: All items in the collection"""
         # url('admin_settings/labs', method={'POST'})
         c.active = 'labs'
 
         application_form = LabsSettingsForm()()
         try:
             form_result = application_form.to_python(dict(request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _('Some form inputs contain invalid data.'),
                 category='error')
             return htmlfill.render(
                 render('admin/settings/settings.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding='UTF-8',
                 force_defaults=False
             )
 
         try:
             session = Session()
             for setting in _LAB_SETTINGS:
                 setting_name = setting.key[len('rhodecode_'):]
                 sett = SettingsModel().create_or_update_setting(
                     setting_name, form_result[setting.key], setting.type)
                 session.add(sett)
 
         except Exception:
             log.exception('Exception while updating lab settings')
             h.flash(_('Error occurred during updating labs settings'),
                     category='error')
         else:
             Session().commit()
+            SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated Labs settings'), category='success')
             return redirect(url('admin_settings_labs'))
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding='UTF-8',
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     def settings_labs(self):
         """GET /admin/settings/labs: All items in the collection"""
         # url('admin_settings_labs')
         if not c.labs_active:
             redirect(url('admin_settings'))
 
         c.active = 'labs'
         c.lab_settings = _LAB_SETTINGS
 
         return htmlfill.render(
             render('admin/settings/settings.html'),
             defaults=self._form_defaults(),
             encoding='UTF-8',
             force_defaults=False)
 
-    @HasPermissionAllDecorator('hg.admin')
-    def settings_open_source(self):
-        # url('admin_settings_open_source')
-
-        c.active = 'open_source'
-        c.opensource_licenses = collections.OrderedDict(
-            sorted(read_opensource_licenses().items(), key=lambda t: t[0]))
-
-        return htmlfill.render(
-            render('admin/settings/settings.html'),
-            defaults=self._form_defaults(),
-            encoding='UTF-8',
-            force_defaults=False)
-
     def _form_defaults(self):
         defaults = SettingsModel().get_all_settings()
         defaults.update(self._get_hg_ui_settings())
         defaults.update({
             'new_svn_branch': '',
             'new_svn_tag': '',
         })
         return defaults
 
 
 # :param key: name of the setting including the 'rhodecode_' prefix
 # :param type: the RhodeCodeSetting type to use.
 # :param group: the i18ned group in which we should dispaly this setting
 # :param label: the i18ned label we should display for this setting
 # :param help: the i18ned help we should dispaly for this setting
 LabSetting = collections.namedtuple(
     'LabSetting', ('key', 'type', 'group', 'label', 'help'))
 
 
 # This list has to be kept in sync with the form
 # rhodecode.model.forms.LabsSettingsForm.
 _LAB_SETTINGS = [
     LabSetting(
         key='rhodecode_hg_use_rebase_for_merging',
         type='bool',
         group=lazy_ugettext('Mercurial server-side merge'),
         label=lazy_ugettext('Use rebase instead of creating a merge commit when merging via web interface'),
         help=''  # Do not translate the empty string!
     ),
     LabSetting(
         key='rhodecode_proxy_subversion_http_requests',
         type='bool',
         group=lazy_ugettext('Subversion HTTP Support'),
         label=lazy_ugettext('Proxy subversion HTTP requests'),
         help=''  # Do not translate the empty string!
     ),
     LabSetting(
         key='rhodecode_subversion_http_server_url',
         type='str',
         group=lazy_ugettext('Subversion HTTP Server URL'),
         label='',  # Do not translate the empty string!
         help=lazy_ugettext('e.g. http://localhost:8080/')
     ),
 ]
-
-
-NavListEntry = collections.namedtuple('NavListEntry', ['key', 'name', 'url'])
-
-
-class NavEntry(object):
-
-    def __init__(self, key, name, view_name, pyramid=False):
-        self.key = key
-        self.name = name
-        self.view_name = view_name
-        self.pyramid = pyramid
-
-    def generate_url(self, request):
-        if self.pyramid:
-            if hasattr(request, 'route_path'):
-                return request.route_path(self.view_name)
-            else:
-                # TODO: johbo: Remove this after migrating to pyramid.
-                # We need the pyramid request here to generate URLs to pyramid
-                # views from within pylons views.
-                from pyramid.threadlocal import get_current_request
-                pyramid_request = get_current_request()
-                return pyramid_request.route_path(self.view_name)
-        else:
-            return url(self.view_name)
-
-
-class NavigationRegistry(object):
-
-    _base_entries = [
-        NavEntry('global', lazy_ugettext('Global'), 'admin_settings_global'),
-        NavEntry('vcs', lazy_ugettext('VCS'), 'admin_settings_vcs'),
-        NavEntry('visual', lazy_ugettext('Visual'), 'admin_settings_visual'),
-        NavEntry('mapping', lazy_ugettext('Remap and Rescan'),
-                 'admin_settings_mapping'),
-        NavEntry('issuetracker', lazy_ugettext('Issue Tracker'),
-                 'admin_settings_issuetracker'),
-        NavEntry('email', lazy_ugettext('Email'), 'admin_settings_email'),
-        NavEntry('hooks', lazy_ugettext('Hooks'), 'admin_settings_hooks'),
-        NavEntry('search', lazy_ugettext('Full Text Search'),
-                 'admin_settings_search'),
-        NavEntry('system', lazy_ugettext('System Info'),
-                 'admin_settings_system'),
-        NavEntry('open_source', lazy_ugettext('Open Source Licenses'),
-                 'admin_settings_open_source'),
-        # TODO: marcink: we disable supervisor now until the supervisor stats
-        # page is fixed in the nix configuration
-        # NavEntry('supervisor', lazy_ugettext('Supervisor'),
-        #          'admin_settings_supervisor'),
-    ]
-
-    def __init__(self):
-        self._registered_entries = collections.OrderedDict([
-            (item.key, item) for item in self.__class__._base_entries
-        ])
-
-        # Add the labs entry when it's activated.
-        labs_active = str2bool(
-            rhodecode.CONFIG.get('labs_settings_active', 'false'))
-        if labs_active:
-            self.add_entry(
-                NavEntry('labs', lazy_ugettext('Labs'), 'admin_settings_labs'))
-
-    def add_entry(self, entry):
-        self._registered_entries[entry.key] = entry
-
-    def get_navlist(self, request):
-        navlist = [NavListEntry(i.key, i.name, i.generate_url(request))
-                   for i in self._registered_entries.values()]
-        return navlist
-
-navigation = NavigationRegistry()

rhodecode/controllers/admin/user_groups.py

@@ -1,480 +1,480 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2011-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 User Groups crud controller for pylons
 """
 
 import logging
 import formencode
 
 from formencode import htmlfill
 from pylons import request, tmpl_context as c, url, config
 from pylons.controllers.util import redirect
 from pylons.i18n.translation import _
 
 from sqlalchemy.orm import joinedload
 
 from rhodecode.lib import auth
 from rhodecode.lib import helpers as h
 from rhodecode.lib.exceptions import UserGroupAssignedException,\
     RepoGroupAssignmentError
 from rhodecode.lib.utils import jsonify, action_logger
 from rhodecode.lib.utils2 import safe_unicode, str2bool, safe_int
 from rhodecode.lib.auth import (
     LoginRequired, NotAnonymous, HasUserGroupPermissionAnyDecorator,
     HasPermissionAnyDecorator)
 from rhodecode.lib.base import BaseController, render
 from rhodecode.model.permission import PermissionModel
 from rhodecode.model.scm import UserGroupList
 from rhodecode.model.user_group import UserGroupModel
 from rhodecode.model.db import (
     User, UserGroup, UserGroupRepoToPerm, UserGroupRepoGroupToPerm)
 from rhodecode.model.forms import (
     UserGroupForm, UserGroupPermsForm, UserIndividualPermissionsForm,
     UserPermissionsForm)
 from rhodecode.model.meta import Session
 from rhodecode.lib.utils import action_logger
 from rhodecode.lib.ext_json import json
 
 log = logging.getLogger(__name__)
 
 
 class UserGroupsController(BaseController):
     """REST Controller styled on the Atom Publishing Protocol"""
 
     @LoginRequired()
     def __before__(self):
         super(UserGroupsController, self).__before__()
         c.available_permissions = config['available_permissions']
         PermissionModel().set_global_permission_choices(c, translator=_)
 
     def __load_data(self, user_group_id):
         c.group_members_obj = [x.user for x in c.user_group.members]
         c.group_members_obj.sort(key=lambda u: u.username.lower())
 
         c.group_members = [(x.user_id, x.username) for x in c.group_members_obj]
 
         c.available_members = [(x.user_id, x.username)
                                for x in User.query().all()]
         c.available_members.sort(key=lambda u: u[1].lower())
 
     def __load_defaults(self, user_group_id):
         """
         Load defaults settings for edit, and update
 
         :param user_group_id:
         """
         user_group = UserGroup.get_or_404(user_group_id)
         data = user_group.get_dict()
         # fill owner
         if user_group.user:
             data.update({'user': user_group.user.username})
         else:
-            replacement_user = User.get_first_admin().username
+            replacement_user = User.get_first_super_admin().username
             data.update({'user': replacement_user})
         return data
 
     def _revoke_perms_on_yourself(self, form_result):
         _updates = filter(lambda u: c.rhodecode_user.user_id == int(u[0]),
                           form_result['perm_updates'])
         _additions = filter(lambda u: c.rhodecode_user.user_id == int(u[0]),
                             form_result['perm_additions'])
         _deletions = filter(lambda u: c.rhodecode_user.user_id == int(u[0]),
                             form_result['perm_deletions'])
         admin_perm = 'usergroup.admin'
         if _updates   and _updates[0][1]   != admin_perm or \
            _additions and _additions[0][1] != admin_perm or \
            _deletions and _deletions[0][1] != admin_perm:
             return True
         return False
 
     # permission check inside
     @NotAnonymous()
     def index(self):
         """GET /users_groups: All items in the collection"""
         # url('users_groups')
 
         from rhodecode.lib.utils import PartialRenderer
         _render = PartialRenderer('data_table/_dt_elements.html')
 
         def user_group_name(user_group_id, user_group_name):
             return _render("user_group_name", user_group_id, user_group_name)
 
         def user_group_actions(user_group_id, user_group_name):
             return _render("user_group_actions", user_group_id, user_group_name)
 
         ## json generate
         group_iter = UserGroupList(UserGroup.query().all(),
                                    perm_set=['usergroup.admin'])
 
         user_groups_data = []
         for user_gr in group_iter:
             user_groups_data.append({
                 "group_name": user_group_name(
                     user_gr.users_group_id, h.escape(user_gr.users_group_name)),
                 "group_name_raw": user_gr.users_group_name,
                 "desc": h.escape(user_gr.user_group_description),
                 "members": len(user_gr.members),
                 "active": h.bool2icon(user_gr.users_group_active),
                 "owner": h.escape(h.link_to_user(user_gr.user.username)),
                 "action": user_group_actions(
                     user_gr.users_group_id, user_gr.users_group_name)
             })
 
         c.data = json.dumps(user_groups_data)
         return render('admin/user_groups/user_groups.html')
 
     @HasPermissionAnyDecorator('hg.admin', 'hg.usergroup.create.true')
     @auth.CSRFRequired()
     def create(self):
         """POST /users_groups: Create a new item"""
         # url('users_groups')
 
         users_group_form = UserGroupForm()()
         try:
             form_result = users_group_form.to_python(dict(request.POST))
             user_group = UserGroupModel().create(
                 name=form_result['users_group_name'],
                 description=form_result['user_group_description'],
                 owner=c.rhodecode_user.user_id,
                 active=form_result['users_group_active'])
             Session().flush()
 
             user_group_name = form_result['users_group_name']
             action_logger(c.rhodecode_user,
                           'admin_created_users_group:%s' % user_group_name,
                           None, self.ip_addr, self.sa)
             user_group_link = h.link_to(h.escape(user_group_name),
                                         url('edit_users_group',
                                             user_group_id=user_group.users_group_id))
             h.flash(h.literal(_('Created user group %(user_group_link)s')
                               % {'user_group_link': user_group_link}),
                     category='success')
             Session().commit()
         except formencode.Invalid as errors:
             return htmlfill.render(
                 render('admin/user_groups/user_group_add.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
         except Exception:
             log.exception("Exception creating user group")
             h.flash(_('Error occurred during creation of user group %s') \
                     % request.POST.get('users_group_name'), category='error')
 
         return redirect(
             url('edit_users_group', user_group_id=user_group.users_group_id))
 
     @HasPermissionAnyDecorator('hg.admin', 'hg.usergroup.create.true')
     def new(self):
         """GET /user_groups/new: Form to create a new item"""
         # url('new_users_group')
         return render('admin/user_groups/user_group_add.html')
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     @auth.CSRFRequired()
     def update(self, user_group_id):
         """PUT /user_groups/user_group_id: Update an existing item"""
         # Forms posted to this method should contain a hidden field:
         #    <input type="hidden" name="_method" value="PUT" />
         # Or using helpers:
         #    h.form(url('users_group', user_group_id=ID),
         #           method='put')
         # url('users_group', user_group_id=ID)
 
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'settings'
         self.__load_data(user_group_id)
 
         available_members = [safe_unicode(x[0]) for x in c.available_members]
 
-        users_group_form = UserGroupForm(edit=True,
-                                         old_data=c.user_group.get_dict(),
-                                         available_members=available_members)()
+        users_group_form = UserGroupForm(
+            edit=True, old_data=c.user_group.get_dict(),
+            available_members=available_members, allow_disabled=True)()
 
         try:
             form_result = users_group_form.to_python(request.POST)
             UserGroupModel().update(c.user_group, form_result)
             gr = form_result['users_group_name']
             action_logger(c.rhodecode_user,
                           'admin_updated_users_group:%s' % gr,
                           None, self.ip_addr, self.sa)
             h.flash(_('Updated user group %s') % gr, category='success')
             Session().commit()
         except formencode.Invalid as errors:
             defaults = errors.value
             e = errors.error_dict or {}
 
             return htmlfill.render(
                 render('admin/user_groups/user_group_edit.html'),
                 defaults=defaults,
                 errors=e,
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
         except Exception:
             log.exception("Exception during update of user group")
             h.flash(_('Error occurred during update of user group %s')
                     % request.POST.get('users_group_name'), category='error')
 
         return redirect(url('edit_users_group', user_group_id=user_group_id))
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     @auth.CSRFRequired()
     def delete(self, user_group_id):
         """DELETE /user_groups/user_group_id: Delete an existing item"""
         # Forms posted to this method should contain a hidden field:
         #    <input type="hidden" name="_method" value="DELETE" />
         # Or using helpers:
         #    h.form(url('users_group', user_group_id=ID),
         #           method='delete')
         # url('users_group', user_group_id=ID)
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         force = str2bool(request.POST.get('force'))
 
         try:
             UserGroupModel().delete(c.user_group, force=force)
             Session().commit()
             h.flash(_('Successfully deleted user group'), category='success')
         except UserGroupAssignedException as e:
             h.flash(str(e), category='error')
         except Exception:
             log.exception("Exception during deletion of user group")
             h.flash(_('An error occurred during deletion of user group'),
                     category='error')
         return redirect(url('users_groups'))
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     def edit(self, user_group_id):
         """GET /user_groups/user_group_id/edit: Form to edit an existing item"""
         # url('edit_users_group', user_group_id=ID)
 
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'settings'
         self.__load_data(user_group_id)
 
         defaults = self.__load_defaults(user_group_id)
 
         return htmlfill.render(
             render('admin/user_groups/user_group_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False
         )
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     def edit_perms(self, user_group_id):
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'perms'
 
         defaults = {}
         # fill user group users
         for p in c.user_group.user_user_group_to_perm:
             defaults.update({'u_perm_%s' % p.user.user_id:
                              p.permission.permission_name})
 
         for p in c.user_group.user_group_user_group_to_perm:
             defaults.update({'g_perm_%s' % p.user_group.users_group_id:
                              p.permission.permission_name})
 
         return htmlfill.render(
             render('admin/user_groups/user_group_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False
         )
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     @auth.CSRFRequired()
     def update_perms(self, user_group_id):
         """
         grant permission for given usergroup
 
         :param user_group_id:
         """
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         form = UserGroupPermsForm()().to_python(request.POST)
 
         if not c.rhodecode_user.is_admin:
             if self._revoke_perms_on_yourself(form):
                 msg = _('Cannot change permission for yourself as admin')
                 h.flash(msg, category='warning')
                 return redirect(url('edit_user_group_perms', user_group_id=user_group_id))
 
         try:
             UserGroupModel().update_permissions(user_group_id,
                 form['perm_additions'], form['perm_updates'], form['perm_deletions'])
         except RepoGroupAssignmentError:
             h.flash(_('Target group cannot be the same'), category='error')
             return redirect(url('edit_user_group_perms', user_group_id=user_group_id))
         #TODO: implement this
         #action_logger(c.rhodecode_user, 'admin_changed_repo_permissions',
         #              repo_name, self.ip_addr, self.sa)
         Session().commit()
         h.flash(_('User Group permissions updated'), category='success')
         return redirect(url('edit_user_group_perms', user_group_id=user_group_id))
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     def edit_perms_summary(self, user_group_id):
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'perms_summary'
         permissions = {
             'repositories': {},
             'repositories_groups': {},
         }
         ugroup_repo_perms = UserGroupRepoToPerm.query()\
             .options(joinedload(UserGroupRepoToPerm.permission))\
             .options(joinedload(UserGroupRepoToPerm.repository))\
             .filter(UserGroupRepoToPerm.users_group_id == user_group_id)\
             .all()
 
         for gr in ugroup_repo_perms:
             permissions['repositories'][gr.repository.repo_name]  \
                 = gr.permission.permission_name
 
         ugroup_group_perms = UserGroupRepoGroupToPerm.query()\
             .options(joinedload(UserGroupRepoGroupToPerm.permission))\
             .options(joinedload(UserGroupRepoGroupToPerm.group))\
             .filter(UserGroupRepoGroupToPerm.users_group_id == user_group_id)\
             .all()
 
         for gr in ugroup_group_perms:
             permissions['repositories_groups'][gr.group.group_name] \
                 = gr.permission.permission_name
         c.permissions = permissions
         return render('admin/user_groups/user_group_edit.html')
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     def edit_global_perms(self, user_group_id):
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'global_perms'
 
         c.default_user = User.get_default_user()
         defaults = c.user_group.get_dict()
         defaults.update(c.default_user.get_default_perms(suffix='_inherited'))
         defaults.update(c.user_group.get_default_perms())
 
         return htmlfill.render(
             render('admin/user_groups/user_group_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False
         )
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     @auth.CSRFRequired()
     def update_global_perms(self, user_group_id):
         """PUT /users_perm/user_group_id: Update an existing item"""
         # url('users_group_perm', user_group_id=ID, method='put')
         user_group_id = safe_int(user_group_id)
         user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'global_perms'
 
         try:
             # first stage that verifies the checkbox
             _form = UserIndividualPermissionsForm()
             form_result = _form.to_python(dict(request.POST))
             inherit_perms = form_result['inherit_default_permissions']
             user_group.inherit_default_permissions = inherit_perms
             Session().add(user_group)
 
             if not inherit_perms:
                 # only update the individual ones if we un check the flag
                 _form = UserPermissionsForm(
                     [x[0] for x in c.repo_create_choices],
                     [x[0] for x in c.repo_create_on_write_choices],
                     [x[0] for x in c.repo_group_create_choices],
                     [x[0] for x in c.user_group_create_choices],
                     [x[0] for x in c.fork_choices],
                     [x[0] for x in c.inherit_default_permission_choices])()
 
                 form_result = _form.to_python(dict(request.POST))
                 form_result.update({'perm_user_group_id': user_group.users_group_id})
 
                 PermissionModel().update_user_group_permissions(form_result)
 
             Session().commit()
             h.flash(_('User Group global permissions updated successfully'),
                     category='success')
 
         except formencode.Invalid as errors:
             defaults = errors.value
             c.user_group = user_group
             return htmlfill.render(
                 render('admin/user_groups/user_group_edit.html'),
                 defaults=defaults,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
 
         except Exception:
             log.exception("Exception during permissions saving")
             h.flash(_('An error occurred during permissions saving'),
                     category='error')
 
         return redirect(url('edit_user_group_global_perms', user_group_id=user_group_id))
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     def edit_advanced(self, user_group_id):
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'advanced'
         c.group_members_obj = sorted(
             (x.user for x in c.user_group.members),
             key=lambda u: u.username.lower())
 
         c.group_to_repos = sorted(
             (x.repository for x in c.user_group.users_group_repo_to_perm),
             key=lambda u: u.repo_name.lower())
 
         c.group_to_repo_groups = sorted(
             (x.group for x in c.user_group.users_group_repo_group_to_perm),
             key=lambda u: u.group_name.lower())
 
         return render('admin/user_groups/user_group_edit.html')
 
     @HasUserGroupPermissionAnyDecorator('usergroup.admin')
     def edit_members(self, user_group_id):
         user_group_id = safe_int(user_group_id)
         c.user_group = UserGroup.get_or_404(user_group_id)
         c.active = 'members'
         c.group_members_obj = sorted((x.user for x in c.user_group.members),
                                      key=lambda u: u.username.lower())
 
         group_members = [(x.user_id, x.username) for x in c.group_members_obj]
 
         if request.is_xhr:
             return jsonify(lambda *a, **k: {
                 'members': group_members
             })
 
         c.group_members = group_members
         return render('admin/user_groups/user_group_edit.html')

rhodecode/controllers/admin/users.py

@@ -1,717 +1,719 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 Users crud controller for pylons
 """
 
 import logging
 import formencode
 
 from formencode import htmlfill
 from pylons import request, tmpl_context as c, url, config
 from pylons.controllers.util import redirect
 from pylons.i18n.translation import _
 
 from rhodecode.authentication.plugins import auth_rhodecode
 from rhodecode.lib.exceptions import (
     DefaultUserException, UserOwnsReposException, UserOwnsRepoGroupsException,
     UserOwnsUserGroupsException, UserCreationError)
 from rhodecode.lib import helpers as h
 from rhodecode.lib import auth
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator, AuthUser, generate_auth_token)
 from rhodecode.lib.base import BaseController, render
 from rhodecode.model.auth_token import AuthTokenModel
 
 from rhodecode.model.db import (
     PullRequestReviewers, User, UserEmailMap, UserIpMap, RepoGroup)
 from rhodecode.model.forms import (
     UserForm, UserPermissionsForm, UserIndividualPermissionsForm)
 from rhodecode.model.user import UserModel
 from rhodecode.model.meta import Session
 from rhodecode.model.permission import PermissionModel
 from rhodecode.lib.utils import action_logger
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.utils2 import datetime_to_time, safe_int
 
 log = logging.getLogger(__name__)
 
 
 class UsersController(BaseController):
     """REST Controller styled on the Atom Publishing Protocol"""
 
     @LoginRequired()
     def __before__(self):
         super(UsersController, self).__before__()
         c.available_permissions = config['available_permissions']
         c.allowed_languages = [
             ('en', 'English (en)'),
             ('de', 'German (de)'),
             ('fr', 'French (fr)'),
             ('it', 'Italian (it)'),
             ('ja', 'Japanese (ja)'),
             ('pl', 'Polish (pl)'),
             ('pt', 'Portuguese (pt)'),
             ('ru', 'Russian (ru)'),
             ('zh', 'Chinese (zh)'),
         ]
         PermissionModel().set_global_permission_choices(c, translator=_)
 
     @HasPermissionAllDecorator('hg.admin')
     def index(self):
         """GET /users: All items in the collection"""
         # url('users')
 
         from rhodecode.lib.utils import PartialRenderer
         _render = PartialRenderer('data_table/_dt_elements.html')
 
         def grav_tmpl(user_email, size):
             return _render("user_gravatar", user_email, size)
 
         def username(user_id, username):
             return _render("user_name", user_id, username)
 
         def user_actions(user_id, username):
             return _render("user_actions", user_id, username)
 
         # json generate
         c.users_list = User.query()\
             .filter(User.username != User.DEFAULT_USER) \
             .all()
 
         users_data = []
         for user in c.users_list:
             users_data.append({
                 "gravatar": grav_tmpl(user.email, 20),
                 "username": h.link_to(
                     user.username, h.url('user_profile', username=user.username)),
                 "username_raw": user.username,
                 "email": user.email,
                 "first_name": h.escape(user.name),
                 "last_name": h.escape(user.lastname),
                 "last_login": h.format_date(user.last_login),
                 "last_login_raw": datetime_to_time(user.last_login),
                 "last_activity": h.format_date(
                     h.time_to_datetime(user.user_data.get('last_activity', 0))),
                 "last_activity_raw": user.user_data.get('last_activity', 0),
                 "active": h.bool2icon(user.active),
                 "active_raw": user.active,
                 "admin": h.bool2icon(user.admin),
                 "admin_raw": user.admin,
                 "extern_type": user.extern_type,
                 "extern_name": user.extern_name,
                 "action": user_actions(user.user_id, user.username),
             })
 
 
         c.data = json.dumps(users_data)
         return render('admin/users/users.html')
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def create(self):
         """POST /users: Create a new item"""
         # url('users')
         c.default_extern_type = auth_rhodecode.RhodeCodeAuthPlugin.name
         user_model = UserModel()
         user_form = UserForm()()
         try:
             form_result = user_form.to_python(dict(request.POST))
             user = user_model.create(form_result)
             Session().flush()
             username = form_result['username']
             action_logger(c.rhodecode_user, 'admin_created_user:%s' % username,
                           None, self.ip_addr, self.sa)
 
             user_link = h.link_to(h.escape(username),
                                   url('edit_user',
                                       user_id=user.user_id))
             h.flash(h.literal(_('Created user %(user_link)s')
                               % {'user_link': user_link}), category='success')
             Session().commit()
         except formencode.Invalid as errors:
             return htmlfill.render(
                 render('admin/users/user_add.html'),
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
         except UserCreationError as e:
             h.flash(e, 'error')
         except Exception:
             log.exception("Exception creation of user")
             h.flash(_('Error occurred during creation of user %s')
                     % request.POST.get('username'), category='error')
         return redirect(url('users'))
 
     @HasPermissionAllDecorator('hg.admin')
     def new(self):
         """GET /users/new: Form to create a new item"""
         # url('new_user')
         c.default_extern_type = auth_rhodecode.RhodeCodeAuthPlugin.name
         return render('admin/users/user_add.html')
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def update(self, user_id):
         """PUT /users/user_id: Update an existing item"""
         # Forms posted to this method should contain a hidden field:
         # <input type="hidden" name="_method" value="PUT" />
         # Or using helpers:
         #    h.form(url('update_user', user_id=ID),
         #           method='put')
         # url('user', user_id=ID)
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         c.active = 'profile'
         c.extern_type = c.user.extern_type
         c.extern_name = c.user.extern_name
         c.perm_user = AuthUser(user_id=user_id, ip_addr=self.ip_addr)
         available_languages = [x[0] for x in c.allowed_languages]
         _form = UserForm(edit=True, available_languages=available_languages,
                          old_data={'user_id': user_id,
                                    'email': c.user.email})()
         form_result = {}
         try:
             form_result = _form.to_python(dict(request.POST))
             skip_attrs = ['extern_type', 'extern_name']
             # TODO: plugin should define if username can be updated
             if c.extern_type != "rhodecode":
                 # forbid updating username for external accounts
                 skip_attrs.append('username')
 
             UserModel().update_user(user_id, skip_attrs=skip_attrs, **form_result)
             usr = form_result['username']
             action_logger(c.rhodecode_user, 'admin_updated_user:%s' % usr,
                           None, self.ip_addr, self.sa)
             h.flash(_('User updated successfully'), category='success')
             Session().commit()
         except formencode.Invalid as errors:
             defaults = errors.value
             e = errors.error_dict or {}
 
             return htmlfill.render(
                 render('admin/users/user_edit.html'),
                 defaults=defaults,
                 errors=e,
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
+        except UserCreationError as e:
+            h.flash(e, 'error')
         except Exception:
             log.exception("Exception updating user")
             h.flash(_('Error occurred during update of user %s')
                     % form_result.get('username'), category='error')
         return redirect(url('edit_user', user_id=user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def delete(self, user_id):
         """DELETE /users/user_id: Delete an existing item"""
         # Forms posted to this method should contain a hidden field:
         # <input type="hidden" name="_method" value="DELETE" />
         # Or using helpers:
         #    h.form(url('delete_user', user_id=ID),
         #           method='delete')
         # url('user', user_id=ID)
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
 
         _repos = c.user.repositories
         _repo_groups = c.user.repository_groups
         _user_groups = c.user.user_groups
 
         handle_repos = None
         handle_repo_groups = None
         handle_user_groups = None
         # dummy call for flash of handle
         set_handle_flash_repos = lambda: None
         set_handle_flash_repo_groups = lambda: None
         set_handle_flash_user_groups = lambda: None
 
         if _repos and request.POST.get('user_repos'):
             do = request.POST['user_repos']
             if do == 'detach':
                 handle_repos = 'detach'
                 set_handle_flash_repos = lambda: h.flash(
                     _('Detached %s repositories') % len(_repos),
                     category='success')
             elif do == 'delete':
                 handle_repos = 'delete'
                 set_handle_flash_repos = lambda: h.flash(
                     _('Deleted %s repositories') % len(_repos),
                     category='success')
 
         if _repo_groups and request.POST.get('user_repo_groups'):
             do = request.POST['user_repo_groups']
             if do == 'detach':
                 handle_repo_groups = 'detach'
                 set_handle_flash_repo_groups = lambda: h.flash(
                     _('Detached %s repository groups') % len(_repo_groups),
                     category='success')
             elif do == 'delete':
                 handle_repo_groups = 'delete'
                 set_handle_flash_repo_groups = lambda: h.flash(
                     _('Deleted %s repository groups') % len(_repo_groups),
                     category='success')
 
         if _user_groups and request.POST.get('user_user_groups'):
             do = request.POST['user_user_groups']
             if do == 'detach':
                 handle_user_groups = 'detach'
                 set_handle_flash_user_groups = lambda: h.flash(
                     _('Detached %s user groups') % len(_user_groups),
                     category='success')
             elif do == 'delete':
                 handle_user_groups = 'delete'
                 set_handle_flash_user_groups = lambda: h.flash(
                     _('Deleted %s user groups') % len(_user_groups),
                     category='success')
 
         try:
             UserModel().delete(c.user, handle_repos=handle_repos,
                                handle_repo_groups=handle_repo_groups,
                                handle_user_groups=handle_user_groups)
             Session().commit()
             set_handle_flash_repos()
             set_handle_flash_repo_groups()
             set_handle_flash_user_groups()
             h.flash(_('Successfully deleted user'), category='success')
         except (UserOwnsReposException, UserOwnsRepoGroupsException,
                 UserOwnsUserGroupsException, DefaultUserException) as e:
             h.flash(e, category='warning')
         except Exception:
             log.exception("Exception during deletion of user")
             h.flash(_('An error occurred during deletion of user'),
                     category='error')
         return redirect(url('users'))
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def reset_password(self, user_id):
         """
         toggle reset password flag for this user
 
         :param user_id:
         """
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         try:
             old_value = c.user.user_data.get('force_password_change')
             c.user.update_userdata(force_password_change=not old_value)
             Session().commit()
             if old_value:
                 msg = _('Force password change disabled for user')
             else:
                 msg = _('Force password change enabled for user')
             h.flash(msg, category='success')
         except Exception:
             log.exception("Exception during password reset for user")
             h.flash(_('An error occurred during password reset for user'),
                     category='error')
 
         return redirect(url('edit_user_advanced', user_id=user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def create_personal_repo_group(self, user_id):
         """
         Create personal repository group for this user
 
         :param user_id:
         """
         from rhodecode.model.repo_group import RepoGroupModel
 
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
 
         try:
             desc = RepoGroupModel.PERSONAL_GROUP_DESC % {
                 'username': c.user.username}
             if not RepoGroup.get_by_group_name(c.user.username):
                 RepoGroupModel().create(group_name=c.user.username,
                                         group_description=desc,
                                         owner=c.user.username)
 
                 msg = _('Created repository group `%s`' % (c.user.username,))
                 h.flash(msg, category='success')
         except Exception:
             log.exception("Exception during repository group creation")
             msg = _(
                 'An error occurred during repository group creation for user')
             h.flash(msg, category='error')
 
         return redirect(url('edit_user_advanced', user_id=user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     def show(self, user_id):
         """GET /users/user_id: Show a specific item"""
         # url('user', user_id=ID)
         User.get_or_404(-1)
 
     @HasPermissionAllDecorator('hg.admin')
     def edit(self, user_id):
         """GET /users/user_id/edit: Form to edit an existing item"""
         # url('edit_user', user_id=ID)
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         c.active = 'profile'
         c.extern_type = c.user.extern_type
         c.extern_name = c.user.extern_name
         c.perm_user = AuthUser(user_id=user_id, ip_addr=self.ip_addr)
 
         defaults = c.user.get_dict()
         defaults.update({'language': c.user.user_data.get('language')})
         return htmlfill.render(
             render('admin/users/user_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     def edit_advanced(self, user_id):
         user_id = safe_int(user_id)
         user = c.user = User.get_or_404(user_id)
         if user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         c.active = 'advanced'
         c.perm_user = AuthUser(user_id=user_id, ip_addr=self.ip_addr)
         c.personal_repo_group = RepoGroup.get_by_group_name(user.username)
-        c.first_admin = User.get_first_admin()
+        c.first_admin = User.get_first_super_admin()
         defaults = user.get_dict()
 
         # Interim workaround if the user participated on any pull requests as a
         # reviewer.
         has_review = bool(PullRequestReviewers.query().filter(
             PullRequestReviewers.user_id == user_id).first())
         c.can_delete_user = not has_review
         c.can_delete_user_message = _(
             'The user participates as reviewer in pull requests and '
             'cannot be deleted. You can set the user to '
             '"inactive" instead of deleting it.') if has_review else ''
 
         return htmlfill.render(
             render('admin/users/user_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     def edit_auth_tokens(self, user_id):
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         c.active = 'auth_tokens'
         show_expired = True
         c.lifetime_values = [
             (str(-1), _('forever')),
             (str(5), _('5 minutes')),
             (str(60), _('1 hour')),
             (str(60 * 24), _('1 day')),
             (str(60 * 24 * 30), _('1 month')),
         ]
         c.lifetime_options = [(c.lifetime_values, _("Lifetime"))]
         c.role_values = [(x, AuthTokenModel.cls._get_role_name(x))
                          for x in AuthTokenModel.cls.ROLES]
         c.role_options = [(c.role_values, _("Role"))]
         c.user_auth_tokens = AuthTokenModel().get_auth_tokens(
             c.user.user_id, show_expired=show_expired)
         defaults = c.user.get_dict()
         return htmlfill.render(
             render('admin/users/user_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def add_auth_token(self, user_id):
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         lifetime = safe_int(request.POST.get('lifetime'), -1)
         description = request.POST.get('description')
         role = request.POST.get('role')
         AuthTokenModel().create(c.user.user_id, description, lifetime, role)
         Session().commit()
         h.flash(_("Auth token successfully created"), category='success')
         return redirect(url('edit_user_auth_tokens', user_id=c.user.user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def delete_auth_token(self, user_id):
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         auth_token = request.POST.get('del_auth_token')
         if request.POST.get('del_auth_token_builtin'):
             user = User.get(c.user.user_id)
             if user:
                 user.api_key = generate_auth_token(user.username)
                 Session().add(user)
                 Session().commit()
                 h.flash(_("Auth token successfully reset"), category='success')
         elif auth_token:
             AuthTokenModel().delete(auth_token, c.user.user_id)
             Session().commit()
             h.flash(_("Auth token successfully deleted"), category='success')
 
         return redirect(url('edit_user_auth_tokens', user_id=c.user.user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     def edit_global_perms(self, user_id):
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         c.active = 'global_perms'
 
         c.default_user = User.get_default_user()
         defaults = c.user.get_dict()
         defaults.update(c.default_user.get_default_perms(suffix='_inherited'))
         defaults.update(c.default_user.get_default_perms())
         defaults.update(c.user.get_default_perms())
 
         return htmlfill.render(
             render('admin/users/user_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def update_global_perms(self, user_id):
         """PUT /users_perm/user_id: Update an existing item"""
         # url('user_perm', user_id=ID, method='put')
         user_id = safe_int(user_id)
         user = User.get_or_404(user_id)
         c.active = 'global_perms'
         try:
             # first stage that verifies the checkbox
             _form = UserIndividualPermissionsForm()
             form_result = _form.to_python(dict(request.POST))
             inherit_perms = form_result['inherit_default_permissions']
             user.inherit_default_permissions = inherit_perms
             Session().add(user)
 
             if not inherit_perms:
                 # only update the individual ones if we un check the flag
                 _form = UserPermissionsForm(
                     [x[0] for x in c.repo_create_choices],
                     [x[0] for x in c.repo_create_on_write_choices],
                     [x[0] for x in c.repo_group_create_choices],
                     [x[0] for x in c.user_group_create_choices],
                     [x[0] for x in c.fork_choices],
                     [x[0] for x in c.inherit_default_permission_choices])()
 
                 form_result = _form.to_python(dict(request.POST))
                 form_result.update({'perm_user_id': user.user_id})
 
                 PermissionModel().update_user_permissions(form_result)
 
             Session().commit()
             h.flash(_('User global permissions updated successfully'),
                     category='success')
 
             Session().commit()
         except formencode.Invalid as errors:
             defaults = errors.value
             c.user = user
             return htmlfill.render(
                 render('admin/users/user_edit.html'),
                 defaults=defaults,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False)
         except Exception:
             log.exception("Exception during permissions saving")
             h.flash(_('An error occurred during permissions saving'),
                     category='error')
         return redirect(url('edit_user_global_perms', user_id=user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     def edit_perms_summary(self, user_id):
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         c.active = 'perms_summary'
         c.perm_user = AuthUser(user_id=user_id, ip_addr=self.ip_addr)
 
         return render('admin/users/user_edit.html')
 
     @HasPermissionAllDecorator('hg.admin')
     def edit_emails(self, user_id):
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         c.active = 'emails'
         c.user_email_map = UserEmailMap.query() \
             .filter(UserEmailMap.user == c.user).all()
 
         defaults = c.user.get_dict()
         return htmlfill.render(
             render('admin/users/user_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def add_email(self, user_id):
         """POST /user_emails:Add an existing item"""
         # url('user_emails', user_id=ID, method='put')
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
 
         email = request.POST.get('new_email')
         user_model = UserModel()
 
         try:
             user_model.add_extra_email(user_id, email)
             Session().commit()
             h.flash(_("Added new email address `%s` for user account") % email,
                     category='success')
         except formencode.Invalid as error:
             msg = error.error_dict['email']
             h.flash(msg, category='error')
         except Exception:
             log.exception("Exception during email saving")
             h.flash(_('An error occurred during email saving'),
                     category='error')
         return redirect(url('edit_user_emails', user_id=user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def delete_email(self, user_id):
         """DELETE /user_emails_delete/user_id: Delete an existing item"""
         # url('user_emails_delete', user_id=ID, method='delete')
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         email_id = request.POST.get('del_email_id')
         user_model = UserModel()
         user_model.delete_extra_email(user_id, email_id)
         Session().commit()
         h.flash(_("Removed email address from user account"), category='success')
         return redirect(url('edit_user_emails', user_id=user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     def edit_ips(self, user_id):
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         if c.user.username == User.DEFAULT_USER:
             h.flash(_("You can't edit this user"), category='warning')
             return redirect(url('users'))
 
         c.active = 'ips'
         c.user_ip_map = UserIpMap.query() \
             .filter(UserIpMap.user == c.user).all()
 
         c.inherit_default_ips = c.user.inherit_default_permissions
         c.default_user_ip_map = UserIpMap.query() \
             .filter(UserIpMap.user == User.get_default_user()).all()
 
         defaults = c.user.get_dict()
         return htmlfill.render(
             render('admin/users/user_edit.html'),
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False)
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def add_ip(self, user_id):
         """POST /user_ips:Add an existing item"""
         # url('user_ips', user_id=ID, method='put')
 
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
         user_model = UserModel()
         try:
             ip_list = user_model.parse_ip_range(request.POST.get('new_ip'))
         except Exception as e:
             ip_list = []
             log.exception("Exception during ip saving")
             h.flash(_('An error occurred during ip saving:%s' % (e,)),
                     category='error')
 
         desc = request.POST.get('description')
         added = []
         for ip in ip_list:
             try:
                 user_model.add_extra_ip(user_id, ip, desc)
                 Session().commit()
                 added.append(ip)
             except formencode.Invalid as error:
                 msg = error.error_dict['ip']
                 h.flash(msg, category='error')
             except Exception:
                 log.exception("Exception during ip saving")
                 h.flash(_('An error occurred during ip saving'),
                         category='error')
         if added:
             h.flash(
                 _("Added ips %s to user whitelist") % (', '.join(ip_list), ),
                 category='success')
         if 'default_user' in request.POST:
             return redirect(url('admin_permissions_ips'))
         return redirect(url('edit_user_ips', user_id=user_id))
 
     @HasPermissionAllDecorator('hg.admin')
     @auth.CSRFRequired()
     def delete_ip(self, user_id):
         """DELETE /user_ips_delete/user_id: Delete an existing item"""
         # url('user_ips_delete', user_id=ID, method='delete')
         user_id = safe_int(user_id)
         c.user = User.get_or_404(user_id)
 
         ip_id = request.POST.get('del_ip_id')
         user_model = UserModel()
         user_model.delete_extra_ip(user_id, ip_id)
         Session().commit()
         h.flash(_("Removed ip address from user whitelist"), category='success')
 
         if 'default_user' in request.POST:
             return redirect(url('admin_permissions_ips'))
         return redirect(url('edit_user_ips', user_id=user_id))

rhodecode/controllers/home.py

@@ -1,277 +1,289 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 Home controller for RhodeCode Enterprise
 """
 
 import logging
 import time
 import re
 
 from pylons import tmpl_context as c, request, url, config
 from pylons.i18n.translation import _
 from sqlalchemy.sql import func
 
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator, AuthUser,
     HasRepoGroupPermissionAnyDecorator, XHRRequired)
 from rhodecode.lib.base import BaseController, render
 from rhodecode.lib.index import searcher_from_config
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.utils import jsonify
-from rhodecode.lib.utils2 import safe_unicode
+from rhodecode.lib.utils2 import safe_unicode, str2bool
 from rhodecode.model.db import Repository, RepoGroup
 from rhodecode.model.repo import RepoModel
 from rhodecode.model.repo_group import RepoGroupModel
 from rhodecode.model.scm import RepoList, RepoGroupList
 
 
 log = logging.getLogger(__name__)
 
 
 class HomeController(BaseController):
     def __before__(self):
         super(HomeController, self).__before__()
 
     def ping(self):
         """
         Ping, doesn't require login, good for checking out the platform
         """
         instance_id = getattr(c, 'rhodecode_instanceid', '')
         return 'pong[%s] => %s' % (instance_id, self.ip_addr,)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def error_test(self):
         """
         Test exception handling and emails on errors
         """
         class TestException(Exception):
             pass
 
         msg = ('RhodeCode Enterprise %s test exception. Generation time: %s'
                % (c.rhodecode_name, time.time()))
         raise TestException(msg)
 
     def _get_groups_and_repos(self, repo_group_id=None):
         # repo groups groups
         repo_group_list = RepoGroup.get_all_repo_groups(group_id=repo_group_id)
         _perms = ['group.read', 'group.write', 'group.admin']
         repo_group_list_acl = RepoGroupList(repo_group_list, perm_set=_perms)
         repo_group_data = RepoGroupModel().get_repo_groups_as_dict(
             repo_group_list=repo_group_list_acl, admin=False)
 
         # repositories
         repo_list = Repository.get_all_repos(group_id=repo_group_id)
         _perms = ['repository.read', 'repository.write', 'repository.admin']
         repo_list_acl = RepoList(repo_list, perm_set=_perms)
         repo_data = RepoModel().get_repos_as_dict(
             repo_list=repo_list_acl, admin=False)
 
         return repo_data, repo_group_data
 
     @LoginRequired()
     def index(self):
         c.repo_group = None
 
         repo_data, repo_group_data = self._get_groups_and_repos()
         # json used to render the grids
         c.repos_data = json.dumps(repo_data)
         c.repo_groups_data = json.dumps(repo_group_data)
 
         return render('/index.html')
 
     @LoginRequired()
     @HasRepoGroupPermissionAnyDecorator('group.read', 'group.write',
                                         'group.admin')
     def index_repo_group(self, group_name):
         """GET /repo_group_name: Show a specific item"""
         c.repo_group = RepoGroupModel()._get_repo_group(group_name)
         repo_data, repo_group_data = self._get_groups_and_repos(
             c.repo_group.group_id)
 
         # json used to render the grids
         c.repos_data = json.dumps(repo_data)
         c.repo_groups_data = json.dumps(repo_group_data)
 
         return render('index_repo_group.html')
 
     def _get_repo_list(self, name_contains=None, repo_type=None, limit=20):
         query = Repository.query()\
             .order_by(func.length(Repository.repo_name))\
             .order_by(Repository.repo_name)
 
         if repo_type:
             query = query.filter(Repository.repo_type == repo_type)
 
         if name_contains:
             ilike_expression = u'%{}%'.format(safe_unicode(name_contains))
             query = query.filter(
                 Repository.repo_name.ilike(ilike_expression))
             query = query.limit(limit)
 
         all_repos = query.all()
         repo_iter = self.scm_model.get_repos(all_repos)
         return [
             {
                 'id': obj['name'],
                 'text': obj['name'],
                 'type': 'repo',
                 'obj': obj['dbrepo'],
                 'url': url('summary_home', repo_name=obj['name'])
             }
             for obj in repo_iter]
 
     def _get_repo_group_list(self, name_contains=None, limit=20):
         query = RepoGroup.query()\
             .order_by(func.length(RepoGroup.group_name))\
             .order_by(RepoGroup.group_name)
 
         if name_contains:
             ilike_expression = u'%{}%'.format(safe_unicode(name_contains))
             query = query.filter(
                 RepoGroup.group_name.ilike(ilike_expression))
             query = query.limit(limit)
 
         all_groups = query.all()
         repo_groups_iter = self.scm_model.get_repo_groups(all_groups)
         return [
             {
                 'id': obj.group_name,
                 'text': obj.group_name,
                 'type': 'group',
                 'obj': {},
                 'url': url('repo_group_home', group_name=obj.group_name)
             }
             for obj in repo_groups_iter]
 
     def _get_hash_commit_list(self, hash_starts_with=None, limit=20):
         if not hash_starts_with or len(hash_starts_with) < 3:
             return []
 
         commit_hashes = re.compile('([0-9a-f]{2,40})').findall(hash_starts_with)
 
         if len(commit_hashes) != 1:
             return []
 
         commit_hash_prefix = commit_hashes[0]
 
         auth_user = AuthUser(
             user_id=c.rhodecode_user.user_id, ip_addr=self.ip_addr)
         searcher = searcher_from_config(config)
         result = searcher.search(
             'commit_id:%s*' % commit_hash_prefix, 'commit', auth_user)
 
         return [
             {
                 'id': entry['commit_id'],
                 'text': entry['commit_id'],
                 'type': 'commit',
                 'obj': {'repo': entry['repository']},
                 'url': url('changeset_home',
                     repo_name=entry['repository'], revision=entry['commit_id'])
             }
             for entry in result['results']]
 
     @LoginRequired()
     @XHRRequired()
     @jsonify
     def goto_switcher_data(self):
         query = request.GET.get('query')
         log.debug('generating goto switcher list, query %s', query)
 
         res = []
         repo_groups = self._get_repo_group_list(query)
         if repo_groups:
             res.append({
                 'text': _('Groups'),
                 'children': repo_groups
             })
 
         repos = self._get_repo_list(query)
         if repos:
             res.append({
                 'text': _('Repositories'),
                 'children': repos
             })
 
         commits = self._get_hash_commit_list(query)
         if commits:
             unique_repos = {}
             for commit in commits:
                 unique_repos.setdefault(commit['obj']['repo'], []
                     ).append(commit)
 
             for repo in unique_repos:
                 res.append({
                     'text': _('Commits in %(repo)s') % {'repo': repo},
                     'children': unique_repos[repo]
                 })
 
         data = {
             'more': False,
             'results': res
         }
         return data
 
     @LoginRequired()
     @XHRRequired()
     @jsonify
     def repo_list_data(self):
         query = request.GET.get('query')
         repo_type = request.GET.get('repo_type')
         log.debug('generating repo list, query:%s', query)
 
         res = []
         repos = self._get_repo_list(query, repo_type=repo_type)
         if repos:
             res.append({
                 'text': _('Repositories'),
                 'children': repos
             })
 
         data = {
             'more': False,
             'results': res
         }
         return data
 
     @LoginRequired()
     @XHRRequired()
     @jsonify
     def user_autocomplete_data(self):
         query = request.GET.get('query')
+        active = str2bool(request.GET.get('active') or True)
 
         repo_model = RepoModel()
-        _users = repo_model.get_users(name_contains=query)
+        _users = repo_model.get_users(
+            name_contains=query, only_active=active)
 
         if request.GET.get('user_groups'):
             # extend with user groups
-            _user_groups = repo_model.get_user_groups(name_contains=query)
+            _user_groups = repo_model.get_user_groups(
+                name_contains=query, only_active=active)
             _users = _users + _user_groups
 
         return {'suggestions': _users}
 
     @LoginRequired()
     @XHRRequired()
     @jsonify
     def user_group_autocomplete_data(self):
-        return {'suggestions': []}
+        query = request.GET.get('query')
+        active = str2bool(request.GET.get('active') or True)
+
+        repo_model = RepoModel()
+        _user_groups = repo_model.get_user_groups(
+            name_contains=query, only_active=active)
+        _user_groups = _user_groups
+
+        return {'suggestions': _user_groups}
+

rhodecode/events.py

@@ -1,31 +1,53 b''
 # Copyright (C) 2016-2016  RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 from zope.interface import implementer
-from rhodecode.interfaces import IUserRegistered
+from rhodecode.interfaces import (
+    IUserRegistered, IUserPreCreate, IUserPreUpdate)
 
 
 @implementer(IUserRegistered)
 class UserRegistered(object):
     """
     An instance of this class is emitted as an :term:`event` whenever a user
     account is registered.
     """
     def __init__(self, user, session):
         self.user = user
         self.session = session
+
+
+@implementer(IUserPreCreate)
+class UserPreCreate(object):
+    """
+    An instance of this class is emitted as an :term:`event` before a new user
+    object is created.
+    """
+    def __init__(self, user_data):
+        self.user_data = user_data
+
+
+@implementer(IUserPreUpdate)
+class UserPreUpdate(object):
+    """
+    An instance of this class is emitted as an :term:`event` before a user
+    object is updated.
+    """
+    def __init__(self, user, user_data):
+        self.user = user
+        self.user_data = user_data