release: Merge default into stable for release preparation

docs/admin/svn-path-permissions.rst

@@ -0,0 +1,80 b''
+.. _svn-path-permissions:
+
+|svn| Enabling Path Permissions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Because |RCEE| uses standard svn apache mod_svn we can take advantage of the
+authz configuration to protect paths and branches.
+
+
+Configuring RhodeCode
+=====================
+
+
+1. To configure path based permissions first we need to use a customized
+   mod_dav_svn.conf.
+
+   Open :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
+   And find `svn.proxy.config_template` setting. Now set a new path to read
+   the template from. For example:
+
+    .. code-block:: ini
+
+        svn.proxy.config_template = /home/ubuntu/rhodecode/custom_mod_dav_svn.conf.mako
+
+
+2. Create the file as in example: `/home/ubuntu/rhodecode/custom_mod_dav_svn.conf.mako`
+   You can download one from:
+
+    `<https://code.rhodecode.com/rhodecode-enterprise-ce/files/default/rhodecode/apps/svn_support/templates/mod-dav-svn.conf.mako/>`_
+
+3. Add (if not yet exists) a section `AuthzSVNReposRelativeAccessFile`  in order
+   to read the path auth file.
+
+   Example modified config section enabling reading the authz file relative
+   to repository path. Means located in `/storage_dir/repo_name/conf/authz`
+
+    .. code-block:: text
+
+
+        # snip ...
+
+        # use specific SVN conf/authz file for each repository
+        AuthzSVNReposRelativeAccessFile authz
+
+        Allow from all
+        # snip ...
+
+    .. note::
+
+       The `AuthzSVNReposRelativeAccessFile` should go above the `Allow from all`
+       directive.
+
+
+4. Restart RhodeCode, Go to
+   the :menuselection:`Admin --> Settings --> VCS` page, and
+   click :guilabel:`Generate Apache Config`.
+   This will now generate a new configuration with enabled changes to read
+   the authz file. You can verify if changes were made by checking the generated
+   mod_dav_svn.conf file which is included in your apache configuration.
+
+5. Specify new rules in the repository authz configuration.
+   edit a file in :file:`repo_name/conf/authz`. For example, we specify that
+   only admin is allowed to push to develop branch
+
+    .. code-block:: ini
+
+        [/branches/develop]
+        * = r
+        admin = rw
+
+
+   For more example see:
+   `<https://svn.apache.org/repos/asf/subversion/trunk/subversion/mod_authz_svn/INSTALL/>`_
+
+  Those rules also work for paths, so not only branches but all different
+  paths inside the repository can be specified.
+
+6. Reload Apache. If all is configured correctly it should not be allowed to
+   commit according to specified rules.
+

docs/api/methods/views.rst

@@ -0,0 +1,48 b''
+.. _views-ref:
+
+views
+=====
+
+push (EE only)
+--------------
+
+.. py:function:: push(apiuser, repoid, remote_uri=<Optional:None>)
+
+   Triggers a push on the given repository from a remote location. You
+   can use this to keep remote repositories up-to-date.
+
+   This command can only be run using an |authtoken| with admin
+   rights to the specified repository. For more information,
+   see :ref:`config-token-ref`.
+
+   This command takes the following options:
+
+   :param apiuser: This is filled automatically from the |authtoken|.
+   :type apiuser: AuthUser
+   :param repoid: The repository name or repository ID.
+   :type repoid: str or int
+   :param remote_uri: Optional remote URI to pass in for push
+   :type remote_uri: str
+
+   Example output:
+
+   .. code-block:: bash
+
+     id : <id_given_in_input>
+     result : {
+       "msg": "Pushed to url `<remote_url>` on repo `<repository name>`"
+       "repository": "<repository name>"
+     }
+     error :  null
+
+   Example error output:
+
+   .. code-block:: bash
+
+     id : <id_given_in_input>
+     result : null
+     error :  {
+       "Unable to push changes to `<remote_url>`"
+     }
+
+

docs/auth/auth-ldap-groups.rst

@@ -0,0 +1,112 b''
+.. _config-ldap-groups-ref:
+
+LDAP/AD With User Groups Sync
+-----------------------------
+
+|RCM| supports LDAP (Lightweight Directory Access Protocol) or
+AD (active Directory) authentication.
+All LDAP versions are supported, with the following |RCM| plugins managing each:
+
+* For LDAP/AD with user group sync use ``LDAP + User Groups (egg:rhodecode-enterprise-ee#ldap_group)``
+
+RhodeCode reads all data defined from plugin and creates corresponding
+accounts on local database after receiving data from LDAP. This is done on
+every user log-in including operations like pushing/pulling/checkout.
+In addition group membership is read from LDAP and following operations are done:
+
+- automatic addition of user to |RCM| user group
+- automatic removal of user from any other  |RCM| user groups not specified in LDAP.
+  The removal is done *only* on groups that are marked to be synced from ldap.
+  This setting can be changed in advanced settings on user groups
+- automatic creation of user groups if they aren't yet existing in |RCM|
+- marking user as super-admins if he is a member of any admin group defined in plugin settings
+
+This plugin is available only in EE Edition.
+
+.. important::
+
+   The email used with your |RCE| super-admin account needs to match the email
+   address attached to your admin profile in LDAP. This is because
+   within |RCE| the user email needs to be unique, and multiple users
+   cannot share an email account.
+
+   Likewise, if as an admin you also have a user account, the email address
+   attached to the user account needs to be different.
+
+
+LDAP Configuration Steps
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+To configure |LDAP|, use the following steps:
+
+1. From the |RCM| interface, select
+   :menuselection:`Admin --> Authentication`
+2. Enable the ldap+ groups plugin and select :guilabel:`Save`
+3. Select the :guilabel:`Enabled` check box in the plugin configuration section
+4. Add the required LDAP information and :guilabel:`Save`, for more details,
+   see :ref:`config-ldap-groups-examples`
+
+For a more detailed description of LDAP objects, see :ref:`ldap-gloss-ref`:
+
+.. _config-ldap-groups-examples:
+
+Example LDAP configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: bash
+
+        # Auth Cache TTL, Defines the caching for authentication to offload LDAP server.
+        # This means that cache result will be saved for 3600 before contacting LDAP server to verify the user access
+        3600
+        # Host, comma seperated format is optionally possible to specify more than 1 server
+        https://ldap1.server.com/ldap-admin/,https://ldap2.server.com/ldap-admin/
+        # Default LDAP Port, use 689 for LDAPS
+        389
+        # Account, used for SimpleBind if LDAP server requires an authentication
+        e.g [e-mail]
+        # Password used for simple bind
+        ldap-user-password
+        # LDAP connection security
+        LDAPS
+        # Certificate checks level
+        DEMAND
+        # Base DN
+        cn=Rufus Magillacuddy,ou=users,dc=rhodecode,dc=com
+        # User Search Base
+        ou=groups,ou=users
+        # LDAP search filter to narrow the results
+        (objectClass=person)
+        # LDAP search scope
+        SUBTREE
+        # Login attribute
+        sAMAccountName
+        # First Name Attribute to read
+        givenName
+        # Last Name Attribute to read
+        sn
+        # Email Attribute to read email address from
+        mail
+        # group extraction method
+        rfc2307bis
+        # Group search base
+        ou=RC-Groups
+        # Group Name Attribute, field to read the group name from
+        sAMAAccountName
+        # User Member of Attribute, field in which groups are stored
+        memberOf
+        # LDAP Group Search Filter, allows narrowing the results
+
+        # Admin Groups. Comma separated list of groups. If user is member of
+        # any of those he will be marked as super-admin in RhodeCode
+        admins, management
+
+
+Below is example setup that can be used with Active Directory and ldap groups.
+
+.. image:: ../images/ldap-groups-example.png
+   :alt: LDAP/AD setup example
+   :scale: 50 %
+
+.. toctree::
+
+   ldap-active-directory
+   ldap-authentication
  No newline at end of file

docs/release-notes/release-notes-4.12.0.rst

@@ -0,0 +1,139 b''
+|RCE| 4.12.0 |RNS|
+------------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2018-04-24
+
+
+New Features
+^^^^^^^^^^^^
+
+- Svn: added support for RhodeCode integration framework. All integrations like
+  slack, email, Jenkins now also fully work for SVN.
+- Integrations: added new dedicated Jenkins integration with the support of
+  CSRF authentication. Available in EE edition only.
+- Automation: added new bi-directional remote sync. RhodeCode instances can now
+  automatically push or pull from/to remote locations. This feature is powered
+  by the Scheduler of 4.11 release, and it is required to be enabled for this feature to work.
+  Available in EE edition only.
+- Mercurial: path-based permissions. RhodeCode can now use Mercurials narrowhg
+  to implement path-based permissions. All permissions are read from .hg/hgacl.
+  Thanks to the great contribution from Sandu Turcan.
+- VCS: added new diff caches. Available as an option under vcs settings.
+  Diff caches work on pull-request, or individual commits for greater
+  performance and reduced memory usage. This feature increases speed of large
+  pull requests significantly. In addition for pull requests it will allow
+  showing old closed pull requests even if commits from source were removed,
+  further enhancing auditing capabilities.
+- Audit: added few new audit log entries especially around changing permissions.
+- LDAP: added connection pinning and timeout option to ldap plugin. This should
+  prevent problems when connection to LDAP is not stable causing RhodeCode
+  instances to freeze waiting on LDAP connections.
+- User groups: expose public user group profiles. Allows to see members of a user
+  groups by other team members, if they have proper permissions.
+- UI: show pull request page in quick nav menu on my account for quicker access.
+- UI: hidden/outdated comments now have visible markers next to line numbers.
+  This allows access to them without showing all hidden comments.
+
+
+General
+^^^^^^^
+
+- Ssh: show conflicting fingerprint when adding an already existing key.
+  Helps to track why adding a key failed.
+- System info: added ulimit to system info. This is causing lots of problems
+  when we hit any of those limits, that is why it's important to show this.
+- Repository settings: add hidden view to force re-install hooks.
+  Available under /{repo_name}/settings/advanced/hooks
+- Integrations: Webhook now handles response errors and show response for
+  easier debugging.
+- Cli: speed up CLI execution start by skipping auth plugin search/registry.
+- SVN: added an example in the docs on how to enable path-based permissions.
+- LDAP: enable connection recycling on LDAP plugin.
+- Auth plugins: use a nicer visual display of auth plugins that would
+  highlight that order of enabled plugins does matter.
+- Events: expose shadow repo build url.
+- Events: expose pull request title and uid in event data.
+- API: enable setting sync flag for user groups on create/edit.
+- API: update pull method with a possible specification of the url
+- Logging: improved consistency of auth plugins logs.
+- Logging: improved log for ssl required
+- Dependencies: bumped mercurial to 4.4 series
+- Dependencies: bumped zope.cachedescriptors==4.3.1
+- Dependencies: bumped zope.deprecation==4.3.0
+- Dependencies: bumped zope.event==4.3.0
+- Dependencies: bumped zope.interface==4.4.3
+- Dependencies: bumped graphviz 0.8.2
+- Dependencies: bumped to ipaddress 0.1.19
+- Dependencies: bumped pyexpect to 4.3.1
+- Dependencies: bumped ws4py to 0.4.3
+- Dependencies: bumped bleach to 2.1.2
+- Dependencies: bumped html5lib 1.0.1
+- Dependencies: bumped greenlet to 0.4.13
+- Dependencies: bumped markdown to 2.6.11
+- Dependencies: bumped psutil to 5.4.3
+- Dependencies: bumped beaker to 1.9.1
+- Dependencies: bumped alembic to 0.6.8 release.
+- Dependencies: bumped supervisor to 3.3.4
+- Dependencies: bumped pyexpect to 4.4.0 and scandir to 1.7
+- Dependencies: bumped appenlight client to 0.6.25
+- Dependencies: don't require full mysql lib for the db driver.
+  Reduces installation package size by around 100MB.
+
+
+Security
+^^^^^^^^
+
+- My account: changing email in my account now requires providing user
+  access password. This is a case for only RhodeCode built-in accounts.
+  Prevents adding recovery email by unauthorized users who gain
+  access to logged in session of user.
+- Logging: fix leaking of tokens to logging.
+- General: serialize the repo name in repo checks to prevent potential
+  html injections by providing a malformed url.
+
+
+Performance
+^^^^^^^^^^^
+
+- Diffs: don't use recurred diffset attachment in diffs. This makes
+this structure much harder to garbage collect. Reduces memory usage.
+- Diff cache: added caching for better performance of large pull requests.
+
+
+Fixes
+^^^^^
+
+- Age helper: fix issues with proper timezone detection for certain timezones.
+  Fixes wrong age display in few cases.
+- API: added audit logs for user group related calls that were
+  accidentally missing.
+- Diffs: fix and improve line selections and anchor links.
+- Pull requests: fixed cases with default expected refs are closed or unavailable.
+  For Mercurial with closed default branch a compare across forks could fail.
+- Core: properly report 502 errors for gevent and gunicorn.
+  Gevent wtih Gunicorn doesn't raise normal pycurl errors.
+- Auth plugins: fixed problem with cache of settings in multi-worker mode.
+  The previous implementation had a bug that cached the settings in each class,
+  caused not refreshing the update of settings in multi-worker mode.
+  Only restart of RhodeCode loaded new settings.
+- Audit logs: properly handle query syntax in the search field.
+- Repositories: better handling of missing requirements errors for repositories.
+- API: fixed problems with repository fork/create using celery backend.
+- VCS settings: added missing flash message on validation errors to prevent
+  missing out some field input validation problems.
+
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+- This release adds support for SVN hook. This required lots of changes on how we
+handle SVN protocol. We did thoughtful tests for SVN compatibility.
+Please be advised to check the behaviour of SVN repositories during this update.
+
+- Diff caches are turned off by default for backward compatibility. We however recommend
+turning them on either individually for bigger repositories or globally for every repository.
+This setting can be found in admin > settings > vcs, or repository > settings > vcs
+

pkgs/patch-beaker-lock-func-debug.diff

@@ -0,0 +1,20 b''
+diff -rup Beaker-1.9.1-orig/beaker/container.py Beaker-1.9.1/beaker/container.py
+--- Beaker-1.9.1-orig/beaker/container.py	2018-04-10 10:23:04.000000000 +0200
++++ Beaker-1.9.1/beaker/container.py	2018-04-10 10:23:34.000000000 +0200
+@@ -353,13 +353,13 @@ class Value(object):
+                 debug("get_value returning old value while new one is created")
+                 return value
+             else:
+-                debug("lock_creatfunc (didnt wait)")
++                debug("lock_creatfunc `%s` (didnt wait)", self.createfunc.__name__)
+                 has_createlock = True
+
+         if not has_createlock:
+-            debug("lock_createfunc (waiting)")
++            debug("lock_createfunc `%s` (waiting)", self.createfunc.__name__)
+             creation_lock.acquire()
+-            debug("lock_createfunc (waited)")
++            debug("lock_createfunc `%s` (waited)", self.createfunc.__name__)
+
+         try:
+             # see if someone created the value already

.bumpversion.cfg

@@ -1,6 +1,6 b''
 [bumpversion]
-current_version = 4.11.6
+current_version = 4.12.0
 message = release: Bump version {current_version} to {new_version}
 
 [bumpversion:file:rhodecode/VERSION]
 

.release.cfg

@@ -1,33 +1,28 b''
 [DEFAULT]
 done = false
 
 [task:bump_version]
 done = true
 
 [task:rc_tools_pinned]
-done = true
 
 [task:fixes_on_stable]
-done = true
 
 [task:pip2nix_generated]
-done = true
 
 [task:changelog_updated]
-done = true
 
 [task:generate_api_docs]
-done = true
+
+[task:updated_translation]
 
 [release]
-state = prepared
-version = 4.11.6
-
-[task:updated_translation]
+state = in_progress
+version = 4.12.0
 
 [task:generate_js_routes]
 
 [task:updated_trial_license]
 
 [task:generate_oss_licenses]
 

MANIFEST.in

@@ -1,54 +1,51 b''
 # top level files
 
 include MANIFEST.in
 include README.rst
 include CHANGES.rst
 include LICENSE.txt
 
 include rhodecode/VERSION
 
 # docs
 recursive-include docs *
 
 # all config files
 recursive-include configs *
 
 # translations
 recursive-include rhodecode/i18n *
 
-# hook templates
-recursive-include rhodecode/config/hook_templates *
-
 # non-python core stuff
 recursive-include rhodecode *.cfg
 recursive-include rhodecode *.json
 recursive-include rhodecode *.ini_tmpl
 recursive-include rhodecode *.sh
 recursive-include rhodecode *.mako
 
 # 502 page
 include rhodecode/public/502.html
 
 
 # images, css
 include rhodecode/public/css/*.css
 include rhodecode/public/images/*.*
 include rhodecode/public/images/ee_features/*.*
 
 # sound files
 include rhodecode/public/sounds/*.mp3
 include rhodecode/public/sounds/*.wav
 
 # fonts
 recursive-include rhodecode/public/fonts/ProximaNova *
 recursive-include rhodecode/public/fonts/RCIcons *
 
 # js
 recursive-include rhodecode/public/js *
 
 # templates
 recursive-include rhodecode/templates *
 
 # skip any tests files
 recursive-exclude rhodecode/tests *
 

configs/development.ini

@@ -1,722 +1,719 b''
 
 
 ################################################################################
 ##                 RHODECODE COMMUNITY EDITION CONFIGURATION                  ##
 # The %(here)s variable will be replaced with the parent directory of this file#
 ################################################################################
 
 [DEFAULT]
 debug = true
 
 ################################################################################
 ##                            EMAIL CONFIGURATION                             ##
 ## Uncomment and replace with the email address which should receive          ##
 ## any error reports after an application crash                               ##
 ## Additionally these settings will be used by the RhodeCode mailing system   ##
 ################################################################################
 
 ## prefix all emails subjects with given prefix, helps filtering out emails
 #email_prefix = [RhodeCode]
 
 ## email FROM address all mails will be sent
 #app_email_from = rhodecode-noreply@localhost
 
 ## Uncomment and replace with the address which should receive any error report
 ## note: using appenlight for error handling doesn't need this to be uncommented
 #email_to = admin@localhost
 
 ## in case of Application errors, sent an error email form
 #error_email_from = rhodecode_error@localhost
 
 ## additional error message to be send in case of server crash
 #error_message =
 
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 ## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
 #smtp_auth =
 
 [server:main]
 ## COMMON ##
 host = 127.0.0.1
 port = 5000
 
 ##################################
 ##     WAITRESS WSGI SERVER     ##
 ## Recommended for Development  ##
 ##################################
 
 use = egg:waitress#main
 ## number of worker threads
 threads = 5
 ## MAX BODY SIZE 100GB
 max_request_body_size = 107374182400
 ## Use poll instead of select, fixes file descriptors limits problems.
 ## May not work on old windows systems.
 asyncore_use_poll = true
 
 
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
 ## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 #use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
 ## when this option is set to more than one worker, recommended
 ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
 ## The `instance_id = *` must be set in the [app:main] section below
 #workers = 2
 ## number of threads for each of the worker, must be set to 1 for gevent
 ## generally recommended to be at 1
 #threads = 1
 ## process name
 #proc_name = rhodecode
 ## type of worker class, one of sync, gevent
 ## recommended for bigger setup is using of of other than sync one
 #worker_class = gevent
 ## The maximum number of simultaneous clients. Valid only for Gevent
 #worker_connections = 10
 ## max number of requests that worker will handle before being gracefully
 ## restarted, could prevent memory leaks
 #max_requests = 1000
 #max_requests_jitter = 30
 ## amount of time a worker can spend with handling a request before it
 ## gets killed and restarted. Set to 6hrs
 #timeout = 21600
 
 
 ## prefix middleware for RhodeCode.
 ## recommended when using proxy setup.
 ## allows to set RhodeCode under a prefix in server.
 ## eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
 ## And set your prefix like: `prefix = /custom_prefix`
 ## be sure to also set beaker.session.cookie_path = /custom_prefix if you need
 ## to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
 
 [app:main]
 use = egg:rhodecode-enterprise-ce
 
 ## enable proxy prefix middleware, defined above
 #filter-with = proxy-prefix
 
 # During development the we want to have the debug toolbar enabled
 pyramid.includes =
     pyramid_debugtoolbar
     rhodecode.lib.middleware.request_wrapper
 
 pyramid.reload_templates = true
 
 debugtoolbar.hosts = 0.0.0.0/0
 debugtoolbar.exclude_prefixes =
     /css
     /fonts
     /images
     /js
 
 ## RHODECODE PLUGINS ##
 rhodecode.includes =
     rhodecode.api
 
 
 # api prefix url
 rhodecode.api.url = /_admin/api
 
 
 ## END RHODECODE PLUGINS ##
 
 ## encryption key used to encrypt social plugin tokens,
 ## remote_urls with credentials etc, if not set it defaults to
 ## `beaker.session.secret`
 #rhodecode.encrypted_values.secret =
 
 ## decryption strict mode (enabled by default). It controls if decryption raises
 ## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
 ## return gzipped responses from Rhodecode (static files/application)
 gzip_responses = false
 
 ## autogenerate javascript routes file on startup
 generate_js_files = false
 
 ## Optional Languages
 ## en(default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ## perform a full repository scan on each server start, this should be
 ## set to false after first startup, to allow faster server restarts.
 startup.import_repos = false
 
 ## Uncomment and set this path to use archive download cache.
 ## Once enabled, generated archives will be cached at this location
 ## and served from the cache during subsequent requests for the same archive of
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
 ## URL at which the application is running. This is used for bootstraping
 ## requests in context when no web request is available. Used in ishell, or
 ## SSH calls. Set this for events to receive proper url for SSH calls.
 app.base_url = http://rhodecode.local
 
 ## change this to unique ID for security
 app_instance_uuid = rc-production
 
 ## cut off limit for large diffs (size in bytes). If overall diff size on
 ## commit, or pull request exceeds this limit this diff will be displayed
 ## partially. E.g 512000 == 512Kb
 cut_off_limit_diff = 512000
 
 ## cut off limit for large files inside diffs (size in bytes). Each individual
 ## file inside diff which exceeds this limit will be displayed partially.
 ##  E.g 128000 == 128Kb
 cut_off_limit_file = 128000
 
 ## use cache version of scm repo everywhere
 vcs_full_cache = true
 
 ## force https in RhodeCode, fixes https redirects, assumes it's always https
 ## Normally this is controlled by proper http flags sent from http server
 force_https = false
 
 ## use Strict-Transport-Security headers
 use_htsts = false
 
-## number of commits stats will parse on each iteration
-commit_parse_limit = 25
-
 ## git rev filter option, --all is the default filter, if you need to
 ## hide all refs in changelog switch this to --branches --tags
 git_rev_filter = --branches --tags
 
 # Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ## RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
 ## url that does rewrites to _admin/gists/{gistid}.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ## List of views (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
 ## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ## Additionally @TOKEN syntaxt can be used to bound the view to specific
 ## authentication token. Such view would be only accessible when used together
 ## with this authentication token
 ##
 ## list of all views can be found under `/_admin/permissions/auth_token_access`
 ## The list should be "," separated and on a single line.
 ##
 ## Most common views to enable:
 #    RepoCommitsView:repo_commit_download
 #    RepoCommitsView:repo_commit_patch
 #    RepoCommitsView:repo_commit_raw
 #    RepoCommitsView:repo_commit_raw@TOKEN
 #    RepoFilesView:repo_files_diff
 #    RepoFilesView:repo_archivefile
 #    RepoFilesView:repo_file_raw
 #    GistView:*
 api_access_controllers_whitelist =
 
 ## default encoding used to convert from and to unicode
 ## can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
 ## multiple instances of rhodecode, make sure it's globally unique for
 ## all running rhodecode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
 ## authentication settings. Then you can enable e.g. the internal rhodecode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
 ## egg:rhodecode-enterprise-ce#rhodecode
 ## egg:rhodecode-enterprise-ce#pam
 ## egg:rhodecode-enterprise-ce#ldap
 ## egg:rhodecode-enterprise-ce#jasig_cas
 ## egg:rhodecode-enterprise-ce#headers
 ## egg:rhodecode-enterprise-ce#crowd
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ## alternative return HTTP header for failed authentication. Default HTTP
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ## This will be served instead of default 401 on bad authnetication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
 ## ret_code directly, use 401 initially (Which triggers credentials prompt)
 ## and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ## locking return code. When repository is locked return this HTTP code. 2XX
 ## codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ## allows to change the repository location in settings page
 allow_repo_location_change = true
 
 ## allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ## generated license token, goto license page in RhodeCode settings to obtain
 ## new token
 license_token =
 
 ## supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 ## supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = dev
 
 ## Display extended labs settings
 labs_settings_active = true
 
 ####################################
 ###        CELERY CONFIG        ####
 ####################################
 ## run: /path/to/celery worker \
 ##     -E --beat --app rhodecode.lib.celerylib.loader \
 ##     --scheduler rhodecode.lib.celerylib.scheduler.RcScheduler \
 ##     --loglevel DEBUG --ini /path/to/rhodecode.ini
 
 use_celery = false
 
 ## connection url to the message broker (default rabbitmq)
 celery.broker_url = amqp://rabbitmq:qweqwe@localhost:5672/rabbitmqhost
 
 ## maximum tasks to execute before worker restart
 celery.max_tasks_per_child = 100
 
 ## tasks will never be sent to the queue, but executed locally instead.
 celery.task_always_eager = false
 
 ####################################
 ###         BEAKER CACHE        ####
 ####################################
 # default cache dir for templates.  Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk
 cache_dir = %(here)s/data
 
 ## locking and default file storage for Beaker. Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk/cache/beaker_data
 beaker.cache.data_dir = %(here)s/data/cache/beaker_data
 beaker.cache.lock_dir = %(here)s/data/cache/beaker_lock
 
 beaker.cache.regions = super_short_term, short_term, long_term, sql_cache_short, auth_plugins, repo_cache_long
 
 beaker.cache.super_short_term.type = memory
 beaker.cache.super_short_term.expire = 10
 beaker.cache.super_short_term.key_length = 256
 
 beaker.cache.short_term.type = memory
 beaker.cache.short_term.expire = 60
 beaker.cache.short_term.key_length = 256
 
 beaker.cache.long_term.type = memory
 beaker.cache.long_term.expire = 36000
 beaker.cache.long_term.key_length = 256
 
 beaker.cache.sql_cache_short.type = memory
 beaker.cache.sql_cache_short.expire = 10
 beaker.cache.sql_cache_short.key_length = 256
 
 ## default is memory cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.auth_plugins.type = ext:database
 #beaker.cache.auth_plugins.lock_dir = %(here)s/data/cache/auth_plugin_lock
 #beaker.cache.auth_plugins.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.cache.auth_plugins.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.cache.auth_plugins.sa.pool_recycle = 3600
 #beaker.cache.auth_plugins.sa.pool_size = 10
 #beaker.cache.auth_plugins.sa.max_overflow = 0
 
 beaker.cache.repo_cache_long.type = memorylru_base
 beaker.cache.repo_cache_long.max_items = 4096
 beaker.cache.repo_cache_long.expire = 2592000
 
 ## default is memorylru_base cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.repo_cache_long.type = ext:memcached
 #beaker.cache.repo_cache_long.url = localhost:11211
 #beaker.cache.repo_cache_long.expire = 1209600
 #beaker.cache.repo_cache_long.key_length = 256
 
 ####################################
 ###       BEAKER SESSION        ####
 ####################################
 
 ## .session.type is type of storage options for the session, current allowed
 ## types are file, ext:memcached, ext:database, and memory (default).
 beaker.session.type = file
 beaker.session.data_dir = %(here)s/data/sessions/data
 
 ## db based session, fast, and allows easy management over logged in users
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = develop-rc-uytcxaz
 beaker.session.lock_dir = %(here)s/data/sessions/lock
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = key_for_encryption
 #beaker.session.validate_key = validation_key
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 ## Path to use for the cookie. Set to prefix if you use prefix middleware
 #beaker.session.cookie_path = /custom_prefix
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
 
 ## auto save the session to not to use .save()
 beaker.session.auto = false
 
 ## default cookie expiration time in seconds, set to `true` to set expire
 ## at browser close
 #beaker.session.cookie_expires = 3600
 
 ###################################
 ## SEARCH INDEXING CONFIGURATION ##
 ###################################
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
 ## WHOOSH Backend, doesn't require additional services to run
 ## it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
 ########################################
 ###    CHANNELSTREAM CONFIG         ####
 ########################################
 ## channelstream enables persistent connections and live notification
 ## in the system. It's also used by the chat system
 channelstream.enabled = false
 
 ## server address for channelstream server on the backend
 channelstream.server = 127.0.0.1:9800
 
 ## location of the channelstream server from outside world
 ## use ws:// for http or wss:// for https. This address needs to be handled
 ## by external HTTP server such as Nginx or Apache
 ## see nginx/apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = secret
 channelstream.history.location = %(here)s/channelstream_history
 
 ## Internal application path that Javascript uses to connect into.
 ## If you use proxy-prefix the prefix should be added before /_channelstream
 channelstream.proxy_path = /_channelstream
 
 
 ###################################
 ##       APPENLIGHT CONFIG       ##
 ###################################
 
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
 ## appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
 # used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
 
 ## enables 404 error logging (default False)
 appenlight.report_404 = false
 
 ## time in seconds after request is considered being slow (default 1)
 appenlight.slow_request_time = 1
 
 ## record slow requests in application
 ## (needs to be enabled for slow datastore recording and time tracking)
 appenlight.slow_requests = true
 
 ## enable hooking to application loggers
 appenlight.logging = true
 
 ## minimum log level for log capture
 appenlight.logging.level = WARNING
 
 ## send logs only from erroneous/slow requests
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
 ## list of additonal keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
 ## start with HTTP* this list be extended with additional keywords here
 appenlight.environ_keys_whitelist =
 
 ## list of keywords that should be blanked from request object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always blank keys that contain following words
 ## 'password', 'passwd', 'pwd', 'auth_tkt', 'secret', 'csrf'
 ## this list be extended with additional keywords set here
 appenlight.request_keys_blacklist =
 
 ## list of namespaces that should be ignores when gathering log entries
 ## can be string with comma separated list of namespaces
 ## (by default the client ignores own entries: appenlight_client.client)
 appenlight.log_namespace_blacklist =
 
 
 ################################################################################
 ## WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*  ##
 ## Debug mode will enable the interactive debugging tool, allowing ANYONE to  ##
 ## execute malicious code after an exception is raised.                       ##
 ################################################################################
 #set debug = false
 
 
 ##############
 ## STYLING  ##
 ##############
 debug_style = true
 
 ###########################################
 ###   MAIN RHODECODE DATABASE CONFIG    ###
 ###########################################
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
 sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 
 # see sqlalchemy docs for other advanced settings
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
 ## recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
 ## the number of connections to keep open inside the connection pool.
 ## 0 indicates no limit
 #sqlalchemy.db1.pool_size = 5
 
 ## the number of connections to allow in connection pool "overflow", that is
 ## connections that can be opened above and beyond the pool_size setting,
 ## which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 
 ##################
 ### VCS CONFIG ###
 ##################
 vcs.server.enable = true
 vcs.server = localhost:9900
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
 ## `http` - use http-rpc backend (default)
 vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
 ## `http` - use http-rpc backend (default)
 ##
 vcs.scm_app_implementation = http
 
 ## Push/Pull operations hooks protocol, available options are:
 ## `http` - use http-rpc backend (default)
 vcs.hooks.protocol = http
 
 vcs.server.log_level = debug
 ## Start VCSServer with this instance as a subprocess, usefull for development
 vcs.start_server = false
 
 ## List of enabled VCS backends, available options are:
 ## `hg`  - mercurial
 ## `git` - git
 ## `svn` - subversion
 vcs.backends = hg, git, svn
 
 vcs.connection_timeout = 3600
 ## Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ## Available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible, pre-1.9-compatible
 #vcs.svn.compatible_version = pre-1.8-compatible
 
 
 ############################################################
 ### Subversion proxy support (mod_dav_svn)               ###
 ### Maps RhodeCode repo groups into SVN paths for Apache ###
 ############################################################
 ## Enable or disable the config file generation.
 svn.proxy.generate_config = false
 ## Generate config file with `SVNListParentPath` set to `On`.
 svn.proxy.list_parent_path = true
 ## Set location and file name of generated config file.
 svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
 ## alternative mod_dav config template. This needs to be a mako template
 #svn.proxy.config_template = ~/.rccontrol/enterprise-1/custom_svn_conf.mako
 ## Used as a prefix to the `Location` block in the generated config file.
 ## In most cases it should be set to `/`.
 svn.proxy.location_root = /
 ## Command to reload the mod dav svn configuration on change.
 ## Example: `/etc/init.d/apache2 reload`
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 ## If the timeout expires before the reload command finishes, the command will
 ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
 #svn.proxy.reload_timeout = 10
 
 ############################################################
 ### SSH Support Settings                                 ###
 ############################################################
 
 ## Defines if a custom authorized_keys file should be created and written on
 ## any change user ssh keys. Setting this to false also disables posibility
 ## of adding SSH keys by users from web interface. Super admins can still
 ## manage SSH Keys.
 ssh.generate_authorized_keyfile = false
 
 ## Options for ssh, default is `no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding`
 # ssh.authorized_keys_ssh_opts =
 
 ## Path to the authrozied_keys file where the generate entries are placed.
 ## It is possible to have multiple key files specified in `sshd_config` e.g.
 ## AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/authorized_keys_rhodecode
 ssh.authorized_keys_file_path = ~/.ssh/authorized_keys_rhodecode
 
 ## Command to execute the SSH wrapper. The binary is available in the
 ## rhodecode installation directory.
 ## e.g ~/.rccontrol/community-1/profile/bin/rc-ssh-wrapper
 ssh.wrapper_cmd = ~/.rccontrol/community-1/rc-ssh-wrapper
 
 ## Allow shell when executing the ssh-wrapper command
 ssh.wrapper_cmd_allow_shell = false
 
 ## Enables logging, and detailed output send back to the client during SSH
 ## operations. Usefull for debugging, shouldn't be used in production.
 ssh.enable_debug_logging = true
 
 ## Paths to binary executable, by default they are the names, but we can
 ## override them if we want to use a custom one
 ssh.executable.hg = ~/.rccontrol/vcsserver-1/profile/bin/hg
 ssh.executable.git = ~/.rccontrol/vcsserver-1/profile/bin/git
 ssh.executable.svn = ~/.rccontrol/vcsserver-1/profile/bin/svnserve
 
 
 ## Dummy marker to add new entries after.
 ## Add any custom entries below. Please don't remove.
 custom.conf = 1
 
 
 ################################
 ### LOGGING CONFIGURATION   ####
 ################################
 [loggers]
 keys = root, sqlalchemy, beaker, rhodecode, ssh_wrapper, celery
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, color_formatter, color_formatter_sql
 
 #############
 ## LOGGERS ##
 #############
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_ssh_wrapper]
 level = DEBUG
 handlers =
 qualname = ssh_wrapper
 propagate = 1
 
 [logger_celery]
 level = DEBUG
 handlers =
 qualname = celery
 
 
 ##############
 ## HANDLERS ##
 ##############
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr, )
 level = DEBUG
 formatter = color_formatter
 
 [handler_console_sql]
 class = StreamHandler
 args = (sys.stderr, )
 level = DEBUG
 formatter = color_formatter_sql
 
 ################
 ## FORMATTERS ##
 ################
 
 [formatter_generic]
 class = rhodecode.lib.logging_formatter.ExceptionAwareFormatter
-format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
+format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
-format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
+format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S

configs/gunicorn_config.py

@@ -1,151 +1,152 b''
 """
 gunicorn config extension and hooks. Sets additional configuration that is
 available post the .ini config.
 
 - workers = ${cpu_number}
 - threads = 1
 - proc_name = ${gunicorn_proc_name}
 - worker_class = sync
 - worker_connections = 10
 - max_requests = 1000
 - max_requests_jitter = 30
 - timeout = 21600
 
 """
 
 import multiprocessing
 import sys
 import time
 import datetime
 import threading
 import traceback
 from gunicorn.glogging import Logger
 
 
 # GLOBAL
 errorlog = '-'
 accesslog = '-'
 loglevel = 'debug'
 
 # SECURITY
 
 # The maximum size of HTTP request line in bytes.
 limit_request_line = 4094
 
 # Limit the number of HTTP headers fields in a request.
 limit_request_fields = 1024
 
 # Limit the allowed size of an HTTP request header field.
 # Value is a positive number or 0.
 # Setting it to 0 will allow unlimited header field sizes.
 limit_request_field_size = 0
 
 
 # Timeout for graceful workers restart.
 # After receiving a restart signal, workers have this much time to finish
 # serving requests. Workers still alive after the timeout (starting from the
 # receipt of the restart signal) are force killed.
 graceful_timeout = 30
 
 
 # The number of seconds to wait for requests on a Keep-Alive connection.
 # Generally set in the 1-5 seconds range.
 keepalive = 2
 
 
 # SERVER MECHANICS
 # None == system temp dir
+# worker_tmp_dir is recommended to be set to some tmpfs
 worker_tmp_dir = None
 tmp_upload_dir = None
 
 # Custom log format
 access_log_format = (
-    '%(t)s GNCRN %(p)-8s %(h)-15s rqt:%(L)s %(s)s %(b)-6s "%(m)s:%(U)s %(q)s" usr:%(u)s "%(f)s" "%(a)s"')
+    '%(t)s [%(p)-8s] GNCRN %(h)-15s rqt:%(L)s %(s)s %(b)-6s "%(m)s:%(U)s %(q)s" usr:%(u)s "%(f)s" "%(a)s"')
 
 # self adjust workers based on CPU count
 # workers = multiprocessing.cpu_count() * 2 + 1
 
 
 def post_fork(server, worker):
     server.log.info("[<%-10s>] WORKER spawned", worker.pid)
 
 
 def pre_fork(server, worker):
     pass
 
 
 def pre_exec(server):
     server.log.info("Forked child, re-executing.")
 
 
 def on_starting(server):
     server.log.info("Server is starting.")
 
 
 def when_ready(server):
     server.log.info("Server is ready. Spawning workers")
 
 
 def on_reload(server):
     pass
 
 
 def worker_int(worker):
     worker.log.info("[<%-10s>] worker received INT or QUIT signal", worker.pid)
 
     # get traceback info, on worker crash
     id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
     code = []
     for thread_id, stack in sys._current_frames().items():
         code.append(
             "\n# Thread: %s(%d)" % (id2name.get(thread_id, ""), thread_id))
         for fname, lineno, name, line in traceback.extract_stack(stack):
             code.append('File: "%s", line %d, in %s' % (fname, lineno, name))
             if line:
                 code.append("  %s" % (line.strip()))
     worker.log.debug("\n".join(code))
 
 
 def worker_abort(worker):
     worker.log.info("[<%-10s>] worker received SIGABRT signal", worker.pid)
 
 
 def worker_exit(server, worker):
     worker.log.info("[<%-10s>] worker exit", worker.pid)
 
 
 def child_exit(server, worker):
     worker.log.info("[<%-10s>] worker child exit", worker.pid)
 
 
 def pre_request(worker, req):
-    return
-    worker.log.debug("[<%-10s>] PRE WORKER: %s %s",
-                     worker.pid, req.method, req.path)
+    worker.start_time = time.time()
+    worker.log.debug(
+        "GNCRN PRE  WORKER [cnt:%s]: %s %s", worker.nr, req.method, req.path)
 
 
 def post_request(worker, req, environ, resp):
-    return
-    worker.log.debug("[<%-10s>] POST WORKER: %s %s resp: %s", worker.pid,
-                     req.method, req.path, resp.status_code)
-
+    total_time = time.time() - worker.start_time
+    worker.log.debug(
+        "GNCRN POST WORKER [cnt:%s]: %s %s resp: %s, Load Time: %.3fs",
+        worker.nr, req.method, req.path, resp.status_code, total_time)
 
 
 class RhodeCodeLogger(Logger):
     """
     Custom Logger that allows some customization that gunicorn doesn't allow
     """
 
     datefmt = r"%Y-%m-%d %H:%M:%S"
 
     def __init__(self, cfg):
         Logger.__init__(self, cfg)
 
     def now(self):
         """ return date in RhodeCode Log format """
         now = time.time()
         msecs = int((now - long(now)) * 1000)
         return time.strftime(self.datefmt, time.localtime(now)) + '.{0:03d}'.format(msecs)
 
 
 logger_class = RhodeCodeLogger

configs/production.ini

@@ -1,692 +1,689 b''
 
 
 ################################################################################
 ##                 RHODECODE COMMUNITY EDITION CONFIGURATION                  ##
 # The %(here)s variable will be replaced with the parent directory of this file#
 ################################################################################
 
 [DEFAULT]
 debug = true
 
 ################################################################################
 ##                            EMAIL CONFIGURATION                             ##
 ## Uncomment and replace with the email address which should receive          ##
 ## any error reports after an application crash                               ##
 ## Additionally these settings will be used by the RhodeCode mailing system   ##
 ################################################################################
 
 ## prefix all emails subjects with given prefix, helps filtering out emails
 #email_prefix = [RhodeCode]
 
 ## email FROM address all mails will be sent
 #app_email_from = rhodecode-noreply@localhost
 
 ## Uncomment and replace with the address which should receive any error report
 ## note: using appenlight for error handling doesn't need this to be uncommented
 #email_to = admin@localhost
 
 ## in case of Application errors, sent an error email form
 #error_email_from = rhodecode_error@localhost
 
 ## additional error message to be send in case of server crash
 #error_message =
 
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 ## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
 #smtp_auth =
 
 [server:main]
 ## COMMON ##
 host = 127.0.0.1
 port = 5000
 
 ##################################
 ##     WAITRESS WSGI SERVER     ##
 ## Recommended for Development  ##
 ##################################
 
 #use = egg:waitress#main
 ## number of worker threads
 #threads = 5
 ## MAX BODY SIZE 100GB
 #max_request_body_size = 107374182400
 ## Use poll instead of select, fixes file descriptors limits problems.
 ## May not work on old windows systems.
 #asyncore_use_poll = true
 
 
 ##########################
 ## GUNICORN WSGI SERVER ##
 ##########################
 ## run with gunicorn --log-config rhodecode.ini --paste rhodecode.ini
 
 use = egg:gunicorn#main
 ## Sets the number of process workers. You must set `instance_id = *`
 ## when this option is set to more than one worker, recommended
 ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
 ## The `instance_id = *` must be set in the [app:main] section below
 workers = 2
 ## number of threads for each of the worker, must be set to 1 for gevent
 ## generally recommended to be at 1
 #threads = 1
 ## process name
 proc_name = rhodecode
 ## type of worker class, one of sync, gevent
 ## recommended for bigger setup is using of of other than sync one
 worker_class = gevent
 ## The maximum number of simultaneous clients. Valid only for Gevent
 #worker_connections = 10
 ## max number of requests that worker will handle before being gracefully
 ## restarted, could prevent memory leaks
 max_requests = 1000
 max_requests_jitter = 30
 ## amount of time a worker can spend with handling a request before it
 ## gets killed and restarted. Set to 6hrs
 timeout = 21600
 
 
 ## prefix middleware for RhodeCode.
 ## recommended when using proxy setup.
 ## allows to set RhodeCode under a prefix in server.
 ## eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
 ## And set your prefix like: `prefix = /custom_prefix`
 ## be sure to also set beaker.session.cookie_path = /custom_prefix if you need
 ## to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
 
 [app:main]
 use = egg:rhodecode-enterprise-ce
 
 ## enable proxy prefix middleware, defined above
 #filter-with = proxy-prefix
 
 ## encryption key used to encrypt social plugin tokens,
 ## remote_urls with credentials etc, if not set it defaults to
 ## `beaker.session.secret`
 #rhodecode.encrypted_values.secret =
 
 ## decryption strict mode (enabled by default). It controls if decryption raises
 ## `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
 ## return gzipped responses from Rhodecode (static files/application)
 gzip_responses = false
 
 ## autogenerate javascript routes file on startup
 generate_js_files = false
 
 ## Optional Languages
 ## en(default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ## perform a full repository scan on each server start, this should be
 ## set to false after first startup, to allow faster server restarts.
 startup.import_repos = false
 
 ## Uncomment and set this path to use archive download cache.
 ## Once enabled, generated archives will be cached at this location
 ## and served from the cache during subsequent requests for the same archive of
 ## the repository.
 #archive_cache_dir = /tmp/tarballcache
 
 ## URL at which the application is running. This is used for bootstraping
 ## requests in context when no web request is available. Used in ishell, or
 ## SSH calls. Set this for events to receive proper url for SSH calls.
 app.base_url = http://rhodecode.local
 
 ## change this to unique ID for security
 app_instance_uuid = rc-production
 
 ## cut off limit for large diffs (size in bytes). If overall diff size on
 ## commit, or pull request exceeds this limit this diff will be displayed
 ## partially. E.g 512000 == 512Kb
 cut_off_limit_diff = 512000
 
 ## cut off limit for large files inside diffs (size in bytes). Each individual
 ## file inside diff which exceeds this limit will be displayed partially.
 ##  E.g 128000 == 128Kb
 cut_off_limit_file = 128000
 
 ## use cache version of scm repo everywhere
 vcs_full_cache = true
 
 ## force https in RhodeCode, fixes https redirects, assumes it's always https
 ## Normally this is controlled by proper http flags sent from http server
 force_https = false
 
 ## use Strict-Transport-Security headers
 use_htsts = false
 
-## number of commits stats will parse on each iteration
-commit_parse_limit = 25
-
 ## git rev filter option, --all is the default filter, if you need to
 ## hide all refs in changelog switch this to --branches --tags
 git_rev_filter = --branches --tags
 
 # Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ## RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ## gist URL alias, used to create nicer urls for gist. This should be an
 ## url that does rewrites to _admin/gists/{gistid}.
 ## example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ## RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ## List of views (using glob pattern syntax) that AUTH TOKENS could be
 ## used for access.
 ## Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ## came from the the logged in user who own this authentication token.
 ## Additionally @TOKEN syntaxt can be used to bound the view to specific
 ## authentication token. Such view would be only accessible when used together
 ## with this authentication token
 ##
 ## list of all views can be found under `/_admin/permissions/auth_token_access`
 ## The list should be "," separated and on a single line.
 ##
 ## Most common views to enable:
 #    RepoCommitsView:repo_commit_download
 #    RepoCommitsView:repo_commit_patch
 #    RepoCommitsView:repo_commit_raw
 #    RepoCommitsView:repo_commit_raw@TOKEN
 #    RepoFilesView:repo_files_diff
 #    RepoFilesView:repo_archivefile
 #    RepoFilesView:repo_file_raw
 #    GistView:*
 api_access_controllers_whitelist =
 
 ## default encoding used to convert from and to unicode
 ## can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ## instance-id prefix
 ## a prefix key for this instance used for cache invalidation when running
 ## multiple instances of rhodecode, make sure it's globally unique for
 ## all running rhodecode instances. Leave empty if you don't use it
 instance_id =
 
 ## Fallback authentication plugin. Set this to a plugin ID to force the usage
 ## of an authentication plugin also if it is disabled by it's settings.
 ## This could be useful if you are unable to log in to the system due to broken
 ## authentication settings. Then you can enable e.g. the internal rhodecode auth
 ## module to log in again and fix the settings.
 ##
 ## Available builtin plugin IDs (hash is part of the ID):
 ## egg:rhodecode-enterprise-ce#rhodecode
 ## egg:rhodecode-enterprise-ce#pam
 ## egg:rhodecode-enterprise-ce#ldap
 ## egg:rhodecode-enterprise-ce#jasig_cas
 ## egg:rhodecode-enterprise-ce#headers
 ## egg:rhodecode-enterprise-ce#crowd
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ## alternative return HTTP header for failed authentication. Default HTTP
 ## response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ## handling that causing a series of failed authentication calls.
 ## Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ## This will be served instead of default 401 on bad authnetication
 auth_ret_code =
 
 ## use special detection method when serving auth_ret_code, instead of serving
 ## ret_code directly, use 401 initially (Which triggers credentials prompt)
 ## and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ## locking return code. When repository is locked return this HTTP code. 2XX
 ## codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ## allows to change the repository location in settings page
 allow_repo_location_change = true
 
 ## allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ## generated license token, goto license page in RhodeCode settings to obtain
 ## new token
 license_token =
 
 ## supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 ## supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = prod
 
 ## Display extended labs settings
 labs_settings_active = true
 
 ####################################
 ###        CELERY CONFIG        ####
 ####################################
 ## run: /path/to/celery worker \
 ##     -E --beat --app rhodecode.lib.celerylib.loader \
 ##     --scheduler rhodecode.lib.celerylib.scheduler.RcScheduler \
 ##     --loglevel DEBUG --ini /path/to/rhodecode.ini
 
 use_celery = false
 
 ## connection url to the message broker (default rabbitmq)
 celery.broker_url = amqp://rabbitmq:qweqwe@localhost:5672/rabbitmqhost
 
 ## maximum tasks to execute before worker restart
 celery.max_tasks_per_child = 100
 
 ## tasks will never be sent to the queue, but executed locally instead.
 celery.task_always_eager = false
 
 ####################################
 ###         BEAKER CACHE        ####
 ####################################
 # default cache dir for templates.  Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk
 cache_dir = %(here)s/data
 
 ## locking and default file storage for Beaker. Putting this into a ramdisk
 ## can boost performance, eg. %(here)s/data_ramdisk/cache/beaker_data
 beaker.cache.data_dir = %(here)s/data/cache/beaker_data
 beaker.cache.lock_dir = %(here)s/data/cache/beaker_lock
 
 beaker.cache.regions = super_short_term, short_term, long_term, sql_cache_short, auth_plugins, repo_cache_long
 
 beaker.cache.super_short_term.type = memory
 beaker.cache.super_short_term.expire = 10
 beaker.cache.super_short_term.key_length = 256
 
 beaker.cache.short_term.type = memory
 beaker.cache.short_term.expire = 60
 beaker.cache.short_term.key_length = 256
 
 beaker.cache.long_term.type = memory
 beaker.cache.long_term.expire = 36000
 beaker.cache.long_term.key_length = 256
 
 beaker.cache.sql_cache_short.type = memory
 beaker.cache.sql_cache_short.expire = 10
 beaker.cache.sql_cache_short.key_length = 256
 
 ## default is memory cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.auth_plugins.type = ext:database
 #beaker.cache.auth_plugins.lock_dir = %(here)s/data/cache/auth_plugin_lock
 #beaker.cache.auth_plugins.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.cache.auth_plugins.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.cache.auth_plugins.sa.pool_recycle = 3600
 #beaker.cache.auth_plugins.sa.pool_size = 10
 #beaker.cache.auth_plugins.sa.max_overflow = 0
 
 beaker.cache.repo_cache_long.type = memorylru_base
 beaker.cache.repo_cache_long.max_items = 4096
 beaker.cache.repo_cache_long.expire = 2592000
 
 ## default is memorylru_base cache, configure only if required
 ## using multi-node or multi-worker setup
 #beaker.cache.repo_cache_long.type = ext:memcached
 #beaker.cache.repo_cache_long.url = localhost:11211
 #beaker.cache.repo_cache_long.expire = 1209600
 #beaker.cache.repo_cache_long.key_length = 256
 
 ####################################
 ###       BEAKER SESSION        ####
 ####################################
 
 ## .session.type is type of storage options for the session, current allowed
 ## types are file, ext:memcached, ext:database, and memory (default).
 beaker.session.type = file
 beaker.session.data_dir = %(here)s/data/sessions/data
 
 ## db based session, fast, and allows easy management over logged in users
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = production-rc-uytcxaz
 beaker.session.lock_dir = %(here)s/data/sessions/lock
 
 ## Secure encrypted cookie. Requires AES and AES python libraries
 ## you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = key_for_encryption
 #beaker.session.validate_key = validation_key
 
 ## sets session as invalid(also logging out user) if it haven not been
 ## accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 ## Path to use for the cookie. Set to prefix if you use prefix middleware
 #beaker.session.cookie_path = /custom_prefix
 
 ## uncomment for https secure cookie
 beaker.session.secure = false
 
 ## auto save the session to not to use .save()
 beaker.session.auto = false
 
 ## default cookie expiration time in seconds, set to `true` to set expire
 ## at browser close
 #beaker.session.cookie_expires = 3600
 
 ###################################
 ## SEARCH INDEXING CONFIGURATION ##
 ###################################
 ## Full text search indexer is available in rhodecode-tools under
 ## `rhodecode-tools index` command
 
 ## WHOOSH Backend, doesn't require additional services to run
 ## it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
 ########################################
 ###    CHANNELSTREAM CONFIG         ####
 ########################################
 ## channelstream enables persistent connections and live notification
 ## in the system. It's also used by the chat system
 channelstream.enabled = false
 
 ## server address for channelstream server on the backend
 channelstream.server = 127.0.0.1:9800
 
 ## location of the channelstream server from outside world
 ## use ws:// for http or wss:// for https. This address needs to be handled
 ## by external HTTP server such as Nginx or Apache
 ## see nginx/apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = secret
 channelstream.history.location = %(here)s/channelstream_history
 
 ## Internal application path that Javascript uses to connect into.
 ## If you use proxy-prefix the prefix should be added before /_channelstream
 channelstream.proxy_path = /_channelstream
 
 
 ###################################
 ##       APPENLIGHT CONFIG       ##
 ###################################
 
 ## Appenlight is tailored to work with RhodeCode, see
 ## http://appenlight.com for details how to obtain an account
 
 ## appenlight integration enabled
 appenlight = false
 
 appenlight.server_url = https://api.appenlight.com
 appenlight.api_key = YOUR_API_KEY
 #appenlight.transport_config = https://api.appenlight.com?threaded=1&timeout=5
 
 # used for JS client
 appenlight.api_public_key = YOUR_API_PUBLIC_KEY
 
 ## TWEAK AMOUNT OF INFO SENT HERE
 
 ## enables 404 error logging (default False)
 appenlight.report_404 = false
 
 ## time in seconds after request is considered being slow (default 1)
 appenlight.slow_request_time = 1
 
 ## record slow requests in application
 ## (needs to be enabled for slow datastore recording and time tracking)
 appenlight.slow_requests = true
 
 ## enable hooking to application loggers
 appenlight.logging = true
 
 ## minimum log level for log capture
 appenlight.logging.level = WARNING
 
 ## send logs only from erroneous/slow requests
 ## (saves API quota for intensive logging)
 appenlight.logging_on_error = false
 
 ## list of additonal keywords that should be grabbed from environ object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always send following info:
 ## 'REMOTE_USER', 'REMOTE_ADDR', 'SERVER_NAME', 'CONTENT_TYPE' + all keys that
 ## start with HTTP* this list be extended with additional keywords here
 appenlight.environ_keys_whitelist =
 
 ## list of keywords that should be blanked from request object
 ## can be string with comma separated list of words in lowercase
 ## (by default client will always blank keys that contain following words
 ## 'password', 'passwd', 'pwd', 'auth_tkt', 'secret', 'csrf'
 ## this list be extended with additional keywords set here
 appenlight.request_keys_blacklist =
 
 ## list of namespaces that should be ignores when gathering log entries
 ## can be string with comma separated list of namespaces
 ## (by default the client ignores own entries: appenlight_client.client)
 appenlight.log_namespace_blacklist =
 
 
 ################################################################################
 ## WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*  ##
 ## Debug mode will enable the interactive debugging tool, allowing ANYONE to  ##
 ## execute malicious code after an exception is raised.                       ##
 ################################################################################
 set debug = false
 
 
 ###########################################
 ###   MAIN RHODECODE DATABASE CONFIG    ###
 ###########################################
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
 sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 
 # see sqlalchemy docs for other advanced settings
 
 ## print the sql statements to output
 sqlalchemy.db1.echo = false
 ## recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 sqlalchemy.db1.convert_unicode = true
 
 ## the number of connections to keep open inside the connection pool.
 ## 0 indicates no limit
 #sqlalchemy.db1.pool_size = 5
 
 ## the number of connections to allow in connection pool "overflow", that is
 ## connections that can be opened above and beyond the pool_size setting,
 ## which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 
 ##################
 ### VCS CONFIG ###
 ##################
 vcs.server.enable = true
 vcs.server = localhost:9900
 
 ## Web server connectivity protocol, responsible for web based VCS operatations
 ## Available protocols are:
 ## `http` - use http-rpc backend (default)
 vcs.server.protocol = http
 
 ## Push/Pull operations protocol, available options are:
 ## `http` - use http-rpc backend (default)
 ##
 vcs.scm_app_implementation = http
 
 ## Push/Pull operations hooks protocol, available options are:
 ## `http` - use http-rpc backend (default)
 vcs.hooks.protocol = http
 
 vcs.server.log_level = info
 ## Start VCSServer with this instance as a subprocess, usefull for development
 vcs.start_server = false
 
 ## List of enabled VCS backends, available options are:
 ## `hg`  - mercurial
 ## `git` - git
 ## `svn` - subversion
 vcs.backends = hg, git, svn
 
 vcs.connection_timeout = 3600
 ## Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ## Available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible, pre-1.9-compatible
 #vcs.svn.compatible_version = pre-1.8-compatible
 
 
 ############################################################
 ### Subversion proxy support (mod_dav_svn)               ###
 ### Maps RhodeCode repo groups into SVN paths for Apache ###
 ############################################################
 ## Enable or disable the config file generation.
 svn.proxy.generate_config = false
 ## Generate config file with `SVNListParentPath` set to `On`.
 svn.proxy.list_parent_path = true
 ## Set location and file name of generated config file.
 svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
 ## alternative mod_dav config template. This needs to be a mako template
 #svn.proxy.config_template = ~/.rccontrol/enterprise-1/custom_svn_conf.mako
 ## Used as a prefix to the `Location` block in the generated config file.
 ## In most cases it should be set to `/`.
 svn.proxy.location_root = /
 ## Command to reload the mod dav svn configuration on change.
 ## Example: `/etc/init.d/apache2 reload`
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 ## If the timeout expires before the reload command finishes, the command will
 ## be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
 #svn.proxy.reload_timeout = 10
 
 ############################################################
 ### SSH Support Settings                                 ###
 ############################################################
 
 ## Defines if a custom authorized_keys file should be created and written on
 ## any change user ssh keys. Setting this to false also disables posibility
 ## of adding SSH keys by users from web interface. Super admins can still
 ## manage SSH Keys.
 ssh.generate_authorized_keyfile = false
 
 ## Options for ssh, default is `no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding`
 # ssh.authorized_keys_ssh_opts =
 
 ## Path to the authrozied_keys file where the generate entries are placed.
 ## It is possible to have multiple key files specified in `sshd_config` e.g.
 ## AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/authorized_keys_rhodecode
 ssh.authorized_keys_file_path = ~/.ssh/authorized_keys_rhodecode
 
 ## Command to execute the SSH wrapper. The binary is available in the
 ## rhodecode installation directory.
 ## e.g ~/.rccontrol/community-1/profile/bin/rc-ssh-wrapper
 ssh.wrapper_cmd = ~/.rccontrol/community-1/rc-ssh-wrapper
 
 ## Allow shell when executing the ssh-wrapper command
 ssh.wrapper_cmd_allow_shell = false
 
 ## Enables logging, and detailed output send back to the client during SSH
 ## operations. Usefull for debugging, shouldn't be used in production.
 ssh.enable_debug_logging = false
 
 ## Paths to binary executable, by default they are the names, but we can
 ## override them if we want to use a custom one
 ssh.executable.hg = ~/.rccontrol/vcsserver-1/profile/bin/hg
 ssh.executable.git = ~/.rccontrol/vcsserver-1/profile/bin/git
 ssh.executable.svn = ~/.rccontrol/vcsserver-1/profile/bin/svnserve
 
 
 ## Dummy marker to add new entries after.
 ## Add any custom entries below. Please don't remove.
 custom.conf = 1
 
 
 ################################
 ### LOGGING CONFIGURATION   ####
 ################################
 [loggers]
 keys = root, sqlalchemy, beaker, rhodecode, ssh_wrapper, celery
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, color_formatter, color_formatter_sql
 
 #############
 ## LOGGERS ##
 #############
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_ssh_wrapper]
 level = DEBUG
 handlers =
 qualname = ssh_wrapper
 propagate = 1
 
 [logger_celery]
 level = DEBUG
 handlers =
 qualname = celery
 
 
 ##############
 ## HANDLERS ##
 ##############
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr, )
 level = INFO
 formatter = generic
 
 [handler_console_sql]
 class = StreamHandler
 args = (sys.stderr, )
 level = WARN
 formatter = generic
 
 ################
 ## FORMATTERS ##
 ################
 
 [formatter_generic]
 class = rhodecode.lib.logging_formatter.ExceptionAwareFormatter
-format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
+format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
-format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
+format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S

docs/admin/sec-sophos-umc.rst

@@ -1,101 +1,101 b''
-.. _sec-your-server:
+.. _sec-sophos-umc:
 
 Securing Your Server via Sophos UTM 9
 -------------------------------------
 
 
 
 Below is an example configuration for Sophos UTM 9 Webserver Protection::
 
     Sophos UTM 9 Webserver Protection
     Web Application Firewall based on apache2 modesecurity2
     --------------------------------------------------
     1. Firewall Profiles -> Firewall Profile
     --------------------------------------------------
     Name: RhodeCode (can be anything)
     Mode: Reject
     Hardening & Signing:
         [ ] Static URL hardeninig
         [ ] Form hardening
         [x] Cookie Signing
     Filtering:
         [x] Block clients with bad reputation
         [x] Common Threats Filter
         [ ] Rigid Filtering
             Skip Filter Rules:
                 960015
                 950120
                 981173
                 970901
                 960010
                 960032
                 960035
                 958291
                 970903
                 970003
     Common Threat Filter Categories:
         [x] Protocol violations
         [x] Protocol anomalies
         [x] Request limit
         [x] HTTP policy
         [x] Bad robots
         [x] Generic attacks
         [x] SQL injection attacks
         [x] XSS attacks
         [x] Tight security
         [x] Trojans
         [x] Outbound
     Scanning:
         [ ] Enable antivirus scanning
         [ ] Block uploads by MIME type
     --------------------------------------------------
     2. Web Application Firewall -> Real Webservers
     --------------------------------------------------
     Name: RhodeCode (can be anything)
     Host: Your RhodeCode-Server (UTM object)
     Type: Encrypted (HTTPS)
     Port: 443
     --------------------------------------------------
     3. Web Application Firewall -> Virual Webservers
     --------------------------------------------------
     Name: RhodeCode (can be anything)
     Interface: WAN (your WAN interface)
     Type: Encrypted (HTTPS) & redirect
     Certificate: Wildcard or matching domain certificate
         Domains (in case of Wildcard certificate):
             rhodecode.yourcompany.com (match your DNS configuration)
             gist.yourcompany.com (match your DNS & RhodeCode configuration)
     Real Webservers for path '/':
         [x] RhodeCode (created in step 2)
     Firewall: RhodeCode (created in step 1)
     --------------------------------------------------
     4. Firewall Profiles -> Exceptions
     --------------------------------------------------
     Name: RhodeCode exceptions (can be anything)
     Skip these checks:
         [ ] Cookie signing
         [ ] Static URL Hardening
         [ ] Form hardening
         [x] Antivirus scanning
         [x] True file type control
         [ ] Block clients with bad reputation
     Skip these categories:
         [ ] Protocol violations
         [x] Protocol anomalies
         [x] Request limits
         [ ] HTTP policy
         [ ] Bad robots
         [ ] Generic attacks
         [ ] SQL injection attacks
         [ ] XSS attacks
         [ ] Tight security
         [ ] Trojans
         [x] Outbound
     Virtual Webservers:
         [x] RhodeCode (created in step 3)
     For All Requests:
         Web requests matching this pattern:
             /_channelstream/ws
             /Repository1/*
             /Repository2/*
             /Repository3/*
  No newline at end of file

docs/admin/system-admin.rst

@@ -1,31 +1,32 b''
 .. _rhodecode-admin-ref:
 
 System Administration
 =====================
 
 The following are the most common system administration tasks.
 
 .. only:: latex
 
     * :ref:`vcs-server`
     * :ref:`apache-ws-ref`
     * :ref:`nginx-ws-ref`
     * :ref:`rhodecode-tuning-ref`
     * :ref:`indexing-ref`
     * :ref:`rhodecode-reset-ref`
 
 .. toctree::
 
     config-files-overview
     vcs-server
     svn-http
+    svn-path-permissions
     gunicorn-ssl-support
     apache-config
     nginx-config
     backup-restore
     tuning-rhodecode
     indexing
     reset-information
     enable-debug
     admin-tricks
     cleanup-cmds

docs/api/api.rst

@@ -1,207 +1,208 b''
 .. _api:
 
 API Documentation
 =================
 
 The |RCE| API uses a single scheme for calling all API methods. The API is
 implemented with JSON protocol in both directions. To send API requests to
 your instance of |RCE|, use the following URL format
 ``<your_server>/_admin``
 
 .. note::
 
    To use the API, you should configure the :file:`~/.rhoderc` file with
    access details per instance. For more information, see
    :ref:`config-rhoderc`.
 
 
 API ACCESS FOR WEB VIEWS
 ------------------------
 
 API access can also be turned on for each web view in |RCE| that is
 decorated with a `@LoginRequired` decorator. To enable API access, change
 the standard login decorator to `@LoginRequired(api_access=True)`.
 
 From |RCM| version 1.7.0 you can configure a white list
 of views that have API access enabled by default. To enable these,
 edit the |RCM| configuration ``.ini`` file. The default location is:
 
 * |RCM| Pre-2.2.7 :file:`root/rhodecode/data/production.ini`
 * |RCM| 3.0 :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
 
 To configure the white list, edit this section of the file. In this
 configuration example, API access is granted to the patch/diff raw file and
 archive.
 
 .. code-block:: ini
 
     ## List of controllers (using glob syntax) that AUTH TOKENS could be used for access.
     ## Adding ?auth_token = <token> to the url authenticates this request as if it
     ## came from the the logged in user who own this authentication token.
     ##
     ## Syntax is <ControllerClass>:<function_pattern>.
     ## The list should be "," separated and on a single line.
     ##
     api_access_controllers_whitelist = RepoCommitsView:repo_commit_raw,RepoCommitsView:repo_commit_patch,RepoCommitsView:repo_commit_download
 
 After this change, a |RCE| view can be accessed without login by adding a
 GET parameter ``?auth_token=<auth_token>`` to a url. For example to
 access the raw diff.
 
 .. code-block:: html
 
    http://<server>/<repo>/changeset-diff/<sha>?auth_token=<auth_token>
 
 By default this is only enabled on RSS/ATOM feed views. Exposing raw diffs is a
 good way to integrate with 3rd party services like code review, or build farms
 that could download archives.
 
 API ACCESS
 ----------
 
 All clients are required to send JSON-RPC spec JSON data.
 
 .. code-block:: bash
 
     {
         "id:"<id>",
         "auth_token":"<auth_token>",
         "method":"<method_name>",
         "args":{"<arg_key>":"<arg_val>"}
     }
 
 Example call for auto pulling from remote repositories using curl:
 
 .. code-block:: bash
 
     curl https://server.com/_admin/api -X POST -H 'content-type:text/plain' --data-binary '{"id":1,
     "auth_token":"xe7cdb2v278e4evbdf5vs04v832v0efvcbcve4a3","method":"pull", "args":{"repoid":"CPython"}}'
 
 Provide those parameters:
  - **id** A value of any type, which is used to match the response with the
    request that it is replying to.
  - **auth_token** for access and permission validation.
  - **method** is name of method to call
  - **args** is an ``key:value`` list of arguments to pass to method
 
 .. note::
 
     To get your |authtoken|, from the |RCE| interface,
     go to:
     :menuselection:`username --> My account --> Auth tokens`
 
     For security reasons you should always create a dedicated |authtoken| for
     API use only.
 
 
 The |RCE| API will always return a JSON-RPC response:
 
 .. code-block:: bash
 
     {
         "id": <id>, # matching id sent by request
         "result": "<result>"|null, # JSON formatted result, null if any errors
         "error": "null"|<error_message> # JSON formatted error (if any)
     }
 
 All responses from API will be with `HTTP/1.0 200 OK` status code.
 If there is an error when calling the API, the *error* key will contain a
 failure description and the *result* will be `null`.
 
 API CLIENT
 ----------
 
 To install the |RCE| API, see :ref:`install-tools`. To configure the API per
 instance, see the :ref:`rc-tools` section as you need to configure a
 :file:`~/.rhoderc` file with your |authtokens|.
 
 Once you have set up your instance API access, use the following examples to
 get started.
 
 .. code-block:: bash
 
     # Getting the 'rhodecode' repository
     # from a RhodeCode Enterprise instance
     rhodecode-api --instance-name=enterprise-1 get_repo repoid:rhodecode
 
     Calling method get_repo => http://127.0.0.1:5000
     Server response
     {
         <json data>
     }
 
     # Creating a new mercurial repository called 'brand-new'
     # with a description 'Repo-description'
     rhodecode-api --instance-name=enterprise-1 create_repo repo_name:brand-new repo_type:hg description:Repo-description
     {
       "error": null,
       "id": 1110,
       "result": {
         "msg": "Created new repository `brand-new`",
         "success": true,
         "task": null
       }
     }
 
 A broken example, what not to do.
 
 .. code-block:: bash
 
     # A call missing the required arguments
     # and not specifying the instance
     rhodecode-api get_repo
 
     Calling method get_repo => http://127.0.0.1:5000
     Server response
     "Missing non optional `repoid` arg in JSON DATA"
 
 You can specify pure JSON using the ``--format`` parameter.
 
 .. code-block:: bash
 
     rhodecode-api --format=json get_repo repoid:rhodecode
 
 In such case only output that this function shows is pure JSON, we can use that
 and pipe output to some json formatter.
 
 If output is in pure JSON format, you can pipe output to a JSON formatter.
 
 .. code-block:: bash
 
     rhodecode-api --instance-name=enterprise-1 --format=json get_repo repoid:rhodecode | python -m json.tool
 
 API METHODS
 -----------
 
 Each method by default required following arguments.
 
 .. code-block:: bash
 
     id :      "<id_for_response>"
     auth_token : "<auth_token>"
     method :  "<method name>"
     args :    {}
 
 Use each **param** from docs and put it in args, Optional parameters
 are not required in args.
 
 .. code-block:: bash
 
     args: {"repoid": "rhodecode"}
 
 .. Note: From this point on things are generated by the script in
    `scripts/fabfile.py`. To change things below, update the docstrings in the
    ApiController.
 
 .. --- API DEFS MARKER ---
 .. toctree::
 
+   methods/views
    methods/license-methods
    methods/deprecated-methods
    methods/gist-methods
    methods/pull-request-methods
    methods/repo-methods
    methods/repo-group-methods
    methods/server-methods
    methods/user-methods
    methods/user-group-methods

docs/api/methods/repo-methods.rst

@@ -1,1024 +1,1028 b''
 .. _repo-methods-ref:
 
 repo methods
 ============
 
 add_field_to_repo 
 -----------------
 
 .. py:function:: add_field_to_repo(apiuser, repoid, key, label=<Optional:''>, description=<Optional:''>)
 
    Adds an extra field to a repository.
 
    This command can only be run using an |authtoken| with at least
    write permissions to the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository id.
    :type repoid: str or int
    :param key: Create a unique field key for this repository.
    :type key: str
    :param label:
    :type label: Optional(str)
    :param description:
    :type description: Optional(str)
 
 
 comment_commit 
 --------------
 
 .. py:function:: comment_commit(apiuser, repoid, commit_id, message, status=<Optional:None>, comment_type=<Optional:u'note'>, resolves_comment_id=<Optional:None>, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Set a commit comment, and optionally change the status of the commit.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository ID.
    :type repoid: str or int
    :param commit_id: Specify the commit_id for which to set a comment.
    :type commit_id: str
    :param message: The comment text.
    :type message: str
    :param status: (**Optional**) status of commit, one of: 'not_reviewed',
        'approved', 'rejected', 'under_review'
    :type status: str
    :param comment_type: Comment type, one of: 'note', 'todo'
    :type comment_type: Optional(str), default: 'note'
    :param userid: Set the user name of the comment creator.
    :type userid: Optional(str or int)
 
    Example error output:
 
    .. code-block:: bash
 
        {
            "id" : <id_given_in_input>,
            "result" : {
                "msg": "Commented on commit `<commit_id>` for repository `<repoid>`",
                "status_change": null or <status>,
                "success": true
            },
            "error" :  null
        }
 
 
 create_repo 
 -----------
 
-.. py:function:: create_repo(apiuser, repo_name, repo_type, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, enable_statistics=<Optional:False>, enable_locking=<Optional:False>, enable_downloads=<Optional:False>, copy_permissions=<Optional:False>)
+.. py:function:: create_repo(apiuser, repo_name, repo_type, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, push_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, enable_statistics=<Optional:False>, enable_locking=<Optional:False>, enable_downloads=<Optional:False>, copy_permissions=<Optional:False>)
 
    Creates a repository.
 
    * If the repository name contains "/", repository will be created inside
      a repository group or nested repository groups
 
      For example "foo/bar/repo1" will create |repo| called "repo1" inside
      group "foo/bar". You have to have permissions to access and write to
      the last repository group ("bar" in this example)
 
    This command can only be run using an |authtoken| with at least
    permissions to create repositories, or write permissions to
    parent repository groups.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repo_name: Set the repository name.
    :type repo_name: str
    :param repo_type: Set the repository type; 'hg','git', or 'svn'.
    :type repo_type: str
    :param owner: user_id or username
    :type owner: Optional(str)
    :param description: Set the repository description.
    :type description: Optional(str)
    :param private: set repository as private
    :type private: bool
    :param clone_uri: set clone_uri
    :type clone_uri: str
+   :param push_uri: set push_uri
+   :type push_uri: str
    :param landing_rev: <rev_type>:<rev>
    :type landing_rev: str
    :param enable_locking:
    :type enable_locking: bool
    :param enable_downloads:
    :type enable_downloads: bool
    :param enable_statistics:
    :type enable_statistics: bool
    :param copy_permissions: Copy permission from group in which the
        repository is being created.
    :type copy_permissions: bool
 
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "msg": "Created new repository `<reponame>`",
                  "success": true,
                  "task": "<celery task id or None if done sync>"
                }
        error:  null
 
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
         'failed to create repository `<repo_name>`'
      }
 
 
 delete_repo 
 -----------
 
 .. py:function:: delete_repo(apiuser, repoid, forks=<Optional:''>)
 
    Deletes a repository.
 
    * When the `forks` parameter is set it's possible to detach or delete
      forks of deleted repository.
 
    This command can only be run using an |authtoken| with admin
    permissions on the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository ID.
    :type repoid: str or int
    :param forks: Set to `detach` or `delete` forks from the |repo|.
    :type forks: Optional(str)
 
    Example error output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "msg": "Deleted repository `<reponame>`",
                  "success": true
                }
        error:  null
 
 
 fork_repo 
 ---------
 
 .. py:function:: fork_repo(apiuser, repoid, fork_name, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, copy_permissions=<Optional:False>)
 
    Creates a fork of the specified |repo|.
 
    * If the fork_name contains "/", fork will be created inside
      a repository group or nested repository groups
 
      For example "foo/bar/fork-repo" will create fork called "fork-repo"
      inside group "foo/bar". You have to have permissions to access and
      write to the last repository group ("bar" in this example)
 
    This command can only be run using an |authtoken| with minimum
    read permissions of the forked repo, create fork permissions for an user.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set repository name or repository ID.
    :type repoid: str or int
    :param fork_name: Set the fork name, including it's repository group membership.
    :type fork_name: str
    :param owner: Set the fork owner.
    :type owner: str
    :param description: Set the fork description.
    :type description: str
    :param copy_permissions: Copy permissions from parent |repo|. The
        default is False.
    :type copy_permissions: bool
    :param private: Make the fork private. The default is False.
    :type private: bool
    :param landing_rev: Set the landing revision. The default is tip.
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_for_response>
        api_key : "<api_key>"
        args:     {
                    "repoid" :          "<reponame or repo_id>",
                    "fork_name":        "<forkname>",
                    "owner":            "<username or user_id = Optional(=apiuser)>",
                    "description":      "<description>",
                    "copy_permissions": "<bool>",
                    "private":          "<bool>",
                    "landing_rev":      "<landing_rev>"
                  }
 
    Example error output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "msg": "Created fork of `<reponame>` as `<forkname>`",
                  "success": true,
                  "task": "<celery task id or None if done sync>"
                }
        error:  null
 
 
 get_repo 
 --------
 
 .. py:function:: get_repo(apiuser, repoid, cache=<Optional:True>)
 
    Gets an existing repository by its name or repository_id.
 
    The members section so the output returns users groups or users
    associated with that repository.
 
    This command can only be run using an |authtoken| with admin rights,
    or users with at least read rights to the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository id.
    :type repoid: str or int
    :param cache: use the cached value for last changeset
    :type: cache: Optional(bool)
 
    Example output:
 
    .. code-block:: bash
 
        {
          "error": null,
          "id": <repo_id>,
          "result": {
            "clone_uri": null,
            "created_on": "timestamp",
            "description": "repo description",
            "enable_downloads": false,
            "enable_locking": false,
            "enable_statistics": false,
            "followers": [
              {
                "active": true,
                "admin": false,
                "api_key": "****************************************",
                "api_keys": [
                  "****************************************"
                ],
                "email": "[e-mail]",
                "emails": [
                  "[e-mail]"
                ],
                "extern_name": "rhodecode",
                "extern_type": "rhodecode",
                "firstname": "username",
                "ip_addresses": [],
                "language": null,
                "last_login": "2015-09-16T17:16:35.854",
                "lastname": "surname",
                "user_id": <user_id>,
                "username": "name"
              }
            ],
            "fork_of": "parent-repo",
            "landing_rev": [
              "rev",
              "tip"
            ],
            "last_changeset": {
              "author": "User <[e-mail]>",
              "branch": "default",
              "date": "timestamp",
              "message": "last commit message",
              "parents": [
                {
                  "raw_id": "commit-id"
                }
              ],
              "raw_id": "commit-id",
              "revision": <revision number>,
              "short_id": "short id"
            },
            "lock_reason": null,
            "locked_by": null,
            "locked_date": null,
            "owner": "owner-name",
            "permissions": [
              {
                "name": "super-admin-name",
                "origin": "super-admin",
                "permission": "repository.admin",
                "type": "user"
              },
              {
                "name": "owner-name",
                "origin": "owner",
                "permission": "repository.admin",
                "type": "user"
              },
              {
                "name": "user-group-name",
                "origin": "permission",
                "permission": "repository.write",
                "type": "user_group"
              }
            ],
            "private": true,
            "repo_id": 676,
            "repo_name": "user-group/repo-name",
            "repo_type": "hg"
          }
        }
 
 
 get_repo_changeset 
 ------------------
 
 .. py:function:: get_repo_changeset(apiuser, repoid, revision, details=<Optional:'basic'>)
 
    Returns information about a changeset.
 
    Additionally parameters define the amount of details returned by
    this function.
 
    This command can only be run using an |authtoken| with admin rights,
    or users with at least read rights to the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository id
    :type repoid: str or int
    :param revision: revision for which listing should be done
    :type revision: str
    :param details: details can be 'basic|extended|full' full gives diff
        info details like the diff itself, and number of changed files etc.
    :type details: Optional(str)
 
 
 get_repo_changesets 
 -------------------
 
 .. py:function:: get_repo_changesets(apiuser, repoid, start_rev, limit, details=<Optional:'basic'>)
 
    Returns a set of commits limited by the number starting
    from the `start_rev` option.
 
    Additional parameters define the amount of details returned by this
    function.
 
    This command can only be run using an |authtoken| with admin rights,
    or users with at least read rights to |repos|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
    :param start_rev: The starting revision from where to get changesets.
    :type start_rev: str
    :param limit: Limit the number of commits to this amount
    :type limit: str or int
    :param details: Set the level of detail returned. Valid option are:
        ``basic``, ``extended`` and ``full``.
    :type details: Optional(str)
 
    .. note::
 
       Setting the parameter `details` to the value ``full`` is extensive
       and returns details like the diff itself, and the number
       of changed files.
 
 
 get_repo_nodes 
 --------------
 
 .. py:function:: get_repo_nodes(apiuser, repoid, revision, root_path, ret_type=<Optional:'all'>, details=<Optional:'basic'>, max_file_bytes=<Optional:None>)
 
    Returns a list of nodes and children in a flat list for a given
    path at given revision.
 
    It's possible to specify ret_type to show only `files` or `dirs`.
 
    This command can only be run using an |authtoken| with admin rights,
    or users with at least read rights to |repos|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
    :param revision: The revision for which listing should be done.
    :type revision: str
    :param root_path: The path from which to start displaying.
    :type root_path: str
    :param ret_type: Set the return type. Valid options are
        ``all`` (default), ``files`` and ``dirs``.
    :type ret_type: Optional(str)
    :param details: Returns extended information about nodes, such as
        md5, binary, and or content.  The valid options are ``basic`` and
        ``full``.
    :type details: Optional(str)
    :param max_file_bytes: Only return file content under this file size bytes
    :type details: Optional(int)
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: [
                  {
                    "name" : "<name>"
                    "type" : "<type>",
                    "binary": "<true|false>" (only in extended mode)
                    "md5"  : "<md5 of file content>" (only in extended mode)
                  },
                  ...
                ]
        error:  null
 
 
 get_repo_refs 
 -------------
 
 .. py:function:: get_repo_refs(apiuser, repoid)
 
    Returns a dictionary of current references. It returns
    bookmarks, branches, closed_branches, and tags for given repository
 
    It's possible to specify ret_type to show only `files` or `dirs`.
 
    This command can only be run using an |authtoken| with admin rights,
    or users with at least read rights to |repos|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        "result": {
            "bookmarks": {
              "dev": "5611d30200f4040ba2ab4f3d64e5b06408a02188",
              "master": "367f590445081d8ec8c2ea0456e73ae1f1c3d6cf"
            },
            "branches": {
              "default": "5611d30200f4040ba2ab4f3d64e5b06408a02188",
              "stable": "367f590445081d8ec8c2ea0456e73ae1f1c3d6cf"
            },
            "branches_closed": {},
            "tags": {
              "tip": "5611d30200f4040ba2ab4f3d64e5b06408a02188",
              "v4.4.0": "1232313f9e6adac5ce5399c2a891dc1e72b79022",
              "v4.4.1": "cbb9f1d329ae5768379cdec55a62ebdd546c4e27",
              "v4.4.2": "24ffe44a27fcd1c5b6936144e176b9f6dd2f3a17",
            }
        }
        error:  null
 
 
 get_repo_settings 
 -----------------
 
 .. py:function:: get_repo_settings(apiuser, repoid, key=<Optional:None>)
 
    Returns all settings for a repository. If key is given it only returns the
    setting identified by the key or null.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository id.
    :type repoid: str or int
    :param key: Key of the setting to return.
    :type: key: Optional(str)
 
    Example output:
 
    .. code-block:: bash
 
        {
            "error": null,
            "id": 237,
            "result": {
                "extensions_largefiles": true,
                "extensions_evolve": true,
                "hooks_changegroup_push_logger": true,
                "hooks_changegroup_repo_size": false,
                "hooks_outgoing_pull_logger": true,
                "phases_publish": "True",
                "rhodecode_hg_use_rebase_for_merging": true,
                "rhodecode_pr_merge_enabled": true,
                "rhodecode_use_outdated_comments": true
            }
        }
 
 
 get_repos 
 ---------
 
 .. py:function:: get_repos(apiuser, root=<Optional:None>, traverse=<Optional:True>)
 
    Lists all existing repositories.
 
    This command can only be run using an |authtoken| with admin rights,
    or users with at least read rights to |repos|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param root: specify root repository group to fetch repositories.
        filters the returned repositories to be members of given root group.
    :type root: Optional(None)
    :param traverse: traverse given root into subrepositories. With this flag
        set to False, it will only return top-level repositories from `root`.
        if root is empty it will return just top-level repositories.
    :type traverse: Optional(True)
 
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: [
                  {
                    "repo_id" :          "<repo_id>",
                    "repo_name" :        "<reponame>"
                    "repo_type" :        "<repo_type>",
                    "clone_uri" :        "<clone_uri>",
                    "private": :         "<bool>",
                    "created_on" :       "<datetimecreated>",
                    "description" :      "<description>",
                    "landing_rev":       "<landing_rev>",
                    "owner":             "<repo_owner>",
                    "fork_of":           "<name_of_fork_parent>",
                    "enable_downloads":  "<bool>",
                    "enable_locking":    "<bool>",
                    "enable_statistics": "<bool>",
                  },
                  ...
                ]
        error:  null
 
 
 grant_user_group_permission 
 ---------------------------
 
 .. py:function:: grant_user_group_permission(apiuser, repoid, usergroupid, perm)
 
    Grant permission for a user group on the specified repository,
    or update existing permissions.
 
    This command can only be run using an |authtoken| with admin
    permissions on the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository ID.
    :type repoid: str or int
    :param usergroupid: Specify the ID of the user group.
    :type usergroupid: str or int
    :param perm: Set the user group permissions using the following
        format: (repository.(none|read|write|admin))
    :type perm: str
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg" : "Granted perm: `<perm>` for group: `<usersgroupname>` in repo: `<reponame>`",
        "success": true
 
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "failed to edit permission for user group: `<usergroup>` in repo `<repo>`'
      }
 
 
 grant_user_permission 
 ---------------------
 
 .. py:function:: grant_user_permission(apiuser, repoid, userid, perm)
 
    Grant permissions for the specified user on the given repository,
    or update existing permissions if found.
 
    This command can only be run using an |authtoken| with admin
    permissions on the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository ID.
    :type repoid: str or int
    :param userid: Set the user name.
    :type userid: str
    :param perm: Set the user permissions, using the following format
        ``(repository.(none|read|write|admin))``
    :type perm: str
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "msg" : "Granted perm: `<perm>` for user: `<username>` in repo: `<reponame>`",
                  "success": true
                }
        error:  null
 
 
 invalidate_cache 
 ----------------
 
 .. py:function:: invalidate_cache(apiuser, repoid, delete_keys=<Optional:False>)
 
    Invalidates the cache for the specified repository.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Sets the repository name or repository ID.
    :type repoid: str or int
    :param delete_keys: This deletes the invalidated keys instead of
        just flagging them.
    :type delete_keys: Optional(``True`` | ``False``)
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        'msg': Cache for repository `<repository name>` was invalidated,
        'repository': <repository name>
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error : {
         'Error occurred during cache invalidation action'
      }
 
 
 lock 
 ----
 
 .. py:function:: lock(apiuser, repoid, locked=<Optional:None>, userid=<Optional:<OptionalAttr:apiuser>>)
 
    Sets the lock state of the specified |repo| by the given user.
    From more information, see :ref:`repo-locking`.
 
    * If the ``userid`` option is not set, the repository is locked to the
      user who called the method.
    * If the ``locked`` parameter is not set, the current lock state of the
      repository is displayed.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Sets the repository name or repository ID.
    :type repoid: str or int
    :param locked: Sets the lock state.
    :type locked: Optional(``True`` | ``False``)
    :param userid: Set the repository lock to this user.
    :type userid: Optional(str or int)
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        'repo': '<reponame>',
        'locked': <bool: lock state>,
        'locked_since': <int: lock timestamp>,
        'locked_by': <username of person who made the lock>,
        'lock_reason': <str: reason for locking>,
        'lock_state_changed': <bool: True if lock state has been changed in this request>,
        'msg': 'Repo `<reponame>` locked by `<username>` on <timestamp>.'
        or
        'msg': 'Repo `<repository name>` not locked.'
        or
        'msg': 'User `<user name>` set lock state for repo `<repository name>` to `<new lock state>`'
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        'Error occurred locking repository `<reponame>`'
      }
 
 
 maintenance 
 -----------
 
 .. py:function:: maintenance(apiuser, repoid)
 
    Triggers a maintenance on the given repository.
 
    This command can only be run using an |authtoken| with admin
    rights to the specified repository. For more information,
    see :ref:`config-token-ref`.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": "executed maintenance command",
        "executed_actions": [
           <action_message>, <action_message2>...
        ],
        "repository": "<repository name>"
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "Unable to execute maintenance on `<reponame>`"
      }
 
 
 pull 
 ----
 
-.. py:function:: pull(apiuser, repoid)
+.. py:function:: pull(apiuser, repoid, remote_uri=<Optional:None>)
 
    Triggers a pull on the given repository from a remote location. You
    can use this to keep remote repositories up-to-date.
 
    This command can only be run using an |authtoken| with admin
    rights to the specified repository. For more information,
    see :ref:`config-token-ref`.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
+   :param remote_uri: Optional remote URI to pass in for pull
+   :type remote_uri: str
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
-       "msg": "Pulled from `<repository name>`"
+       "msg": "Pulled from url `<remote_url>` on repo `<repository name>`"
        "repository": "<repository name>"
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
-       "Unable to pull changes from `<reponame>`"
+       "Unable to push changes from `<remote_url>`"
      }
 
 
 remove_field_from_repo 
 ----------------------
 
 .. py:function:: remove_field_from_repo(apiuser, repoid, key)
 
    Removes an extra field from a repository.
 
    This command can only be run using an |authtoken| with at least
    write permissions to the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository ID.
    :type repoid: str or int
    :param key: Set the unique field key for this repository.
    :type key: str
 
 
 revoke_user_group_permission 
 ----------------------------
 
 .. py:function:: revoke_user_group_permission(apiuser, repoid, usergroupid)
 
    Revoke the permissions of a user group on a given repository.
 
    This command can only be run using an |authtoken| with admin
    permissions on the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository ID.
    :type repoid: str or int
    :param usergroupid: Specify the user group ID.
    :type usergroupid: str or int
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "msg" : "Revoked perm for group: `<usersgroupname>` in repo: `<reponame>`",
                  "success": true
                }
        error:  null
 
 
 revoke_user_permission 
 ----------------------
 
 .. py:function:: revoke_user_permission(apiuser, repoid, userid)
 
    Revoke permission for a user on the specified repository.
 
    This command can only be run using an |authtoken| with admin
    permissions on the |repo|.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: Set the repository name or repository ID.
    :type repoid: str or int
    :param userid: Set the user name of revoked user.
    :type userid: str or int
 
    Example error output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "msg" : "Revoked perm for user: `<username>` in repo: `<reponame>`",
                  "success": true
                }
        error:  null
 
 
 set_repo_settings 
 -----------------
 
 .. py:function:: set_repo_settings(apiuser, repoid, settings)
 
    Update repository settings. Returns true on success.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository id.
    :type repoid: str or int
    :param settings: The new settings for the repository.
    :type: settings: dict
 
    Example output:
 
    .. code-block:: bash
 
        {
            "error": null,
            "id": 237,
            "result": true
        }
 
 
 strip 
 -----
 
 .. py:function:: strip(apiuser, repoid, revision, branch)
 
    Strips the given revision from the specified repository.
 
    * This will remove the revision and all of its decendants.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: The repository name or repository ID.
    :type repoid: str or int
    :param revision: The revision you wish to strip.
    :type revision: str
    :param branch: The branch from which to strip the revision.
    :type branch: str
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": "'Stripped commit <commit_hash> from repo `<repository name>`'"
        "repository": "<repository name>"
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "Unable to strip commit <commit_hash> from repo `<repository name>`"
      }
 
 
 update_repo 
 -----------
 
-.. py:function:: update_repo(apiuser, repoid, repo_name=<Optional:None>, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, fork_of=<Optional:None>, enable_statistics=<Optional:False>, enable_locking=<Optional:False>, enable_downloads=<Optional:False>, fields=<Optional:''>)
+.. py:function:: update_repo(apiuser, repoid, repo_name=<Optional:None>, owner=<Optional:<OptionalAttr:apiuser>>, description=<Optional:''>, private=<Optional:False>, clone_uri=<Optional:None>, push_uri=<Optional:None>, landing_rev=<Optional:'rev:tip'>, fork_of=<Optional:None>, enable_statistics=<Optional:False>, enable_locking=<Optional:False>, enable_downloads=<Optional:False>, fields=<Optional:''>)
 
    Updates a repository with the given information.
 
    This command can only be run using an |authtoken| with at least
    admin permissions to the |repo|.
 
    * If the repository name contains "/", repository will be updated
      accordingly with a repository group or nested repository groups
 
      For example repoid=repo-test name="foo/bar/repo-test" will update |repo|
      called "repo-test" and place it inside group "foo/bar".
      You have to have permissions to access and write to the last repository
      group ("bar" in this example)
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param repoid: repository name or repository ID.
    :type repoid: str or int
    :param repo_name: Update the |repo| name, including the
        repository group it's in.
    :type repo_name: str
    :param owner: Set the |repo| owner.
    :type owner: str
    :param fork_of: Set the |repo| as fork of another |repo|.
    :type fork_of: str
    :param description: Update the |repo| description.
    :type description: str
    :param private: Set the |repo| as private. (True | False)
    :type private: bool
    :param clone_uri: Update the |repo| clone URI.
    :type clone_uri: str
    :param landing_rev: Set the |repo| landing revision. Default is ``rev:tip``.
    :type landing_rev: str
    :param enable_statistics: Enable statistics on the |repo|, (True | False).
    :type enable_statistics: bool
    :param enable_locking: Enable |repo| locking.
    :type enable_locking: bool
    :param enable_downloads: Enable downloads from the |repo|, (True | False).
    :type enable_downloads: bool
    :param fields: Add extra fields to the |repo|. Use the following
        example format: ``field_key=field_val,field_key2=fieldval2``.
        Escape ', ' with \,
    :type fields: str
 
 

docs/api/methods/user-group-methods.rst

@@ -1,412 +1,418 b''
 .. _user-group-methods-ref:
 
 user_group methods
 ==================
 
 add_user_to_user_group 
 ----------------------
 
 .. py:function:: add_user_to_user_group(apiuser, usergroupid, userid)
 
    Adds a user to a `user group`. If the user already exists in the group
    this command will return false.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified user group.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Set the name of the `user group` to which a
        user will be added.
    :type usergroupid: int
    :param userid: Set the `user_id` of the user to add to the group.
    :type userid: int
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
          "success": True|False # depends on if member is in group
          "msg": "added member `<username>` to user group `<groupname>` |
                  User is already in that group"
 
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "failed to add member to user group `<user_group_name>`"
      }
 
 
 create_user_group 
 -----------------
 
-.. py:function:: create_user_group(apiuser, group_name, description=<Optional:''>, owner=<Optional:<OptionalAttr:apiuser>>, active=<Optional:True>)
+.. py:function:: create_user_group(apiuser, group_name, description=<Optional:''>, owner=<Optional:<OptionalAttr:apiuser>>, active=<Optional:True>, sync=<Optional:None>)
 
    Creates a new user group.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param group_name: Set the name of the new user group.
    :type group_name: str
    :param description: Give a description of the new user group.
    :type description: str
    :param owner: Set the owner of the new user group.
        If not set, the owner is the |authtoken| user.
    :type owner: Optional(str or int)
    :param active: Set this group as active.
    :type active: Optional(``True`` | ``False``)
+   :param sync: Set enabled or disabled the automatically sync from
+       external authentication types like ldap.
+   :type sync: Optional(``True`` | ``False``)
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "msg": "created new user group `<groupname>`",
                  "user_group": <user_group_object>
                }
        error:  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "user group `<group name>` already exist"
        or
        "failed to create group `<group name>`"
      }
 
 
 delete_user_group 
 -----------------
 
 .. py:function:: delete_user_group(apiuser, usergroupid)
 
    Deletes the specified `user group`.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: filled automatically from apikey
    :type apiuser: AuthUser
    :param usergroupid:
    :type usergroupid: int
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": "deleted user group ID:<user_group_id> <user_group_name>"
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "failed to delete user group ID:<user_group_id> <user_group_name>"
        or
        "RepoGroup assigned to <repo_groups_list>"
      }
 
 
 get_user_group 
 --------------
 
 .. py:function:: get_user_group(apiuser, usergroupid)
 
    Returns the data of an existing user group.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Set the user group from which to return data.
    :type usergroupid: str or int
 
    Example error output:
 
    .. code-block:: bash
 
        {
          "error": null,
          "id": <id>,
          "result": {
            "active": true,
            "group_description": "group description",
            "group_name": "group name",
            "permissions": [
              {
                "name": "owner-name",
                "origin": "owner",
                "permission": "usergroup.admin",
                "type": "user"
              },
              {
              {
                "name": "user name",
                "origin": "permission",
                "permission": "usergroup.admin",
                "type": "user"
              },
              {
                "name": "user group name",
                "origin": "permission",
                "permission": "usergroup.write",
                "type": "user_group"
              }
            ],
            "permissions_summary": {
              "repositories": {
                "aa-root-level-repo-1": "repository.admin"
              },
              "repositories_groups": {}
            },
            "owner": "owner name",
            "users": [],
            "users_group_id": 2
          }
        }
 
 
 get_user_groups 
 ---------------
 
 .. py:function:: get_user_groups(apiuser)
 
    Lists all the existing user groups within RhodeCode.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    This command takes the following options:
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
 
    Example error output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result : [<user_group_obj>,...]
        error : null
 
 
 grant_user_group_permission_to_user_group 
 -----------------------------------------
 
 .. py:function:: grant_user_group_permission_to_user_group(apiuser, usergroupid, sourceusergroupid, perm)
 
    Give one user group permissions to another user group.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Set the user group on which to edit permissions.
    :type usergroupid: str or int
    :param sourceusergroupid: Set the source user group to which
        access/permissions will be granted.
    :type sourceusergroupid: str or int
    :param perm: (usergroup.(none|read|write|admin))
    :type perm: str
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": "Granted perm: `<perm_name>` for user group: `<source_user_group_name>` in user group: `<user_group_name>`",
        "success": true
      }
      error :  null
 
 
 grant_user_permission_to_user_group 
 -----------------------------------
 
 .. py:function:: grant_user_permission_to_user_group(apiuser, usergroupid, userid, perm)
 
    Set permissions for a user in a user group.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Set the user group to edit permissions on.
    :type usergroupid: str or int
    :param userid: Set the user from whom you wish to set permissions.
    :type userid: str
    :param perm: (usergroup.(none|read|write|admin))
    :type perm: str
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": "Granted perm: `<perm_name>` for user: `<username>` in user group: `<user_group_name>`",
        "success": true
      }
      error :  null
 
 
 remove_user_from_user_group 
 ---------------------------
 
 .. py:function:: remove_user_from_user_group(apiuser, usergroupid, userid)
 
    Removes a user from a user group.
 
    * If the specified user is not in the group, this command will return
      `false`.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified user group.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Sets the user group name.
    :type usergroupid: str or int
    :param userid: The user you wish to remove from |RCE|.
    :type userid: str or int
 
    Example output:
 
    .. code-block:: bash
 
        id : <id_given_in_input>
        result: {
                  "success":  True|False,  # depends on if member is in group
                  "msg": "removed member <username> from user group <groupname> |
                          User wasn't in group"
                }
        error:  null
 
 
 revoke_user_group_permission_from_user_group 
 --------------------------------------------
 
 .. py:function:: revoke_user_group_permission_from_user_group(apiuser, usergroupid, sourceusergroupid)
 
    Revoke the permissions that one user group has to another.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Set the user group on which to edit permissions.
    :type usergroupid: str or int
    :param sourceusergroupid: Set the user group from which permissions
        are revoked.
    :type sourceusergroupid: str or int
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": "Revoked perm for user group: `<user_group_name>` in user group: `<target_user_group_name>`",
        "success": true
      }
      error :  null
 
 
 revoke_user_permission_from_user_group 
 --------------------------------------
 
 .. py:function:: revoke_user_permission_from_user_group(apiuser, usergroupid, userid)
 
    Revoke a users permissions in a user group.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Set the user group from which to revoke the user
        permissions.
    :type: usergroupid: str or int
    :param userid: Set the userid of the user whose permissions will be
        revoked.
    :type userid: str
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": "Revoked perm for user: `<username>` in user group: `<user_group_name>`",
        "success": true
      }
      error :  null
 
 
 update_user_group 
 -----------------
 
-.. py:function:: update_user_group(apiuser, usergroupid, group_name=<Optional:''>, description=<Optional:''>, owner=<Optional:None>, active=<Optional:True>)
+.. py:function:: update_user_group(apiuser, usergroupid, group_name=<Optional:''>, description=<Optional:''>, owner=<Optional:None>, active=<Optional:True>, sync=<Optional:None>)
 
    Updates the specified `user group` with the details provided.
 
    This command can only be run using an |authtoken| with admin rights to
    the specified repository.
 
    :param apiuser: This is filled automatically from the |authtoken|.
    :type apiuser: AuthUser
    :param usergroupid: Set the id of the `user group` to update.
    :type usergroupid: str or int
    :param group_name: Set the new name the `user group`
    :type group_name: str
    :param description: Give a description for the `user group`
    :type description: str
    :param owner: Set the owner of the `user group`.
    :type owner: Optional(str or int)
    :param active: Set the group as active.
    :type active: Optional(``True`` | ``False``)
+   :param sync: Set enabled or disabled the automatically sync from
+       external authentication types like ldap.
+   :type sync: Optional(``True`` | ``False``)
 
    Example output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : {
        "msg": 'updated user group ID:<user group id> <user group name>',
        "user_group": <user_group_object>
      }
      error :  null
 
    Example error output:
 
    .. code-block:: bash
 
      id : <id_given_in_input>
      result : null
      error :  {
        "failed to update user group `<user group name>`"
      }
 
 

docs/auth/auth-ldap.rst

@@ -1,90 +1,89 b''
 .. _config-ldap-ref:
 
-LDAP
-----
+LDAP/AD
+-------
 
 |RCM| supports LDAP (Lightweight Directory Access Protocol) or
 AD (active Directory) authentication.
 All LDAP versions are supported, with the following |RCM| plugins managing each:
 
-* For LDAPv3 use ``LDAP (egg:rhodecode-enterprise-ce#ldap)``
-* For LDAPv3 with user group sync use ``LDAP + User Groups (egg:rhodecode-enterprise-ee#ldap_group)``
+* For LDAP or Active Directory use ``LDAP (egg:rhodecode-enterprise-ce#ldap)``
+
+RhodeCode reads all data defined from plugin and creates corresponding
+accounts on local database after receiving data from LDAP. This is done on
+every user log-in including operations like pushing/pulling/checkout.
 
 
 .. important::
 
    The email used with your |RCE| super-admin account needs to match the email
    address attached to your admin profile in LDAP. This is because
    within |RCE| the user email needs to be unique, and multiple users
    cannot share an email account.
 
    Likewise, if as an admin you also have a user account, the email address
    attached to the user account needs to be different.
 
+
 LDAP Configuration Steps
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 To configure |LDAP|, use the following steps:
 
 1. From the |RCM| interface, select
    :menuselection:`Admin --> Authentication`
-2. Enable the required plugin and select :guilabel:`Save`
+2. Enable the ldap plugin and select :guilabel:`Save`
 3. Select the :guilabel:`Enabled` check box in the plugin configuration section
 4. Add the required LDAP information and :guilabel:`Save`, for more details,
    see :ref:`config-ldap-examples`
 
 For a more detailed description of LDAP objects, see :ref:`ldap-gloss-ref`:
 
 .. _config-ldap-examples:
 
 Example LDAP configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 .. code-block:: bash
 
-        # Auth Cache TTL
+        # Auth Cache TTL, Defines the caching for authentication to offload LDAP server.
+        # This means that cache result will be saved for 3600 before contacting LDAP server to verify the user access
         3600
-        # Host
+        # Host, comma seperated format is optionally possible to specify more than 1 server
         https://ldap1.server.com/ldap-admin/,https://ldap2.server.com/ldap-admin/
-        # Port
+        # Default LDAP Port, use 689 for LDAPS
         389
-        # Account
-        cn=admin,dc=rhodecode,dc=com
-        # Password
+        # Account, used for SimpleBind if LDAP server requires an authentication
+        e.g [e-mail]
+        # Password used for simple bind
         ldap-user-password
         # LDAP connection security
         LDAPS
         # Certificate checks level
         DEMAND
         # Base DN
         cn=Rufus Magillacuddy,ou=users,dc=rhodecode,dc=com
-        # User Search Base
-        ou=groups,ou=users
-        # LDAP search filter
+        # LDAP search filter to narrow the results
         (objectClass=person)
         # LDAP search scope
         SUBTREE
         # Login attribute
-        rmagillacuddy
-        # First Name Attribute
-        Rufus
-        # Last Name Attribute
-        Magillacuddy
-        # Email Attribute
-        [e-mail]
-        # User Member of Attribute
-        Organizational Role
-        # Group search base
-        cn=users,ou=groups,dc=rhodecode,dc=com
-        # LDAP Group Search Filter
-        (objectclass=posixGroup)
-        # Group Name Attribute
-        users
-        # Group Member Of Attribute
-        cn
-        # Admin Groups
-        admin,devops,qa
+        sAMAccountName
+        # First Name Attribute to read
+        givenName
+        # Last Name Attribute to read
+        sn
+        # Email Attribute to read email address from
+        mail
+
+
+Below is example setup that can be used with Active Directory/LDAP server.
+
+.. image:: ../images/ldap-example.png
+   :alt: LDAP/AD setup example
+   :scale: 50 %
+
 
 .. toctree::
 
    ldap-active-directory
    ldap-authentication

docs/auth/auth.rst

@@ -1,37 +1,32 b''
 .. _authentication-ref:
 
 Authentication Options
 ======================
 
-|RCE| provides a built in authentication plugin
-``rhodecode.lib.auth_rhodecode``. This is enabled by default and accessed
-through the administrative interface. Additionally,
-|RCE| provides a Pluggable Authentication System (PAS). This gives the
+|RCE| provides a built in authentication against its own database. This is
+implemented using ``rhodecode.lib.auth_rhodecode`` plugin. This plugin is
+enabled by default.
+Additionally, |RCE| provides a Pluggable Authentication System. This gives the
 administrator greater control over how users authenticate with the system.
 
 .. important::
 
   You can disable the built in |RCM| authentication plugin
   ``rhodecode.lib.auth_rhodecode`` and force all authentication to go
-  through your authentication plugin. However, if you do this,
-  and your external authentication tools fails, you will be unable to
-  access |RCM|.
+  through your authentication plugin of choice e.g LDAP only.
+  However, if you do this, and your external authentication tools fails,
+  you will be unable to access |RCM|.
 
 |RCM| comes with the following user authentication management plugins:
 
-.. only:: latex
-
-    * :ref:`config-ldap-ref`
-    * :ref:`config-pam-ref`
-    * :ref:`config-crowd-ref`
-    * :ref:`config-token-ref`
 
 .. toctree::
 
-    ldap-config-steps
-    crowd-auth
-    pam-auth
-    token-auth
+    auth-ldap
+    auth-ldap-groups
+    auth-crowd
+    auth-pam
+    auth-token
     ssh-connection
 
 

docs/conf.py

@@ -1,317 +1,316 b''
 # -*- coding: utf-8 -*-
 #
 # RhodeCode Enterprise documentation build configuration file, created by
 # sphinx-quickstart on Tue Nov  4 11:48:37 2014.
 #
 # This file is execfile()d with the current directory set to its
 # containing dir.
 #
 # Note that not all possible configuration values are present in this
 # autogenerated file.
 #
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
 import sys
 import os
 import datetime
 import sphinx_rtd_theme
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath('.'))
 import common
 
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
 #needs_sphinx = '1.0'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
     'sphinx.ext.intersphinx',
     'sphinx.ext.todo',
-    'sphinx.ext.pngmath'
+    'sphinx.ext.imgmath'
 ]
 
 intersphinx_mapping = {
     'enterprise': ('https://docs.rhodecode.com/RhodeCode-Enterprise/', None),
     'control': ('https://docs.rhodecode.com/RhodeCode-Control/', None),
 }
 
 if tags.has('dev'):
     intersphinx_mapping.update({
         'enterprise': ('https://ci.rhodecode.com/documentation/Momentum/', None),
         'control': ('https://ci.rhodecode.com/documentation/Control/', None),
     })
 
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
 # The suffix of source filenames.
 source_suffix = '.rst'
 
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
 
 # The master toctree document.
 master_doc = 'index'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 
 # TODO: johbo: Move into common package for documentation utilities
 def _get_version():
     with open('../rhodecode/VERSION') as f:
         return f.read().strip()
 
 # The full version, including alpha/beta/rc tags.
 release = _get_version()
 # The short X.Y version.
 version = '.'.join(release.split('.', 2)[:2])  # First two parts of release
 
 # General information about the project.
 project = u'RhodeCode Enterprise %s ' % _get_version()
 copyright = u'2010-{now.year}, RhodeCode GmbH'.format(
     now=datetime.datetime.today())
 
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
 #language = None
 
 rst_epilog = common.rst_epilog + """
 .. |async| replace:: asynchronous
 """
 
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
 #today = ''
 # Else, today_fmt is used as the format for a strftime call.
 #today_fmt = '%B %d, %Y'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 exclude_patterns = [
     # Special directories
     '_build',
     'result',
 
     # Other RST files
     'admin/rhodecode-backup.rst',
-    'auth/ldap-configuration-example.rst',
     'issue-trackers/redmine.rst',
     'known-issues/error-msg-guide.rst',
     'tutorials/docs-build.rst',
     'integrations/example-ext.py',
     'collaboration/supported-workflows.rst',
 ]
 
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
 #default_role = None
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
 #add_function_parentheses = True
 
 # If true, the current module name will be prepended to all description
 # unit titles (such as .. function::).
 #add_module_names = True
 
 # If true, sectionauthor and moduleauthor directives will be shown in the
 # output. They are ignored by default.
 #show_authors = False
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = 'sphinx'
 
 # A list of ignored prefixes for module index sorting.
 #modindex_common_prefix = []
 
 # If true, keep warnings as "system message" paragraphs in the built documents.
 keep_warnings = tags.has("dev")
 
 
 # -- Options for HTML output ----------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #html_theme = 'rctheme'
 html_theme = 'sphinx_rtd_theme'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 #html_theme_options = {}
 html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 
 # Add any paths that contain custom themes here, relative to this directory.
 #html_theme_path = []
 
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".
 #html_title = None
 
 # A shorter title for the navigation bar.  Default is the same as html_title.
 #html_short_title = None
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
 #html_logo = None
 html_sidebars = {
    '**': ['globaltoc.html'],
 }
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
 html_favicon = 'images/favicon.ico'
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['static/css/add.css']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
 # directly to the root of the documentation.
 #html_extra_path = []
 
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.
 #html_last_updated_fmt = '%b %d, %Y'
 
 # If true, SmartyPants will be used to convert quotes and dashes to
 # typographically correct entities.
 #html_use_smartypants = True
 
 # Custom sidebar templates, maps document names to template names.
 #html_sidebars = {}
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.
 #html_additional_pages = {}
 
 # If false, no module index is generated.
 #html_domain_indices = True
 
 # If false, no index is generated.
 #html_use_index = True
 
 # If true, the index is split into individual pages for each letter.
 #html_split_index = False
 
 # If true, links to the reST sources are added to the pages.
 #html_show_sourcelink = True
 
 # If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
 #html_show_sphinx = True
 
 # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
 #html_show_copyright = True
 
 # If true, an OpenSearch description file will be output, and all pages will
 # contain a <link> tag referring to it.  The value of this option must be the
 # base URL from which the finished HTML is served.
 #html_use_opensearch = ''
 
 # This is the file name suffix for HTML files (e.g. ".xhtml").
 #html_file_suffix = None
 
 # Output file base name for HTML help builder.
 htmlhelp_basename = 'rhodecode-enterprise'
 
 
 # -- Options for LaTeX output ---------------------------------------------
 
 latex_elements = {
     'classoptions': ',oneside',
     'babel': '\\usepackage[english]{babel}',
 
 # The paper size ('letterpaper' or 'a4paper').
 #'papersize': 'letterpaper',
 
 # The font size ('10pt', '11pt' or '12pt').
 #'pointsize': '10pt',
 
 # Additional stuff for the LaTeX preamble.
 #'preamble': '',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
   ('index', 'RhodeCodeEnterprise.tex', u'RhodeCode Enterprise',
    u'RhodeCode GmbH', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
 # the title page.
 #latex_logo = None
 
 # For "manual" documents, if this is true, then toplevel headings are parts,
 # not chapters.
 #latex_use_parts = False
 
 # If true, show page references after internal links.
 latex_show_pagerefs = True
 
 # If true, show URL addresses after external links.
 latex_show_urls = 'footnote'
 
 # Documents to append as an appendix to all manuals.
 #latex_appendices = []
 
 # If false, no module index is generated.
 #latex_domain_indices = True
 
 # Mode for literal blocks wider than the frame. Can be
 # overflow, shrink or truncate
 pdf_fit_mode = "truncate"
 
 
 # -- Options for manual page output ---------------------------------------
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
     ('index', 'rhodecodeenterprise', u'RhodeCode Enterprise',
      [u'RhodeCode GmbH'], 1)
 ]
 
 # If true, show URL addresses after external links.
 #man_show_urls = False
 
 
 # -- Options for Texinfo output -------------------------------------------
 
 # Grouping the document tree into Texinfo files. List of tuples
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
   ('index', 'RhodeCodeEnterprise', u'RhodeCode Enterprise',
    u'RhodeCode Docs Team', 'RhodeCodeEnterprise', 'RhodeCode Docs Project',
    'Miscellaneous'),
 ]
 
 # Documents to append as an appendix to all manuals.
 #texinfo_appendices = []
 
 # If false, no module index is generated.
 #texinfo_domain_indices = True
 
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
 #texinfo_show_urls = 'footnote'
 
 # If true, do not generate a @detailmenu in the "Top" node's menu.
 #texinfo_no_detailmenu = False
 
 # We want to see todo notes in case of a pre-release build of the documentation
 todo_include_todos = tags.has("dev")

docs/contributing/dev-setup.rst

@@ -1,219 +1,226 b''
 .. _dev-setup:
 
 ===================
  Development setup
 ===================
 
 
 RhodeCode Enterprise runs inside a Nix managed environment. This ensures build
 environment dependencies are correctly declared and installed during setup.
 It also enables atomic upgrades, rollbacks, and multiple instances of RhodeCode
 Enterprise running with isolation.
 
 To set up RhodeCode Enterprise inside the Nix environment, use the following steps:
 
 
 
 Setup Nix Package Manager
 -------------------------
 
 To install the Nix Package Manager, please run::
 
    $ curl https://nixos.org/nix/install | sh
 
 or go to https://nixos.org/nix/ and follow the installation instructions.
 Once this is correctly set up on your system, you should be able to use the
 following commands:
 
 * `nix-env`
 
 * `nix-shell`
 
 
 .. tip::
 
    Update your channels frequently by running ``nix-channel --update``.
 
+.. note::
+
+   To uninstall nix run the following:
+
+    remove the . "$HOME/.nix-profile/etc/profile.d/nix.sh" line in your ~/.profile or ~/.bash_profile
+    rm -rf $HOME/{.nix-channels,.nix-defexpr,.nix-profile,.config/nixpkgs}
+    sudo rm -rf /nix
 
 Switch nix to the latest STABLE channel
 ---------------------------------------
 
 run::
 
    nix-channel --add https://nixos.org/channels/nixos-16.03 nixpkgs
 
 Followed by::
 
    nix-channel --update
 
 
 Install required binaries
 -------------------------
 
 We need some handy tools first.
 
 run::
 
     nix-env -i nix-prefetch-hg
     nix-env -i nix-prefetch-git
 
 
 Clone the required repositories
 -------------------------------
 
 After Nix is set up, clone the RhodeCode Enterprise Community Edition and
 RhodeCode VCSServer repositories into the same directory.
 RhodeCode currently is using Mercurial Version Control System, please make sure
 you have it installed before continuing.
 
 To obtain the required sources, use the following commands::
 
     mkdir rhodecode-develop && cd rhodecode-develop
     hg clone https://code.rhodecode.com/rhodecode-enterprise-ce
     hg clone https://code.rhodecode.com/rhodecode-vcsserver
 
 .. note::
 
    If you cannot clone the repository, please contact us via [e-mail]
 
 
 Install some required libraries
 -------------------------------
 
 There are some required drivers and dev libraries that we need to install to
 test RhodeCode under different types of databases. For example in Ubuntu we
 need to install the following.
 
 required libraries::
 
     sudo apt-get install libapr1-dev libaprutil1-dev
     sudo apt-get install libsvn-dev
     sudo apt-get install mysql-server libmysqlclient-dev
     sudo apt-get install postgresql postgresql-contrib libpq-dev
     sudo apt-get install libcurl4-openssl-dev
 
 
 Enter the Development Shell
 ---------------------------
 
 The final step is to start the development shells. To do this, run the
 following command from inside the cloned repository::
 
    #first, the vcsserver
    cd ~/rhodecode-vcsserver
    nix-shell
 
    # then enterprise sources
    cd ~/rhodecode-enterprise-ce
    nix-shell
 
 .. note::
 
    On the first run, this will take a while to download and optionally compile
    a few things. The following runs will be faster. The development shell works
    fine on both MacOS and Linux platforms.
 
 
 Create config.nix for development
 ---------------------------------
 
 In order to run proper tests and setup linking across projects, a config.nix
 file needs to be setup::
 
     # create config
     mkdir -p ~/.nixpkgs
     touch ~/.nixpkgs/config.nix
 
     # put the below content into the ~/.nixpkgs/config.nix file
     # adjusts, the path to where you cloned your repositories.
 
     {
       rc = {
        sources = {
         rhodecode-vcsserver = "/home/dev/rhodecode-vcsserver";
         rhodecode-enterprise-ce = "/home/dev/rhodecode-enterprise-ce";
         rhodecode-enterprise-ee = "/home/dev/rhodecode-enterprise-ee";
        };
       };
     }
 
 
 
 Creating a Development Configuration
 ------------------------------------
 
 To create a development environment for RhodeCode Enterprise,
 use the following steps:
 
 1. Create a copy of vcsserver config:
     `cp ~/rhodecode-vcsserver/configs/development.ini ~/rhodecode-vcsserver/configs/dev.ini`
 2. Create a copy of rhodocode config:
     `cp ~/rhodecode-enterprise-ce/configs/development.ini ~/rhodecode-enterprise-ce/configs/dev.ini`
 3. Adjust the configuration settings to your needs if needed.
 
 .. note::
 
   It is recommended to use the name `dev.ini` since it's included in .hgignore file.
 
 
 Setup the Development Database
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To create a development database, use the following example. This is a one
 time operation executed from the nix-shell of rhodecode-enterprise-ce sources ::
 
     rc-setup-app dev.ini \
         --user=admin --password=secret \
         --email=[e-mail] \
         --repos=~/my_dev_repos
 
 
 Compile CSS and JavaScript
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To use the application's frontend and prepare it for production deployment,
 you will need to compile the CSS and JavaScript with Grunt.
 This is easily done from within the nix-shell using the following command::
 
     grunt
 
 When developing new features you will need to recompile following any
 changes made to the CSS or JavaScript files when developing the code::
 
     grunt watch
 
 This prepares the development (with comments/whitespace) versions of files.
 
 Start the Development Servers
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 From the rhodecode-vcsserver directory, start the development server in another
 nix-shell, using the following command::
 
       pserve configs/dev.ini
 
 In the adjacent nix-shell which you created for your development server, you may
 now start CE with the following command::
 
 
       pserve --reload configs/dev.ini
 
 .. note::
 
   `--reload` flag will automatically reload the server when source file changes.
 
 
 Run the Environment Tests
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Please make sure that the tests are passing to verify that your environment is
 set up correctly. RhodeCode uses py.test to run tests.
 While your instance is running, start a new nix-shell and simply run
 ``make test`` to run the basic test suite.
 
 
 Need Help?
 ^^^^^^^^^^
 
 Join us on Slack via https://rhodecode.com/join or post questions in our
 Community Portal at https://community.rhodecode.com

docs/default.nix

@@ -1,269 +1,261 b''
 { system ? builtins.currentSystem
 }:
 
 let
 
   pkgs = import <nixpkgs> { inherit system; };
 
   inherit (pkgs) fetchurl fetchgit;
 
   buildPythonPackage = pkgs.python27Packages.buildPythonPackage;
   python = pkgs.python27Packages.python;
 
   Jinja2 = buildPythonPackage rec {
     name = "Jinja2-2.9.6";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/90/61/f820ff0076a2599dd39406dcb858ecb239438c02ce706c8e91131ab9c7f1/Jinja2-2.9.6.tar.gz";
       md5 = "6411537324b4dba0956aaa8109f3c77b";
     };
   };
 
   MarkupSafe = buildPythonPackage rec {
     name = "MarkupSafe-1.0";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz";
       md5 = "2fcedc9284d50e577b5192e8e3578355";
     };
   };
 
   Pygments = buildPythonPackage {
     name = "Pygments-2.2.0";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/71/2a/2e4e77803a8bd6408a2903340ac498cb0a2181811af7c9ec92cb70b0308a/Pygments-2.2.0.tar.gz";
       md5 = "13037baca42f16917cbd5ad2fab50844";
     };
   };
 
   Sphinx = buildPythonPackage (rec {
     name = "Sphinx-1.6.5";
     src = fetchurl {
       url = "https://pypi.python.org/packages/8b/7e/b188d9a3b9c938e736e02a74c1363c2888e095d770df2c72b4c312f9fdcb/Sphinx-1.6.5.tar.gz";
       md5 = "cd73118c21ec610432e63e6421ec54f1";
     };
     propagatedBuildInputs = [
         six
         Jinja2
         Pygments
         docutils
         snowballstemmer
         babel
         alabaster
         imagesize
         requests
         setuptools
         sphinxcontrib-websupport
         typing
 
         # special cases
         pytz
         sphinx_rtd_theme
 
     ];
   });
 
   alabaster = buildPythonPackage rec {
     name = "alabaster-0.7.10";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d0/a5/e3a9ad3ee86aceeff71908ae562580643b955ea1b1d4f08ed6f7e8396bd7/alabaster-0.7.10.tar.gz";
       md5 = "7934dccf38801faa105f6e7b4784f493";
     };
   };
 
   babel = buildPythonPackage {
     name = "babel-2.5.1";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [pytz];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5a/22/63f1dbb8514bb7e0d0c8a85cc9b14506599a075e231985f98afd70430e1f/Babel-2.5.1.tar.gz";
       md5 = "60228b3ce93a203357158b909afe8ae1";
     };
   };
 
   certifi = buildPythonPackage {
     name = "certifi-2017.11.5";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/23/3f/8be01c50ed24a4bd6b8da799839066ce0288f66f5e11f0367323467f0cbc/certifi-2017.11.5.tar.gz";
       md5 = "c15ac46ed1fe4b607ff3405928f9a992";
     };
   };
 
   chardet = buildPythonPackage {
     name = "chardet-3.0.4";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz";
       md5 = "7dd1ba7f9c77e32351b0a0cfacf4055c";
     };
   };
 
   docutils = buildPythonPackage {
     name = "docutils-0.14";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz";
       md5 = "c53768d63db3873b7d452833553469de";
     };
   };
 
   idna = buildPythonPackage {
     name = "idna-2.6";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f4/bd/0467d62790828c23c47fc1dfa1b1f052b24efdf5290f071c7a91d0d82fd3/idna-2.6.tar.gz";
       md5 = "c706e2790b016bd0ed4edd2d4ba4d147";
     };
   };
 
   imagesize = buildPythonPackage {
     name = "imagesize-0.7.1";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/53/72/6c6f1e787d9cab2cc733cf042f125abec07209a58308831c9f292504e826/imagesize-0.7.1.tar.gz";
       md5 = "976148283286a6ba5f69b0f81aef8052";
     };
   };
 
   pytz = buildPythonPackage {
     name = "pytz-2017.3";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/60/88/d3152c234da4b2a1f7a989f89609ea488225eaea015bc16fbde2b3fdfefa/pytz-2017.3.zip";
       md5 = "7006b56c0d68a162d9fe57d4249c3171";
     };
   };
 
   requests = buildPythonPackage {
     name = "requests-2.18.4";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [chardet idna urllib3 certifi];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b0/e1/eab4fc3752e3d240468a8c0b284607899d2fbfb236a56b7377a329aa8d09/requests-2.18.4.tar.gz";
       md5 = "081412b2ef79bdc48229891af13f4d82";
     };
   };
 
-  setuptools = buildPythonPackage {
-    name = "setuptools-36.6.0";
-    buildInputs = [];
-    doCheck = false;
-    propagatedBuildInputs = [];
-    src = fetchurl {
-      url = "https://pypi.python.org/packages/45/29/8814bf414e7cd1031e1a3c8a4169218376e284ea2553cc0822a6ea1c2d78/setuptools-36.6.0.zip";
-      md5 = "74663b15117d9a2cc5295d76011e6fd1";
-    };
-  };
-
   six = buildPythonPackage {
     name = "six-1.11.0";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz";
       md5 = "d12789f9baf7e9fb2524c0c64f1773f8";
     };
   };
 
   snowballstemmer = buildPythonPackage {
     name = "snowballstemmer-1.2.1";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/20/6b/d2a7cb176d4d664d94a6debf52cd8dbae1f7203c8e42426daa077051d59c/snowballstemmer-1.2.1.tar.gz";
       md5 = "643b019667a708a922172e33a99bf2fa";
     };
   };
 
   sphinx-rtd-theme = buildPythonPackage {
     name = "sphinx-rtd-theme-0.2.5b1";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/59/e4/9e3a74a3271e6734911d3f549e8439db53b8ac29adf10c8f698e6c86246b/sphinx_rtd_theme-0.2.5b1.tar.gz";
       md5 = "0923473a43bd2527f32151f195f2a521";
     };
   };
 
   sphinxcontrib-websupport = buildPythonPackage {
     name = "sphinxcontrib-websupport-1.0.1";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c5/6b/f0630436b931ad4f8331a9399ca18a7d447f0fcc0c7178fb56b1aee68d01/sphinxcontrib-websupport-1.0.1.tar.gz";
       md5 = "84df26463b1ba65b07f926dbe2055665";
     };
   };
 
   typing = buildPythonPackage {
     name = "typing-3.6.2";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ca/38/16ba8d542e609997fdcd0214628421c971f8c395084085354b11ff4ac9c3/typing-3.6.2.tar.gz";
       md5 = "143af0bf3afd1887622771f2f1ffe8e1";
     };
   };
 
   urllib3 = buildPythonPackage {
     name = "urllib3-1.22";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ee/11/7c59620aceedcc1ef65e156cc5ce5a24ef87be4107c2b74458464e437a5d/urllib3-1.22.tar.gz";
       md5 = "0da7bed3fe94bf7dc59ae37885cc72f7";
     };
   };
 
 
   sphinx_rtd_theme = buildPythonPackage rec {
     name = "sphinx-rtd-theme-0.2.5b1";
     buildInputs = [];
     doCheck = false;
     propagatedBuildInputs = [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/59/e4/9e3a74a3271e6734911d3f549e8439db53b8ac29adf10c8f698e6c86246b/sphinx_rtd_theme-0.2.5b1.tar.gz";
       md5 = "0923473a43bd2527f32151f195f2a521";
     };
 
 
   };
+  # Avoid that setuptools is replaced, this leads to trouble
+  # with buildPythonPackage.
+  setuptools = pkgs.python27Packages.setuptools;
 
 in python.buildEnv.override {
   inherit python;
   extraLibs = [
     Sphinx
     sphinx_rtd_theme
   ];
 }

docs/release-notes/release-notes.rst

@@ -1,114 +1,115 b''
 .. _rhodecode-release-notes-ref:
 
 Release Notes
 =============
 
 |RCE| 4.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
+   release-notes-4.12.0.rst
    release-notes-4.11.6.rst
    release-notes-4.11.5.rst
    release-notes-4.11.4.rst
    release-notes-4.11.3.rst
    release-notes-4.11.2.rst
    release-notes-4.11.1.rst
    release-notes-4.11.0.rst
    release-notes-4.10.6.rst
    release-notes-4.10.5.rst
    release-notes-4.10.4.rst
    release-notes-4.10.3.rst
    release-notes-4.10.2.rst
    release-notes-4.10.1.rst
    release-notes-4.10.0.rst
    release-notes-4.9.1.rst
    release-notes-4.9.0.rst
    release-notes-4.8.0.rst
    release-notes-4.7.2.rst
    release-notes-4.7.1.rst
    release-notes-4.7.0.rst
    release-notes-4.6.1.rst
    release-notes-4.6.0.rst
    release-notes-4.5.2.rst
    release-notes-4.5.1.rst
    release-notes-4.5.0.rst
    release-notes-4.4.2.rst
    release-notes-4.4.1.rst
    release-notes-4.4.0.rst
    release-notes-4.3.1.rst
    release-notes-4.3.0.rst
    release-notes-4.2.1.rst
    release-notes-4.2.0.rst
    release-notes-4.1.2.rst
    release-notes-4.1.1.rst
    release-notes-4.1.0.rst
    release-notes-4.0.1.rst
    release-notes-4.0.0.rst
 
 |RCE| 3.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-3.8.4.rst
    release-notes-3.8.3.rst
    release-notes-3.8.2.rst
    release-notes-3.8.1.rst
    release-notes-3.8.0.rst
    release-notes-3.7.1.rst
    release-notes-3.7.0.rst
    release-notes-3.6.1.rst
    release-notes-3.6.0.rst
    release-notes-3.5.2.rst
    release-notes-3.5.1.rst
    release-notes-3.5.0.rst
    release-notes-3.4.1.rst
    release-notes-3.4.0.rst
    release-notes-3.3.4.rst
    release-notes-3.3.3.rst
    release-notes-3.3.2.rst
    release-notes-3.3.1.rst
    release-notes-3.3.0.rst
    release-notes-3.2.3.rst
    release-notes-3.2.2.rst
    release-notes-3.2.1.rst
    release-notes-3.2.0.rst
    release-notes-3.1.1.rst
    release-notes-3.1.0.rst
    release-notes-3.0.2.rst
    release-notes-3.0.1.rst
    release-notes-3.0.0.rst
 
 |RCE| 2.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-2.2.8.rst
    release-notes-2.2.7.rst
    release-notes-2.2.6.rst
    release-notes-2.2.5.rst
    release-notes-2.2.4.rst
    release-notes-2.2.3.rst
    release-notes-2.2.2.rst
    release-notes-2.2.1.rst
    release-notes-2.2.0.rst
    release-notes-2.1.0.rst
    release-notes-2.0.2.rst
    release-notes-2.0.1.rst
    release-notes-2.0.0.rst
 
 |RCE| 1.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-1.7.2.rst
    release-notes-1.7.1.rst
    release-notes-1.7.0.rst
    release-notes-1.6.0.rst

docs/tutorials/deploy-from-host.rst

@@ -1,220 +1,220 b''
 .. _hosted-solution:
 
 Deploy |RCE| From a Hosted Server
 =================================
 
 If you wish to deploy your own |RCE| instance from something like a
 `Digital Ocean`_ droplet, or a `hetzner`_ server use the following
 instructions to get it setup.
 
 I'm using an Ubuntu 14.04 image for the purposes of this
 tutorial, but all other Unix environments will be pretty similar. You can
 check out the full lists of supported platforms and versions in the
 :ref:`system-overview-ref` section.
 
 
 Create a Digital Ocean Droplet
 ------------------------------
 
 1. Sign into Digital Ocean.
 2. Create a Droplet choosing Ubuntu 14.04 as your |os|.
 3. (Optional) Add SSH keys if you have them set up.
 
 Configure Your Server
 ---------------------
 
 Once you have your server created, you need to sign into it and set it up to
 host |RCE|.
 
 1. Open a terminal and sign into your server. Digital Ocean will mail you the
    IP address. You'll need to change your password on the first login if you
    don not have SSH keys set up.
 
 .. code-block:: bash
 
     $ ssh root@203.0.113.113
 
 2. It is not advised to install |RCE| as the ``root`` user. So create a user
    with sudo permissions and then carry out the rest of the steps from that user
    account.
 
 .. code-block:: bash
 
     # Create a user with sudo permissions
     root@rhodecode:~# sudo useradd -m -s /bin/bash -d /home/brian -U brian
     root@rhodecode:~# sudo usermod -a -G sudo brian
 
     # Set the password for that user
     root@rhodecode:~# passwd brian
     Enter new UNIX password:
     Retype new UNIX password:
     passwd: password updated successfully
 
     # Switch to that user for the rest of the steps
     root@rhodecode:~# su brian
 
     # You should see your home dir change to what was set during installation
     brian@rhodecode:~$ cd ~
     brian@rhodecode:~$ pwd
     /home/brian
 
 Once you have this set up, you are ready to install |RCC|.
 
 Install |RCC|
 -------------
 
 |RCC| will install and manage the package dependencies for your |RCE| instance.
 
 1. Download the |RCC| installer from https://rhodecode.com/download/
 2. Once downloaded to your computer, transfer the package to your server
 
 .. note::
 
    These steps happen on your computer, not on the server.
 
 .. code-block:: bash
 
     # Change to where the file is downloaded
     $ cd Downloads/
 
     # SFTP to your server
     $ sftp brian@203.0.113.113
 
     # Use mput to transfer the file
     sftp> mput RhodeCode-installer-linux-391_b1a804c4d69b_d6c087d520e3
     Uploading RhodeCode-installer-linux-391_b1a804c4d69b_d6c087d520e3 to /home/brian/RhodeCode-installer-linux-391_b1a804c4d69b_d6c087d520e3
     RhodeCode-installer-linux-391_b1a804c4d69b_d6c087d 100%  289MB   4.1MB/s   01:11
     sftp> exit
 
 The |RCC| installer is now on your server, and you can read the full
 instructions here
-:ref:`Install RhodeCode Control <control:rcc-install-ref>`,
+:ref:`Install RhodeCode Control <control:rcc-linux-ref>` ,
 but below is the example shortcut.
 
 .. code-block:: bash
 
     # Check that the script is uploaded to your home directory
     $ ls -1
     RhodeCode-installer-linux-391_b1a804c4d69b_d6c087d520e3
 
     # Change the script permissions
     $ chmod 755 RhodeCode-installer-linux*
 
     # Run the installer and accept the prompts
     $ ./RhodeCode-installer-linux-*
 
 .. important::
 
    Once finished, exit the terminal and sign in again. This is to refresh you
    session to pick up the new commands.
 
 Install |RCE|
 -------------
 
 Now that |RCC| is installed, you can install |RCE|. For the full
 instructions, see
 :ref:`Install RhodeCode Enterprise <control:rce-cli-install-ref>`,
 but the below is an example shortcut.
 
 .. code-block:: bash
 
     # Install a VCS Server and follow the prompts
     $ rccontrol install VCSServer --start-at-boot
 
     Extracting VCSServer ...
     Configuring RhodeCode VCS Server ...
     Supervisord state is: RUNNING
     Added process group vcsserver-1
 
     # Install a RhodeCode Enterprise instance and follow the prompts
     $ rccontrol install Enterprise --start-at-boot
 
     Configuration of RhodeCode Enterprise passed.
     Supervisord state is: RUNNING
     Added process group enterprise-1
 
 |RCE| is now installed on your server, and is running on the port displayed
 by the ``rccontrol status`` command.
 
 .. code-block:: bash
 
     brian@rhodecode:~$ rccontrol status
 
      - NAME: enterprise-1
      - STATUS: RUNNING
      - TYPE: Enterprise
      - VERSION: 3.1.1
      - URL: http://127.0.0.1:10002
 
      - NAME: vcsserver-1
      - STATUS: RUNNING
      - TYPE: VCSServer
      - VERSION: 1.1.1
      - URL: http://127.0.0.1:10001
 
 Serve |RCE| using Nginx
 -----------------------
 
 Now that |RCE| is running, you need to use Nginx or Apache to serve it to
 users. For detailed instructions about setting up your webserver, see the
 :ref:`rhodecode-admin-ref` section. But the below shortcut should help serve
 it.
 
 1. Install Nginx on your server.
 
 .. code-block:: bash
 
     # Install nginx
     $ sudo apt-get install nginx
 
 2. Create a virtual hosts file for RhodeCode Enterprise. Create
    the file in this location :file:`/etc/nginx/sites-available`. In this demo
    I have called it ``vcs.conf``
 
 .. code-block:: bash
 
    # Create the file
    $ sudo vi /etc/nginx/sites-available/vcs.conf
 
 Use the following example to create yours.
 
 .. code-block:: nginx
 
     server {
         listen 80;
         # Change to your IP, or a domain name if you've set that up
         server_name     203.0.113.113 ;
 
         location / {
         # Set this line to match the RhodeCode Enterprise Instance URL
             proxy_pass		http://127.0.0.1:10002/;
             proxy_set_header	Host $Host;
             proxy_buffering		off;
         # Setting this to a high number allows large repo pushes
             client_max_body_size	4G;
             }
         }
 
 3. Symlink the virtual hosts file to the ``sites-enabled`` folder,
    and then restart Nginx.
 
 .. code-block:: bash
 
    # Symlink the virtual hosts file
    $ ln -s /etc/nginx/sites-available/vcs.conf /etc/nginx/sites-enabled/vcs.conf
 
    # You can also delete the Nginx default symlink
    $ rm /etc/nginx/sites-enabled/default
 
    # Restart Nginx
    $ sudo  /etc/init.d/nginx restart
      * Restarting nginx nginx                                     [ OK ]
 
 Once restarted, you should see a clean |RCE| instance running on the IP
 address, or the domain you have set up.
 
 .. image:: ../images/clean-rce.png
    :alt: A fresh RhodeCode Enterprise Instance
 
 .. _Digital Ocean: https://www.digitalocean.com/
 .. _hetzner: https://www.hetzner.de/en/

pkgs/python-packages-overrides.nix

@@ -1,251 +1,257 b''
 # Overrides for the generated python-packages.nix
 #
 # This function is intended to be used as an extension to the generated file
 # python-packages.nix. The main objective is to add needed dependencies of C
 # libraries and tweak the build instructions where needed.
 
 { pkgs, basePythonPackages }:
 
 let
   sed = "sed -i";
   localLicenses = {
     repoze = {
       fullName = "Repoze License";
       url =  http://www.repoze.org/LICENSE.txt;
     };
   };
 
 in
 
 self: super: {
 
   appenlight-client = super.appenlight-client.override (attrs: {
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   });
 
+  Beaker = super.Beaker.override (attrs: {
+    patches = [
+      ./patch-beaker-lock-func-debug.diff
+    ];
+  });
+
   future = super.future.override (attrs: {
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   });
 
   testpath = super.testpath.override (attrs: {
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   });
 
   gnureadline = super.gnureadline.override (attrs: {
     buildInputs = attrs.buildInputs ++ [
       pkgs.ncurses
     ];
     patchPhase = ''
       substituteInPlace setup.py --replace "/bin/bash" "${pkgs.bash}/bin/bash"
     '';
   });
 
   gunicorn = super.gunicorn.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       # johbo: futures is needed as long as we are on Python 2, otherwise
       # gunicorn explodes if used with multiple threads per worker.
       self.futures
     ];
   });
 
   nbconvert = super.nbconvert.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       # marcink: plug in jupyter-client for notebook rendering
       self.jupyter-client
     ];
   });
 
   ipython = super.ipython.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       self.gnureadline
     ];
   });
 
   lxml = super.lxml.override (attrs: {
     # johbo: On 16.09 we need this to compile on darwin, otherwise compilation
     # fails on Darwin.
     hardeningDisable = if pkgs.stdenv.isDarwin then [ "format" ] else null;
     buildInputs = with self; [
       pkgs.libxml2
       pkgs.libxslt
     ];
   });
 
   MySQL-python = super.MySQL-python.override (attrs: {
     buildInputs = attrs.buildInputs ++ [
       pkgs.openssl
     ];
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
-      pkgs.mysql.lib
+      pkgs.libmysql
       pkgs.zlib
     ];
   });
 
   psutil = super.psutil.override (attrs: {
     buildInputs = attrs.buildInputs ++
       pkgs.lib.optional pkgs.stdenv.isDarwin pkgs.darwin.IOKit;
   });
 
   psycopg2 = super.psycopg2.override (attrs: {
     buildInputs = attrs.buildInputs ++
       pkgs.lib.optional pkgs.stdenv.isDarwin pkgs.openssl;
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.postgresql
     ];
     meta = {
       license = pkgs.lib.licenses.lgpl3Plus;
     };
   });
 
   pycurl = super.pycurl.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.curl
       pkgs.openssl
     ];
     preConfigure = ''
       substituteInPlace setup.py --replace '--static-libs' '--libs'
       export PYCURL_SSL_LIBRARY=openssl
     '';
     meta = {
       # TODO: It is LGPL and MIT
       license = pkgs.lib.licenses.mit;
     };
   });
 
   pyramid = super.pyramid.override (attrs: {
     postFixup = ''
       wrapPythonPrograms
       # TODO: johbo: "wrapPython" adds this magic line which
       # confuses pserve.
       ${sed} '/import sys; sys.argv/d' $out/bin/.pserve-wrapped
     '';
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   pyramid-debugtoolbar = super.pyramid-debugtoolbar.override (attrs: {
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal localLicenses.repoze ];
     };
   });
 
   pysqlite = super.pysqlite.override (attrs: {
     propagatedBuildInputs = [
       pkgs.sqlite
     ];
     meta = {
       license = [ pkgs.lib.licenses.zlib pkgs.lib.licenses.libpng ];
     };
   });
 
   pytest-runner = super.pytest-runner.override (attrs: {
     propagatedBuildInputs = [
       self.setuptools-scm
     ];
   });
 
   python-ldap = super.python-ldap.override (attrs: {
     propagatedBuildInputs = attrs.propagatedBuildInputs ++ [
       pkgs.cyrus_sasl
       pkgs.openldap
       pkgs.openssl
     ];
     # TODO: johbo: Remove the "or" once we drop 16.03 support.
     NIX_CFLAGS_COMPILE = "-I${pkgs.cyrus_sasl.dev or pkgs.cyrus_sasl}/include/sasl";
   });
 
   python-pam = super.python-pam.override (attrs:
     let
       includeLibPam = pkgs.stdenv.isLinux;
     in {
       # TODO: johbo: Move the option up into the default.nix, we should
       # include python-pam only on supported platforms.
       propagatedBuildInputs = attrs.propagatedBuildInputs ++
         pkgs.lib.optional includeLibPam [
           pkgs.pam
         ];
       # TODO: johbo: Check if this can be avoided, or transform into
       # a real patch
       patchPhase = pkgs.lib.optionals includeLibPam ''
         substituteInPlace pam.py \
           --replace 'find_library("pam")' '"${pkgs.pam}/lib/libpam.so.0"'
       '';
     });
 
   URLObject = super.URLObject.override (attrs: {
     meta = {
       license = {
         spdxId = "Unlicense";
         fullName = "The Unlicense";
         url = http://unlicense.org/;
       };
     };
   });
 
   docutils = super.docutils.override (attrs: {
     meta = {
       license = pkgs.lib.licenses.bsd2;
     };
   });
 
   colander = super.colander.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   pyramid-beaker  = super.pyramid-beaker.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   pyramid-mako = super.pyramid-mako.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   repoze.lru = super.repoze.lru.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   recaptcha-client = super.recaptcha-client.override (attrs: {
     meta = {
       # TODO: It is MIT/X11
       license = pkgs.lib.licenses.mit;
     };
   });
 
   python-editor = super.python-editor.override (attrs: {
     meta = {
       license = pkgs.lib.licenses.asl20;
     };
   });
 
   translationstring = super.translationstring.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   venusian = super.venusian.override (attrs: {
     meta = {
       license = localLicenses.repoze;
     };
   });
 
   # Avoid that setuptools is replaced, this leads to trouble
   # with buildPythonPackage.
   setuptools = basePythonPackages.setuptools;
 
 }

pkgs/python-packages.nix

@@ -1,2086 +1,2099 b''
 # Generated by pip2nix 0.4.0
 # See https://github.com/johbo/pip2nix
 
 {
   Babel = super.buildPythonPackage {
     name = "Babel-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytz];
     src = fetchurl {
       url = "https://pypi.python.org/packages/33/27/e3978243a03a76398c384c83f7ca879bc6e8f1511233a621fcada135606e/Babel-1.3.tar.gz";
       md5 = "5264ceb02717843cbc9ffce8e6e06bdb";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   Beaker = super.buildPythonPackage {
-    name = "Beaker-1.9.0";
+    name = "Beaker-1.9.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [funcsigs];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/93/b2/12de6937b06e9615dbb3cb3a1c9af17f133f435bdef59f4ad42032b6eb49/Beaker-1.9.0.tar.gz";
-      md5 = "38b3fcdfa24faf97c6cf66991eb54e9c";
+      url = "https://pypi.python.org/packages/ca/14/a626188d0d0c7b55dd7cf1902046c2743bd392a7078bb53073e13280eb1e/Beaker-1.9.1.tar.gz";
+      md5 = "46fda0a164e2b0d24ccbda51a2310301";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   CProfileV = super.buildPythonPackage {
     name = "CProfileV-1.0.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [bottle];
     src = fetchurl {
       url = "https://pypi.python.org/packages/df/50/d8c1ada7d537c64b0f76453fa31dedb6af6e27b82fcf0331e5f71a4cf98b/CProfileV-1.0.7.tar.gz";
       md5 = "db4c7640438aa3d8887e194c81c7a019";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Chameleon = super.buildPythonPackage {
     name = "Chameleon-2.24";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5a/9e/637379ffa13c5172b5c0e704833ffea6bf51cec7567f93fd6e903d53ed74/Chameleon-2.24.tar.gz";
       md5 = "1b01f1f6533a8a11d0d2f2366dec5342";
     };
     meta = {
       license = [ { fullName = "BSD-like (http://repoze.org/license.html)"; } ];
     };
   };
   FormEncode = super.buildPythonPackage {
     name = "FormEncode-1.2.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8e/59/0174271a6f004512e0201188593e6d319db139d14cb7490e488bbb078015/FormEncode-1.2.4.tar.gz";
       md5 = "6bc17fb9aed8aea198975e888e2077f4";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   Jinja2 = super.buildPythonPackage {
     name = "Jinja2-2.9.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/90/61/f820ff0076a2599dd39406dcb858ecb239438c02ce706c8e91131ab9c7f1/Jinja2-2.9.6.tar.gz";
       md5 = "6411537324b4dba0956aaa8109f3c77b";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   Mako = super.buildPythonPackage {
     name = "Mako-1.0.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/eb/f3/67579bb486517c0d49547f9697e36582cd19dafb5df9e687ed8e22de57fa/Mako-1.0.7.tar.gz";
       md5 = "5836cc997b1b773ef389bf6629c30e65";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Markdown = super.buildPythonPackage {
-    name = "Markdown-2.6.9";
+    name = "Markdown-2.6.11";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/29/82/dfe242bcfd9eec0e7bf93a80a8f8d8515a95b980c44f5c0b45606397a423/Markdown-2.6.9.tar.gz";
-      md5 = "56547d362a9abcf30955b8950b08b5e3";
+      url = "https://pypi.python.org/packages/b3/73/fc5c850f44af5889192dff783b7b0d8f3fe8d30b65c8e3f78f8f0265fecf/Markdown-2.6.11.tar.gz";
+      md5 = "a67c1b2914f7d74eeede2ebe0fdae470";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   MarkupSafe = super.buildPythonPackage {
     name = "MarkupSafe-1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz";
       md5 = "2fcedc9284d50e577b5192e8e3578355";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   MySQL-python = super.buildPythonPackage {
     name = "MySQL-python-1.2.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a5/e9/51b544da85a36a68debe7a7091f068d802fc515a3a202652828c73453cad/MySQL-python-1.2.5.zip";
       md5 = "654f75b302db6ed8dc5a898c625e030c";
     };
     meta = {
       license = [ pkgs.lib.licenses.gpl1 ];
     };
   };
   Paste = super.buildPythonPackage {
     name = "Paste-2.0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/30/c3/5c2f7c7a02e4f58d4454353fa1c32c94f79fa4e36d07a67c0ac295ea369e/Paste-2.0.3.tar.gz";
       md5 = "1231e14eae62fa7ed76e9130b04bc61e";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   PasteDeploy = super.buildPythonPackage {
     name = "PasteDeploy-1.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0f/90/8e20cdae206c543ea10793cbf4136eb9a8b3f417e04e40a29d72d9922cbd/PasteDeploy-1.5.2.tar.gz";
       md5 = "352b7205c78c8de4987578d19431af3b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   PasteScript = super.buildPythonPackage {
     name = "PasteScript-2.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Paste PasteDeploy six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/e5/f0/78e766c3dcc61a4f3a6f71dd8c95168ae9c7a31722b5663d19c1fdf62cb6/PasteScript-2.0.2.tar.gz";
       md5 = "ccb3045445097192ca71a13b746c77b2";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Pygments = super.buildPythonPackage {
     name = "Pygments-2.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/71/2a/2e4e77803a8bd6408a2903340ac498cb0a2181811af7c9ec92cb70b0308a/Pygments-2.2.0.tar.gz";
       md5 = "13037baca42f16917cbd5ad2fab50844";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   Routes = super.buildPythonPackage {
     name = "Routes-2.4.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six repoze.lru];
     src = fetchurl {
       url = "https://pypi.python.org/packages/33/38/ea827837e68d9c7dde4cff7ec122a93c319f0effc08ce92a17095576603f/Routes-2.4.1.tar.gz";
       md5 = "c058dff6832941dec47e0d0052548ad8";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   SQLAlchemy = super.buildPythonPackage {
     name = "SQLAlchemy-1.1.15";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c2/f6/11fcc1ce19a7cb81b1c9377f4e27ce3813265611922e355905e57c44d164/SQLAlchemy-1.1.15.tar.gz";
       md5 = "077f9bd3339957f53068b5572a152674";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Tempita = super.buildPythonPackage {
     name = "Tempita-0.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/56/c8/8ed6eee83dbddf7b0fc64dd5d4454bc05e6ccaafff47991f73f2894d9ff4/Tempita-0.5.2.tar.gz";
       md5 = "4c2f17bb9d481821c41b6fbee904cea1";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   URLObject = super.buildPythonPackage {
     name = "URLObject-2.4.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/cb/b6/e25e58500f9caef85d664bec71ec67c116897bfebf8622c32cb75d1ca199/URLObject-2.4.0.tar.gz";
       md5 = "2ed819738a9f0a3051f31dc9924e3065";
     };
     meta = {
       license = [  ];
     };
   };
   WebError = super.buildPythonPackage {
     name = "WebError-0.10.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb Tempita Pygments Paste];
     src = fetchurl {
       url = "https://pypi.python.org/packages/35/76/e7e5c2ce7e9c7f31b54c1ff295a495886d1279a002557d74dd8957346a79/WebError-0.10.3.tar.gz";
       md5 = "84b9990b0baae6fd440b1e60cdd06f9a";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   WebHelpers = super.buildPythonPackage {
     name = "WebHelpers-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ee/68/4d07672821d514184357f1552f2dad923324f597e722de3b016ca4f7844f/WebHelpers-1.3.tar.gz";
       md5 = "32749ffadfc40fea51075a7def32588b";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   WebHelpers2 = super.buildPythonPackage {
     name = "WebHelpers2-2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [MarkupSafe six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ff/30/56342c6ea522439e3662427c8d7b5e5b390dff4ff2dc92d8afcb8ab68b75/WebHelpers2-2.0.tar.gz";
       md5 = "0f6b68d70c12ee0aed48c00b24da13d3";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   WebOb = super.buildPythonPackage {
     name = "WebOb-1.7.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/75/34/731e23f52371852dfe7490a61644826ba7fe70fd52a377aaca0f4956ba7f/WebOb-1.7.4.tar.gz";
       md5 = "397e46892d7f199b1a07eb20a2d3d9bd";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   WebTest = super.buildPythonPackage {
     name = "WebTest-2.0.29";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six WebOb waitress beautifulsoup4];
     src = fetchurl {
       url = "https://pypi.python.org/packages/94/de/8f94738be649997da99c47b104aa3c3984ecec51a1d8153ed09638253d56/WebTest-2.0.29.tar.gz";
       md5 = "30b4cf0d340b9a5335fac4389e6f84fc";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   Whoosh = super.buildPythonPackage {
     name = "Whoosh-2.7.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/25/2b/6beed2107b148edc1321da0d489afc4617b9ed317ef7b72d4993cad9b684/Whoosh-2.7.4.tar.gz";
       md5 = "c2710105f20b3e29936bd2357383c325";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.bsd2 ];
     };
   };
   alembic = super.buildPythonPackage {
-    name = "alembic-0.9.6";
+    name = "alembic-0.9.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [SQLAlchemy Mako python-editor python-dateutil];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/bf/b3/b28ea715824f8455635ece3c12f59d5d205f98cc378858e414e3aa6ebdbc/alembic-0.9.6.tar.gz";
-      md5 = "fcb096bccc87c8770bd07a04606cb989";
+      url = "https://pypi.python.org/packages/a1/95/2252783859df9ec76b9a25d968c2827ed75a43ba34c6e8d38f87a5c0fb26/alembic-0.9.8.tar.gz";
+      md5 = "5cfef58641c9a94d4a5d547e951a7dda";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   amqp = super.buildPythonPackage {
     name = "amqp-2.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [vine];
     src = fetchurl {
       url = "https://pypi.python.org/packages/e0/70/9ab9ccd8247fb7d2adb717e9f6a0ed358c9e1ab2c349048b0352f9e80ee2/amqp-2.2.2.tar.gz";
       md5 = "0971a3fd2d635ded45c349cfc17106bd";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   appenlight-client = super.buildPythonPackage {
-    name = "appenlight-client-0.6.22";
+    name = "appenlight-client-0.6.25";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [WebOb requests six];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/73/37/0a64460fa9670b17c061adc433bc8be5079cba21e8b3a92d824adccb12bc/appenlight_client-0.6.22.tar.gz";
-      md5 = "641afc114a9a3b3af4f75b11c70968ee";
+      url = "https://pypi.python.org/packages/fa/44/2911ef85ea4f4fe65058fd22959d8dad598fab6a3c84e5bcb569d15c8783/appenlight_client-0.6.25.tar.gz";
+      md5 = "76dd2f9d42659fae8f290982078dc880";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   authomatic = super.buildPythonPackage {
     name = "authomatic-0.1.0.post1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/08/1a/8a930461e604c2d5a7a871e1ac59fa82ccf994c32e807230c8d2fb07815a/Authomatic-0.1.0.post1.tar.gz";
       md5 = "be3f3ce08747d776aae6d6cc8dcb49a9";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   backports.shutil-get-terminal-size = super.buildPythonPackage {
     name = "backports.shutil-get-terminal-size-1.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ec/9c/368086faa9c016efce5da3e0e13ba392c9db79e3ab740b763fe28620b18b/backports.shutil_get_terminal_size-1.0.0.tar.gz";
       md5 = "03267762480bd86b50580dc19dff3c66";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   beautifulsoup4 = super.buildPythonPackage {
     name = "beautifulsoup4-4.6.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/fa/8d/1d14391fdaed5abada4e0f63543fef49b8331a34ca60c88bd521bcf7f782/beautifulsoup4-4.6.0.tar.gz";
       md5 = "c17714d0f91a23b708a592cb3c697728";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   billiard = super.buildPythonPackage {
     name = "billiard-3.5.0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/39/ac/f5571210cca2e4f4532e38aaff242f26c8654c5e2436bee966c230647ccc/billiard-3.5.0.3.tar.gz";
       md5 = "113ba481e48400adbf6fbbf59a2f8554";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   bleach = super.buildPythonPackage {
-    name = "bleach-2.1.1";
+    name = "bleach-2.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six html5lib];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/d4/3f/d517089af35b01bb9bc4eac5ea04bae342b37a5e9abbb27b7c3ce0eae070/bleach-2.1.1.tar.gz";
-      md5 = "7c5dfb1d66ea979b5a465afb12c82ec4";
+      url = "https://pypi.python.org/packages/b3/5f/0da670d30d3ffbc57cc97fa82947f81bbe3eab8d441e2d42e661f215baf2/bleach-2.1.2.tar.gz";
+      md5 = "d0b14ae43a437ee0c650e04c6063eedd";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   bottle = super.buildPythonPackage {
     name = "bottle-0.12.13";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bd/99/04dc59ced52a8261ee0f965a8968717a255ea84a36013e527944dbf3468c/bottle-0.12.13.tar.gz";
       md5 = "d2fe1b48c1d49217e78bf326b1cad437";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   bumpversion = super.buildPythonPackage {
     name = "bumpversion-0.5.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/14/41/8c9da3549f8e00c84f0432c3a8cf8ed6898374714676aab91501d48760db/bumpversion-0.5.3.tar.gz";
       md5 = "c66a3492eafcf5ad4b024be9fca29820";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   celery = super.buildPythonPackage {
     name = "celery-4.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytz billiard kombu];
     src = fetchurl {
       url = "https://pypi.python.org/packages/07/65/88a2a45fc80f487872c93121a701a53bbbc3d3d832016876fac84fc8d46a/celery-4.1.0.tar.gz";
       md5 = "db91e1d26936381127f01e150fe3054a";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   channelstream = super.buildPythonPackage {
     name = "channelstream-0.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [gevent ws4py pyramid pyramid-jinja2 itsdangerous requests six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2b/31/29a8e085cf5bf97fa88e7b947adabfc581a18a3463adf77fb6dada34a65f/channelstream-0.5.2.tar.gz";
       md5 = "1c5eb2a8a405be6f1073da94da6d81d3";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   click = super.buildPythonPackage {
     name = "click-6.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/7a/00/c14926d8232b36b08218067bcd5853caefb4737cda3f0a47437151344792/click-6.6.tar.gz";
       md5 = "d0b09582123605220ad6977175f3e51d";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   colander = super.buildPythonPackage {
     name = "colander-1.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [translationstring iso8601];
     src = fetchurl {
       url = "https://pypi.python.org/packages/cc/e2/c4e716ac4a426d8ad4dfe306c34f0018a22275d2420815784005bf771c84/colander-1.4.tar.gz";
       md5 = "cbb8e403c2ba05aeaa419d51fdb93736";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   configobj = super.buildPythonPackage {
     name = "configobj-5.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz";
       md5 = "e472a3a1c2a67bb0ec9b5d54c13a47d6";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   configparser = super.buildPythonPackage {
     name = "configparser-3.5.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/7c/69/c2ce7e91c89dc073eb1aa74c0621c3eefbffe8216b3f9af9d3885265c01c/configparser-3.5.0.tar.gz";
       md5 = "cfdd915a5b7a6c09917a64a573140538";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   cov-core = super.buildPythonPackage {
     name = "cov-core-1.15.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [coverage];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4b/87/13e75a47b4ba1be06f29f6d807ca99638bedc6b57fa491cd3de891ca2923/cov-core-1.15.0.tar.gz";
       md5 = "f519d4cb4c4e52856afb14af52919fe6";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   coverage = super.buildPythonPackage {
     name = "coverage-3.7.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/09/4f/89b06c7fdc09687bca507dc411c342556ef9c5a3b26756137a4878ff19bf/coverage-3.7.1.tar.gz";
       md5 = "c47b36ceb17eaff3ecfab3bcd347d0df";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   cssselect = super.buildPythonPackage {
     name = "cssselect-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/77/ff/9c865275cd19290feba56344eba570e719efb7ca5b34d67ed12b22ebbb0d/cssselect-1.0.1.tar.gz";
       md5 = "3fa03bf82a9f0b1223c0f1eb1369e139";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   decorator = super.buildPythonPackage {
     name = "decorator-4.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bb/e0/f6e41e9091e130bf16d4437dabbac3993908e4d6485ecbc985ef1352db94/decorator-4.1.2.tar.gz";
       md5 = "a0f7f4fe00ae2dde93494d90c192cf8c";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal { fullName = "new BSD License"; } ];
     };
   };
   deform = super.buildPythonPackage {
     name = "deform-2.0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Chameleon colander iso8601 peppercorn translationstring zope.deprecation];
     src = fetchurl {
       url = "https://pypi.python.org/packages/66/3b/eefcb07abcab7a97f6665aa2d0cf1af741d9d6e78a2e4657fd2b89f89880/deform-2.0.4.tar.gz";
       md5 = "34756e42cf50dd4b4430809116c4ec0a";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   docutils = super.buildPythonPackage {
     name = "docutils-0.14";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz";
       md5 = "c53768d63db3873b7d452833553469de";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.publicDomain pkgs.lib.licenses.gpl1 { fullName = "public domain, Python, 2-Clause BSD, GPL 3 (see COPYING.txt)"; } pkgs.lib.licenses.psfl ];
     };
   };
   dogpile.cache = super.buildPythonPackage {
     name = "dogpile.cache-0.6.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b6/3d/35c05ca01c070bb70d9d422f2c4858ecb021b05b21af438fec5ccd7b945c/dogpile.cache-0.6.4.tar.gz";
       md5 = "66e0a6cae6c08cb1ea25f89d0eadfeb0";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   dogpile.core = super.buildPythonPackage {
     name = "dogpile.core-0.4.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0e/77/e72abc04c22aedf874301861e5c1e761231c288b5de369c18be8f4b5c9bb/dogpile.core-0.4.1.tar.gz";
       md5 = "01cb19f52bba3e95c9b560f39341f045";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ecdsa = super.buildPythonPackage {
     name = "ecdsa-0.13";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f9/e5/99ebb176e47f150ac115ffeda5fedb6a3dbb3c00c74a59fd84ddf12f5857/ecdsa-0.13.tar.gz";
       md5 = "1f60eda9cb5c46722856db41a3ae6670";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   elasticsearch = super.buildPythonPackage {
     name = "elasticsearch-2.3.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [urllib3];
     src = fetchurl {
       url = "https://pypi.python.org/packages/10/35/5fd52c5f0b0ee405ed4b5195e8bce44c5e041787680dc7b94b8071cac600/elasticsearch-2.3.0.tar.gz";
       md5 = "2550f3b51629cf1ef9636608af92c340";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   elasticsearch-dsl = super.buildPythonPackage {
     name = "elasticsearch-dsl-2.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six python-dateutil elasticsearch];
     src = fetchurl {
       url = "https://pypi.python.org/packages/66/2f/52a086968788e58461641570f45c3207a52d46ebbe9b77dc22b6a8ffda66/elasticsearch-dsl-2.2.0.tar.gz";
       md5 = "fa6bd3c87ea3caa8f0f051bc37c53221";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   entrypoints = super.buildPythonPackage {
     name = "entrypoints-0.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [configparser];
     src = fetchurl {
       url = "https://code.rhodecode.com/upstream/entrypoints/archive/96e6d645684e1af3d7df5b5272f3fe85a546b233.tar.gz?md5=7db37771aea9ac9fefe093e5d6987313";
       md5 = "7db37771aea9ac9fefe093e5d6987313";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   enum34 = super.buildPythonPackage {
     name = "enum34-1.1.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bf/3e/31d502c25302814a7c2f1d3959d2a3b3f78e509002ba91aea64993936876/enum34-1.1.6.tar.gz";
       md5 = "5f13a0841a61f7fc295c514490d120d0";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   funcsigs = super.buildPythonPackage {
     name = "funcsigs-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/94/4a/db842e7a0545de1cdb0439bb80e6e42dfe82aaeaadd4072f2263a4fbed23/funcsigs-1.0.2.tar.gz";
       md5 = "7e583285b1fb8a76305d6d68f4ccc14e";
     };
     meta = {
       license = [ { fullName = "ASL"; } pkgs.lib.licenses.asl20 ];
     };
   };
   functools32 = super.buildPythonPackage {
     name = "functools32-3.2.3.post2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/5e/1a/0aa2c8195a204a9f51284018562dea77e25511f02fe924fac202fc012172/functools32-3.2.3-2.zip";
-      md5 = "d55232eb132ec779e6893c902a0bc5ad";
+      url = "https://pypi.python.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz";
+      md5 = "09f24ffd9af9f6cd0f63cb9f4e23d4b2";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   future = super.buildPythonPackage {
     name = "future-0.14.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/83/80/8ef3a11a15f8eaafafa0937b20c1b3f73527e69ab6b3fa1cf94a5a96aabb/future-0.14.3.tar.gz";
       md5 = "e94079b0bd1fc054929e8769fc0f6083";
     };
     meta = {
       license = [ { fullName = "OSI Approved"; } pkgs.lib.licenses.mit ];
     };
   };
   futures = super.buildPythonPackage {
     name = "futures-3.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f8/e7/fc0fcbeb9193ba2d4de00b065e7fd5aecd0679e93ce95a07322b2b1434f4/futures-3.0.2.tar.gz";
       md5 = "42aaf1e4de48d6e871d77dc1f9d96d5a";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   gevent = super.buildPythonPackage {
     name = "gevent-1.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [greenlet];
     src = fetchurl {
       url = "https://pypi.python.org/packages/1b/92/b111f76e54d2be11375b47b213b56687214f258fd9dae703546d30b837be/gevent-1.2.2.tar.gz";
       md5 = "7f0baf355384fe5ff2ecf66853422554";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   gnureadline = super.buildPythonPackage {
     name = "gnureadline-6.3.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/50/64/86085c823cd78f9df9d8e33dce0baa71618016f8860460b82cf6610e1eb3/gnureadline-6.3.8.tar.gz";
       md5 = "ba341f4b907250bd1f47dbc06290604f";
     };
     meta = {
       license = [ { fullName = "GNU General Public License v3 (GPLv3)"; } pkgs.lib.licenses.gpl1 ];
     };
   };
   gprof2dot = super.buildPythonPackage {
     name = "gprof2dot-2017.9.19";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/9d/36/f977122502979f3dfb50704979c9ed70e6b620787942b089bf1af15f5aba/gprof2dot-2017.9.19.tar.gz";
       md5 = "cda2d552bb0d0b9f16e6824a9aabd225";
     };
     meta = {
       license = [ { fullName = "GNU Lesser General Public License v3 or later (LGPLv3+)"; } { fullName = "LGPL"; } ];
     };
   };
   graphviz = super.buildPythonPackage {
-    name = "graphviz-0.8.1";
+    name = "graphviz-0.8.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/a9/a6/ee6721349489a2da6eedd3dba124f2b5ac15ee1e0a7bd4d3cfdc4fff0327/graphviz-0.8.1.zip";
-      md5 = "88d8efa88c02a735b3659fe0feaf0b96";
+      url = "https://pypi.python.org/packages/fa/d1/63b62dee9e55368f60b5ea445e6afb361bb47e692fc27553f3672e16efb8/graphviz-0.8.2.zip";
+      md5 = "50866e780f43e1cb0d073c70424fcaff";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   greenlet = super.buildPythonPackage {
-    name = "greenlet-0.4.12";
+    name = "greenlet-0.4.13";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/be/76/82af375d98724054b7e273b5d9369346937324f9bcc20980b45b068ef0b0/greenlet-0.4.12.tar.gz";
-      md5 = "e8637647d58a26c4a1f51ca393e53c00";
+      url = "https://pypi.python.org/packages/13/de/ba92335e9e76040ca7274224942282a80d54f85e342a5e33c5277c7f87eb/greenlet-0.4.13.tar.gz";
+      md5 = "6e0b9dd5385f81d478451ec8ed1d62b3";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   gunicorn = super.buildPythonPackage {
     name = "gunicorn-19.7.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/30/3a/10bb213cede0cc4d13ac2263316c872a64bf4c819000c8ccd801f1d5f822/gunicorn-19.7.1.tar.gz";
       md5 = "174d3c3cd670a5be0404d84c484e590c";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   html5lib = super.buildPythonPackage {
-    name = "html5lib-1.0b10";
+    name = "html5lib-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
-    propagatedBuildInputs = with self; [six webencodings setuptools];
+    propagatedBuildInputs = with self; [six webencodings];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/97/16/982214624095c1420c75f3bd295d9e658794aafb95fc075823de107e0ae4/html5lib-1.0b10.tar.gz";
-      md5 = "5ada1243b7a863624b2f35245b2186e9";
+      url = "https://pypi.python.org/packages/85/3e/cf449cf1b5004e87510b9368e7a5f1acd8831c2d6691edd3c62a0823f98f/html5lib-1.0.1.tar.gz";
+      md5 = "942a0688d6bdf20d087c9805c40182ad";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   hupper = super.buildPythonPackage {
     name = "hupper-1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2e/07/df892c564dc09bb3cf6f6deb976c26adf9117db75ba218cb4353dbc9d826/hupper-1.0.tar.gz";
       md5 = "26e77da7d5ac5858f59af050d1a6eb5a";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   infrae.cache = super.buildPythonPackage {
     name = "infrae.cache-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Beaker repoze.lru];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bb/f0/e7d5e984cf6592fd2807dc7bc44a93f9d18e04e6a61f87fdfb2622422d74/infrae.cache-1.0.1.tar.gz";
       md5 = "b09076a766747e6ed2a755cc62088e32";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   invoke = super.buildPythonPackage {
     name = "invoke-0.13.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/47/bf/d07ef52fa1ac645468858bbac7cb95b246a972a045e821493d17d89c81be/invoke-0.13.0.tar.gz";
       md5 = "c0d1ed4bfb34eaab551662d8cfee6540";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipaddress = super.buildPythonPackage {
-    name = "ipaddress-1.0.18";
+    name = "ipaddress-1.0.19";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/4e/13/774faf38b445d0b3a844b65747175b2e0500164b7c28d78e34987a5bfe06/ipaddress-1.0.18.tar.gz";
-      md5 = "310c2dfd64eb6f0df44aa8c59f2334a7";
+      url = "https://pypi.python.org/packages/f0/ba/860a4a3e283456d6b7e2ab39ce5cf11a3490ee1a363652ac50abf9f0f5df/ipaddress-1.0.19.tar.gz";
+      md5 = "d0687efaf93a32476d81e90ba0609c57";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   ipdb = super.buildPythonPackage {
     name = "ipdb-0.10.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools ipython];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ad/cc/0e7298e1fbf2efd52667c9354a12aa69fb6f796ce230cca03525051718ef/ipdb-0.10.3.tar.gz";
       md5 = "def1f6ac075d54bdee07e6501263d4fa";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipython = super.buildPythonPackage {
     name = "ipython-5.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools decorator pickleshare simplegeneric traitlets prompt-toolkit Pygments pexpect backports.shutil-get-terminal-size pathlib2 pexpect];
     src = fetchurl {
       url = "https://pypi.python.org/packages/89/63/a9292f7cd9d0090a0f995e1167f3f17d5889dcbc9a175261719c513b9848/ipython-5.1.0.tar.gz";
       md5 = "47c8122420f65b58784cb4b9b4af35e3";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ipython-genutils = super.buildPythonPackage {
     name = "ipython-genutils-0.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/e8/69/fbeffffc05236398ebfcfb512b6d2511c622871dca1746361006da310399/ipython_genutils-0.2.0.tar.gz";
       md5 = "5a4f9781f78466da0ea1a648f3e1f79f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   iso8601 = super.buildPythonPackage {
     name = "iso8601-0.1.12";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/45/13/3db24895497345fb44c4248c08b16da34a9eb02643cea2754b21b5ed08b0/iso8601-0.1.12.tar.gz";
       md5 = "4de940f691c5ea759fb254384c8ddcf6";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   itsdangerous = super.buildPythonPackage {
     name = "itsdangerous-0.24";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz";
       md5 = "a3d55aa79369aef5345c036a8a26307f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   jsonschema = super.buildPythonPackage {
     name = "jsonschema-2.6.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [functools32];
     src = fetchurl {
       url = "https://pypi.python.org/packages/58/b9/171dbb07e18c6346090a37f03c7e74410a1a56123f847efed59af260a298/jsonschema-2.6.0.tar.gz";
       md5 = "50c6b69a373a8b55ff1e0ec6e78f13f4";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   jupyter-client = super.buildPythonPackage {
     name = "jupyter-client-5.0.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [traitlets jupyter-core pyzmq python-dateutil];
     src = fetchurl {
       url = "https://pypi.python.org/packages/e5/6f/65412ed462202b90134b7e761b0b7e7f949e07a549c1755475333727b3d0/jupyter_client-5.0.0.tar.gz";
       md5 = "1acd331b5c9fb4d79dae9939e79f2426";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   jupyter-core = super.buildPythonPackage {
     name = "jupyter-core-4.4.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [traitlets];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b6/2d/2804f4de3a95583f65e5dcb4d7c8c7183124882323758996e867f47e72af/jupyter_core-4.4.0.tar.gz";
       md5 = "7829fc07884ed98459e170f217e2a5ba";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   kombu = super.buildPythonPackage {
     name = "kombu-4.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [amqp];
     src = fetchurl {
       url = "https://pypi.python.org/packages/03/5e/1a47d1e543d4943d65330af4e4406049f443878818fb65bfdc651bb93a96/kombu-4.1.0.tar.gz";
       md5 = "2fb2be9fec0e6514231bba23a3779439";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   lxml = super.buildPythonPackage {
     name = "lxml-3.7.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/39/e8/a8e0b1fa65dd021d48fe21464f71783655f39a41f218293c1c590d54eb82/lxml-3.7.3.tar.gz";
       md5 = "075692ce442e69bbd604d44e21c02753";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   meld3 = super.buildPythonPackage {
     name = "meld3-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/45/a0/317c6422b26c12fe0161e936fc35f36552069ba8e6f7ecbd99bbffe32a5f/meld3-1.0.2.tar.gz";
       md5 = "3ccc78cd79cffd63a751ad7684c02c91";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   mistune = super.buildPythonPackage {
     name = "mistune-0.8.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/9d/be/e06d4346cc608a01dec6bf770d7d0303a4fd6db588b318ced18f5f257145/mistune-0.8.3.tar.gz";
       md5 = "a5e4043e93fb8f9082e27f29eeb5e054";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   mock = super.buildPythonPackage {
     name = "mock-1.0.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/15/45/30273ee91feb60dabb8fbb2da7868520525f02cf910279b3047182feed80/mock-1.0.1.zip";
-      md5 = "869f08d003c289a97c1a6610faf5e913";
+      url = "https://pypi.python.org/packages/a2/52/7edcd94f0afb721a2d559a5b9aae8af4f8f2c79bc63fdbe8a8a6c9b23bbe/mock-1.0.1.tar.gz";
+      md5 = "c3971991738caa55ec7c356bbc154ee2";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   msgpack-python = super.buildPythonPackage {
     name = "msgpack-python-0.4.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/21/27/8a1d82041c7a2a51fcc73675875a5f9ea06c2663e02fcfeb708be1d081a0/msgpack-python-0.4.8.tar.gz";
       md5 = "dcd854fb41ee7584ebbf35e049e6be98";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   nbconvert = super.buildPythonPackage {
     name = "nbconvert-5.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [mistune Jinja2 Pygments traitlets jupyter-core nbformat entrypoints bleach pandocfilters testpath];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b9/a4/d0a0938ad6f5eeb4dea4e73d255c617ef94b0b2849d51194c9bbdb838412/nbconvert-5.3.1.tar.gz";
       md5 = "c128d0d93d02f70a85429a383dae96d2";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   nbformat = super.buildPythonPackage {
     name = "nbformat-4.4.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [ipython-genutils traitlets jsonschema jupyter-core];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6e/0e/160754f7ae3e984863f585a3743b0ed1702043a81245907c8fae2d537155/nbformat-4.4.0.tar.gz";
       md5 = "2d5f873138d9fbc2a3f9eaaebca3b8a1";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   objgraph = super.buildPythonPackage {
     name = "objgraph-3.1.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [graphviz];
     src = fetchurl {
       url = "https://pypi.python.org/packages/be/58/9ca81a20cc837054e94866df1475d899caaa94f3732b8a46006858b015f7/objgraph-3.1.1.tar.gz";
       md5 = "253af9944763377877c3678d8aaebb8b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   packaging = super.buildPythonPackage {
     name = "packaging-15.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/24/c4/185da1304f07047dc9e0c46c31db75c0351bd73458ac3efad7da3dbcfbe1/packaging-15.2.tar.gz";
       md5 = "c16093476f6ced42128bf610e5db3784";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   pandocfilters = super.buildPythonPackage {
     name = "pandocfilters-1.4.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/4c/ea/236e2584af67bb6df960832731a6e5325fd4441de001767da328c33368ce/pandocfilters-1.4.2.tar.gz";
       md5 = "dc391791ef54c7de1572d7b46b63361f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   pathlib2 = super.buildPythonPackage {
     name = "pathlib2-2.3.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six scandir];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a1/14/df0deb867c2733f7d857523c10942b3d6612a1b222502fdffa9439943dfb/pathlib2-2.3.0.tar.gz";
       md5 = "89c90409d11fd5947966b6a30a47d18c";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   peppercorn = super.buildPythonPackage {
     name = "peppercorn-0.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/45/ec/a62ec317d1324a01567c5221b420742f094f05ee48097e5157d32be3755c/peppercorn-0.5.tar.gz";
       md5 = "f08efbca5790019ab45d76b7244abd40";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pexpect = super.buildPythonPackage {
-    name = "pexpect-4.3.0";
+    name = "pexpect-4.4.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [ptyprocess];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/f8/44/5466c30e49762bb92e442bbdf4472d6904608d211258eb3198a11f0309a4/pexpect-4.3.0.tar.gz";
-      md5 = "047a486dcd26134b74f2e67046bb61a0";
+      url = "https://pypi.python.org/packages/fa/c3/60c0cbf96f242d0b47a82e9ca634dcd6dcb043832cf05e17540812e1c707/pexpect-4.4.0.tar.gz";
+      md5 = "e9b07f0765df8245ac72201d757baaef";
     };
     meta = {
       license = [ pkgs.lib.licenses.isc { fullName = "ISC License (ISCL)"; } ];
     };
   };
   pickleshare = super.buildPythonPackage {
     name = "pickleshare-0.7.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pathlib2];
     src = fetchurl {
       url = "https://pypi.python.org/packages/69/fe/dd137d84daa0fd13a709e448138e310d9ea93070620c9db5454e234af525/pickleshare-0.7.4.tar.gz";
       md5 = "6a9e5dd8dfc023031f6b7b3f824cab12";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   plaster = super.buildPythonPackage {
     name = "plaster-1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/37/e1/56d04382d718d32751017d32f351214384e529b794084eee20bb52405563/plaster-1.0.tar.gz";
       md5 = "80e6beb4760c16fea31754babcc0576e";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   plaster-pastedeploy = super.buildPythonPackage {
     name = "plaster-pastedeploy-0.4.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [PasteDeploy plaster];
     src = fetchurl {
       url = "https://pypi.python.org/packages/2c/62/0daf9c0be958e785023e583e51baac15863699e956bfb3d448898d80edd8/plaster_pastedeploy-0.4.2.tar.gz";
       md5 = "58fd7852002909378e818c9d5b71e90a";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   prompt-toolkit = super.buildPythonPackage {
     name = "prompt-toolkit-1.0.15";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six wcwidth];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8a/ad/cf6b128866e78ad6d7f1dc5b7f99885fb813393d9860778b2984582e81b5/prompt_toolkit-1.0.15.tar.gz";
       md5 = "8fe70295006dbc8afedd43e5eba99032";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   psutil = super.buildPythonPackage {
-    name = "psutil-5.4.0";
+    name = "psutil-5.4.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/8d/96/1fc6468be91521192861966c40bd73fdf8b065eae6d82dd0f870b9825a65/psutil-5.4.0.tar.gz";
-      md5 = "01af6219b1e8fcfd53603023967713bf";
+      url = "https://pypi.python.org/packages/e2/e1/600326635f97fee89bf8426fef14c5c29f4849c79f68fd79f433d8c1bd96/psutil-5.4.3.tar.gz";
+      md5 = "3b291833dbea631db9d271aa602a169a";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   psycopg2 = super.buildPythonPackage {
     name = "psycopg2-2.7.3.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/dd/47/000b405d73ca22980684fd7bd3318690cc03cfa3b2ae1c5b7fff8050b28a/psycopg2-2.7.3.2.tar.gz";
       md5 = "8114e672d5f23fa5329874a4314fbd6f";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 { fullName = "GNU Library or Lesser General Public License (LGPL)"; } { fullName = "LGPL with exceptions or ZPL"; } ];
     };
   };
   ptyprocess = super.buildPythonPackage {
     name = "ptyprocess-0.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/51/83/5d07dc35534640b06f9d9f1a1d2bc2513fb9cc7595a1b0e28ae5477056ce/ptyprocess-0.5.2.tar.gz";
       md5 = "d3b8febae1b8c53b054bd818d0bb8665";
     };
     meta = {
       license = [  ];
     };
   };
   py = super.buildPythonPackage {
     name = "py-1.5.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/90/e3/e075127d39d35f09a500ebb4a90afd10f9ef0a1d28a6d09abeec0e444fdd/py-1.5.2.tar.gz";
       md5 = "279ca69c632069e1b71e11b14641ca28";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   py-bcrypt = super.buildPythonPackage {
     name = "py-bcrypt-0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/68/b1/1c3068c5c4d2e35c48b38dcc865301ebfdf45f54507086ac65ced1fd3b3d/py-bcrypt-0.4.tar.gz";
       md5 = "dd8b367d6b716a2ea2e72392525f4e36";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   py-gfm = super.buildPythonPackage {
     name = "py-gfm-0.1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools Markdown];
     src = fetchurl {
       url = "https://pypi.python.org/packages/12/e4/6b3d8678da04f97d7490d8264d8de51c2dc9fb91209ccee9c515c95e14c5/py-gfm-0.1.3.tar.gz";
       md5 = "e588d9e69640a241b97e2c59c22527a6";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   pycrypto = super.buildPythonPackage {
     name = "pycrypto-2.6.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/60/db/645aa9af249f059cc3a368b118de33889219e0362141e75d4eaf6f80f163/pycrypto-2.6.1.tar.gz";
       md5 = "55a61a054aa66812daf5161a0d5d7eda";
     };
     meta = {
       license = [ pkgs.lib.licenses.publicDomain ];
     };
   };
   pycurl = super.buildPythonPackage {
     name = "pycurl-7.19.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/6c/48/13bad289ef6f4869b1d8fc11ae54de8cfb3cc4a2eb9f7419c506f763be46/pycurl-7.19.5.tar.gz";
       md5 = "47b4eac84118e2606658122104e62072";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit { fullName = "LGPL/MIT"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
     };
   };
   pyflakes = super.buildPythonPackage {
     name = "pyflakes-0.8.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/75/22/a90ec0252f4f87f3ffb6336504de71fe16a49d69c4538dae2f12b9360a38/pyflakes-0.8.1.tar.gz";
       md5 = "905fe91ad14b912807e8fdc2ac2e2c23";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pygments-markdown-lexer = super.buildPythonPackage {
     name = "pygments-markdown-lexer-0.1.0.dev39";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [Pygments];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c3/12/674cdee66635d638cedb2c5d9c85ce507b7b2f91bdba29e482f1b1160ff6/pygments-markdown-lexer-0.1.0.dev39.zip";
       md5 = "6360fe0f6d1f896e35b7a0142ce6459c";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   pyparsing = super.buildPythonPackage {
     name = "pyparsing-1.5.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/2e/26/e8fb5b4256a5f5036be7ce115ef8db8d06bc537becfbdc46c6af008314ee/pyparsing-1.5.7.zip";
-      md5 = "b86854857a368d6ccb4d5b6e76d0637f";
+      url = "https://pypi.python.org/packages/6f/2c/47457771c02a8ff0f302b695e094ec309e30452232bd79198ee94fda689f/pyparsing-1.5.7.tar.gz";
+      md5 = "9be0fcdcc595199c646ab317c1d9a709";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pyramid = super.buildPythonPackage {
     name = "pyramid-1.9.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools WebOb repoze.lru zope.interface zope.deprecation venusian translationstring PasteDeploy plaster plaster-pastedeploy hupper];
     src = fetchurl {
       url = "https://pypi.python.org/packages/9a/57/73447be9e7d0512d601e3f0a1fb9d7d1efb941911f49efdfe036d2826507/pyramid-1.9.1.tar.gz";
       md5 = "0163e19c58c2d12976a3b6fdb57e052d";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pyramid-beaker = super.buildPythonPackage {
     name = "pyramid-beaker-0.8";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid Beaker];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d9/6e/b85426e00fd3d57f4545f74e1c3828552d8700f13ededeef9233f7bca8be/pyramid_beaker-0.8.tar.gz";
       md5 = "22f14be31b06549f80890e2c63a93834";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pyramid-debugtoolbar = super.buildPythonPackage {
     name = "pyramid-debugtoolbar-4.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid pyramid-mako repoze.lru Pygments ipaddress];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a4/40/f09d8800bfc3c09bdb6c95f37bb61c890dc62c19c4e7caa304da7aa77403/pyramid_debugtoolbar-4.3.tar.gz";
       md5 = "9c49029e9f0695130499ef6416ffaaf8";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } pkgs.lib.licenses.bsdOriginal ];
     };
   };
   pyramid-jinja2 = super.buildPythonPackage {
     name = "pyramid-jinja2-2.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid zope.deprecation Jinja2 MarkupSafe];
     src = fetchurl {
       url = "https://pypi.python.org/packages/d8/80/d60a7233823de22ce77bd864a8a83736a1fe8b49884b08303a2e68b2c853/pyramid_jinja2-2.7.tar.gz";
       md5 = "c2f8b2cd7b73a6f1d9a311fcfaf4fb92";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pyramid-mako = super.buildPythonPackage {
     name = "pyramid-mako-1.0.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pyramid Mako];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f1/92/7e69bcf09676d286a71cb3bbb887b16595b96f9ba7adbdc239ffdd4b1eb9/pyramid_mako-1.0.2.tar.gz";
       md5 = "ee25343a97eb76bd90abdc2a774eb48a";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   pysqlite = super.buildPythonPackage {
     name = "pysqlite-2.8.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/42/02/981b6703e3c83c5b25a829c6e77aad059f9481b0bbacb47e6e8ca12bd731/pysqlite-2.8.3.tar.gz";
       md5 = "033f17b8644577715aee55e8832ac9fc";
     };
     meta = {
       license = [ { fullName = "zlib/libpng License"; } { fullName = "zlib/libpng license"; } ];
     };
   };
   pytest = super.buildPythonPackage {
     name = "pytest-3.2.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [py setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/1f/f8/8cd74c16952163ce0db0bd95fdd8810cbf093c08be00e6e665ebf0dc3138/pytest-3.2.5.tar.gz";
       md5 = "6dbe9bb093883f75394a689a1426ac6f";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-catchlog = super.buildPythonPackage {
     name = "pytest-catchlog-1.2.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [py pytest];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f2/2b/2faccdb1a978fab9dd0bf31cca9f6847fbe9184a0bdcc3011ac41dd44191/pytest-catchlog-1.2.2.zip";
       md5 = "09d890c54c7456c818102b7ff8c182c8";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-cov = super.buildPythonPackage {
     name = "pytest-cov-2.5.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytest coverage];
     src = fetchurl {
       url = "https://pypi.python.org/packages/24/b4/7290d65b2f3633db51393bdf8ae66309b37620bc3ec116c5e357e3e37238/pytest-cov-2.5.1.tar.gz";
       md5 = "5acf38d4909e19819eb5c1754fbfc0ac";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal pkgs.lib.licenses.mit ];
     };
   };
   pytest-profiling = super.buildPythonPackage {
     name = "pytest-profiling-1.2.11";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six pytest gprof2dot];
     src = fetchurl {
       url = "https://pypi.python.org/packages/c0/4a/b4aa786e93c07a86f1f87c581a36bf355a9e06a9da7e00dbd05047626bd2/pytest-profiling-1.2.11.tar.gz";
       md5 = "9ef6b60248731be5d44477980408e8f7";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-runner = super.buildPythonPackage {
     name = "pytest-runner-3.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/65/b4/ae89338cd2d81e2cc54bd6db2e962bfe948f612303610d68ab24539ac2d1/pytest-runner-3.0.tar.gz";
       md5 = "8f8363a52bbabc4cedd5e239beb2ba11";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pytest-sugar = super.buildPythonPackage {
     name = "pytest-sugar-0.9.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytest termcolor];
     src = fetchurl {
       url = "https://pypi.python.org/packages/49/d8/c5ff6cca3ce2ebd8b73eec89779bf6b4a7737456a70e8ea4d44c1ff90f71/pytest-sugar-0.9.0.tar.gz";
       md5 = "89fbff17277fa6a95a560a04b68cb9f9";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   pytest-timeout = super.buildPythonPackage {
     name = "pytest-timeout-1.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pytest];
     src = fetchurl {
       url = "https://pypi.python.org/packages/cc/b7/b2a61365ea6b6d2e8881360ae7ed8dad0327ad2df89f2f0be4a02304deb2/pytest-timeout-1.2.0.tar.gz";
       md5 = "83607d91aa163562c7ee835da57d061d";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit { fullName = "DFSG approved"; } ];
     };
   };
   python-dateutil = super.buildPythonPackage {
     name = "python-dateutil-2.6.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/54/bb/f1db86504f7a49e1d9b9301531181b00a1c7325dc85a29160ee3eaa73a54/python-dateutil-2.6.1.tar.gz";
       md5 = "db38f6b4511cefd76014745bb0cc45a4";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal { fullName = "Simplified BSD"; } ];
     };
   };
   python-editor = super.buildPythonPackage {
     name = "python-editor-1.0.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/65/1e/adf6e000ea5dc909aa420352d6ba37f16434c8a3c2fa030445411a1ed545/python-editor-1.0.3.tar.gz";
       md5 = "0aca5f2ef176ce68e98a5b7e31372835";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 { fullName = "Apache"; } ];
     };
   };
   python-ldap = super.buildPythonPackage {
     name = "python-ldap-2.4.45";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
       url = "https://pypi.python.org/packages/ce/52/6b5372d0166820f4a4b0a88ed73dc7504219355049fc1d266d8ccdb7942e/python-ldap-2.4.45.tar.gz";
       md5 = "6108e189a44eea8bc7d1cc281c222978";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   python-memcached = super.buildPythonPackage {
     name = "python-memcached-1.58";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f7/62/14b2448cfb04427366f24104c9da97cf8ea380d7258a3233f066a951a8d8/python-memcached-1.58.tar.gz";
       md5 = "23b258105013d14d899828d334e6b044";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   python-pam = super.buildPythonPackage {
     name = "python-pam-1.8.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/de/8c/f8f5d38b4f26893af267ea0b39023d4951705ab0413a39e0cf7cf4900505/python-pam-1.8.2.tar.gz";
       md5 = "db71b6b999246fb05d78ecfbe166629d";
     };
     meta = {
       license = [ { fullName = "License :: OSI Approved :: MIT License"; } pkgs.lib.licenses.mit ];
     };
   };
   pytz = super.buildPythonPackage {
-    name = "pytz-2017.3";
+    name = "pytz-2018.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/60/88/d3152c234da4b2a1f7a989f89609ea488225eaea015bc16fbde2b3fdfefa/pytz-2017.3.zip";
-      md5 = "7006b56c0d68a162d9fe57d4249c3171";
+      url = "https://pypi.python.org/packages/1b/50/4cdc62fc0753595fc16c8f722a89740f487c6e5670c644eb8983946777be/pytz-2018.3.tar.gz";
+      md5 = "abb07c09c79f78d7c04f222a550c99ef";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   pyzmq = super.buildPythonPackage {
     name = "pyzmq-14.6.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8a/3b/5463d5a9d712cd8bbdac335daece0d69f6a6792da4e3dd89956c0db4e4e6/pyzmq-14.6.0.tar.gz";
       md5 = "395b5de95a931afa5b14c9349a5b8024";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal { fullName = "LGPL+BSD"; } { fullName = "GNU Library or Lesser General Public License (LGPL)"; } ];
     };
   };
   recaptcha-client = super.buildPythonPackage {
     name = "recaptcha-client-1.0.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0a/ea/5f2fbbfd894bdac1c68ef8d92019066cfcf9fbff5fe3d728d2b5c25c8db4/recaptcha-client-1.0.6.tar.gz";
       md5 = "74228180f7e1fb76c4d7089160b0d919";
     };
     meta = {
       license = [ { fullName = "MIT/X11"; } ];
     };
   };
   redis = super.buildPythonPackage {
     name = "redis-2.10.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/09/8d/6d34b75326bf96d4139a2ddd8e74b80840f800a0a79f9294399e212cb9a7/redis-2.10.6.tar.gz";
       md5 = "048348d8cfe0b5d0bba2f4d835005c3b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   repoze.lru = super.buildPythonPackage {
     name = "repoze.lru-0.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/12/bc/595a77c4b5e204847fdf19268314ef59c85193a9dc9f83630fc459c0fee5/repoze.lru-0.7.tar.gz";
       md5 = "c08cc030387e0b1fc53c5c7d964b35e2";
     };
     meta = {
       license = [ { fullName = "Repoze Public License"; } { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   requests = super.buildPythonPackage {
     name = "requests-2.9.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f9/6d/07c44fb1ebe04d069459a189e7dab9e4abfe9432adcd4477367c25332748/requests-2.9.1.tar.gz";
       md5 = "0b7f480d19012ec52bab78292efd976d";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
   rhodecode-enterprise-ce = super.buildPythonPackage {
-    name = "rhodecode-enterprise-ce-4.11.6";
+    name = "rhodecode-enterprise-ce-4.12.0";
     buildInputs = with self; [pytest py pytest-cov pytest-sugar pytest-runner pytest-catchlog pytest-profiling gprof2dot pytest-timeout mock WebTest cov-core coverage configobj];
     doCheck = true;
-    propagatedBuildInputs = with self; [setuptools-scm amqp authomatic Babel Beaker celery Chameleon channelstream click colander configobj cssselect decorator deform docutils dogpile.cache dogpile.core ecdsa FormEncode future futures gnureadline infrae.cache iso8601 itsdangerous Jinja2 billiard kombu lxml Mako Markdown MarkupSafe msgpack-python MySQL-python objgraph packaging Paste PasteDeploy PasteScript pathlib2 peppercorn psutil psycopg2 py-bcrypt pycrypto pycurl pyflakes pygments-markdown-lexer Pygments pyparsing pyramid-beaker pyramid-debugtoolbar pyramid-jinja2 pyramid-mako pyramid pysqlite python-dateutil python-ldap python-memcached python-pam pytz pyzmq py-gfm recaptcha-client redis repoze.lru requests Routes setproctitle simplejson six SQLAlchemy sshpubkeys subprocess32 supervisor Tempita translationstring trollius urllib3 URLObject venusian WebError WebHelpers2 WebHelpers WebOb Whoosh wsgiref zope.cachedescriptors zope.deprecation zope.event zope.interface nbconvert bleach nbformat jupyter-client alembic invoke bumpversion transifex-client gevent greenlet gunicorn waitress uWSGI ipdb ipython CProfileV bottle rhodecode-tools appenlight-client pytest py pytest-cov pytest-sugar pytest-runner pytest-catchlog pytest-profiling gprof2dot pytest-timeout mock WebTest cov-core coverage];
+    propagatedBuildInputs = with self; [setuptools-scm amqp authomatic Babel Beaker celery Chameleon channelstream click colander configobj cssselect decorator deform docutils dogpile.cache dogpile.core ecdsa FormEncode future futures gnureadline infrae.cache iso8601 itsdangerous Jinja2 billiard kombu lxml Mako Markdown MarkupSafe msgpack-python MySQL-python objgraph packaging Paste PasteDeploy PasteScript pathlib2 peppercorn psutil psycopg2 py-bcrypt pycrypto pycurl pyflakes pygments-markdown-lexer Pygments pyparsing pyramid-beaker pyramid-debugtoolbar pyramid-jinja2 pyramid-mako pyramid pysqlite python-dateutil python-ldap python-memcached python-pam pytz tzlocal pyzmq py-gfm recaptcha-client redis repoze.lru requests Routes setproctitle simplejson six SQLAlchemy sshpubkeys subprocess32 supervisor Tempita translationstring trollius urllib3 URLObject venusian WebError WebHelpers2 WebHelpers WebOb Whoosh wsgiref zope.cachedescriptors zope.deprecation zope.event zope.interface nbconvert bleach nbformat jupyter-client alembic invoke bumpversion transifex-client gevent greenlet gunicorn waitress uWSGI ipdb ipython CProfileV bottle rhodecode-tools appenlight-client pytest py pytest-cov pytest-sugar pytest-runner pytest-catchlog pytest-profiling gprof2dot pytest-timeout mock WebTest cov-core coverage];
     src = ./.;
     meta = {
       license = [ { fullName = "Affero GNU General Public License v3 or later (AGPLv3+)"; } { fullName = "AGPLv3, and Commercial License"; } ];
     };
   };
   rhodecode-tools = super.buildPythonPackage {
     name = "rhodecode-tools-0.14.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [click future six Mako MarkupSafe requests elasticsearch elasticsearch-dsl urllib3 Whoosh];
     src = fetchurl {
       url = "https://code.rhodecode.com/rhodecode-tools-ce/archive/v0.14.1.tar.gz?md5=0b9c2caad160b68889f8172ea54af7b2";
       md5 = "0b9c2caad160b68889f8172ea54af7b2";
     };
     meta = {
       license = [ { fullName = "AGPLv3 and Proprietary"; } ];
     };
   };
   scandir = super.buildPythonPackage {
-    name = "scandir-1.6";
+    name = "scandir-1.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/77/3f/916f524f50ee65e3f465a280d2851bd63685250fddb3020c212b3977664d/scandir-1.6.tar.gz";
-      md5 = "0180ddb97c96cbb2d4f25d2ae11c64ac";
+      url = "https://pypi.python.org/packages/13/bb/e541b74230bbf7a20a3949a2ee6631be299378a784f5445aa5d0047c192b/scandir-1.7.tar.gz";
+      md5 = "037e5f24d1a0e78b17faca72dea9555f";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal { fullName = "New BSD License"; } ];
     };
   };
   setproctitle = super.buildPythonPackage {
     name = "setproctitle-1.1.10";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5a/0d/dc0d2234aacba6cf1a729964383e3452c52096dc695581248b548786f2b3/setproctitle-1.1.10.tar.gz";
       md5 = "2dcdd1b761700a5a13252fea3dfd1977";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   setuptools = super.buildPythonPackage {
     name = "setuptools-30.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/1e/43/002c8616db9a3e7be23c2556e39b90a32bb40ba0dc652de1999d5334d372/setuptools-30.1.0.tar.gz";
       md5 = "cac497f42e5096ac8df29e38d3f81c3e";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   setuptools-scm = super.buildPythonPackage {
     name = "setuptools-scm-1.15.6";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/03/6d/aafdd01edd227ee879b691455bf19895091872af7e48192bea1758c82032/setuptools_scm-1.15.6.tar.gz";
       md5 = "f17493d53f0d842bb0152f214775640b";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   simplegeneric = super.buildPythonPackage {
     name = "simplegeneric-0.8.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip";
       md5 = "f9c1fab00fd981be588fc32759f474e3";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   simplejson = super.buildPythonPackage {
     name = "simplejson-3.11.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/08/48/c97b668d6da7d7bebe7ea1817a6f76394b0ec959cb04214ca833c34359df/simplejson-3.11.1.tar.gz";
       md5 = "6e2f1bd5fb0a926facf5d89d217a7183";
     };
     meta = {
       license = [ { fullName = "Academic Free License (AFL)"; } pkgs.lib.licenses.mit ];
     };
   };
   six = super.buildPythonPackage {
     name = "six-1.11.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz";
       md5 = "d12789f9baf7e9fb2524c0c64f1773f8";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   sshpubkeys = super.buildPythonPackage {
     name = "sshpubkeys-2.2.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [pycrypto ecdsa];
     src = fetchurl {
       url = "https://pypi.python.org/packages/27/da/337fabeb3dca6b62039a93ceaa636f25065e0ae92b575b1235342076cf0a/sshpubkeys-2.2.0.tar.gz";
       md5 = "458e45f6b92b1afa84f0ffe1f1c90935";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   subprocess32 = super.buildPythonPackage {
     name = "subprocess32-3.2.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/b8/2f/49e53b0d0e94611a2dc624a1ad24d41b6d94d0f1b0a078443407ea2214c2/subprocess32-3.2.7.tar.gz";
       md5 = "824c801e479d3e916879aae3e9c15e16";
     };
     meta = {
       license = [ pkgs.lib.licenses.psfl ];
     };
   };
   supervisor = super.buildPythonPackage {
-    name = "supervisor-3.3.3";
+    name = "supervisor-3.3.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [meld3];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/31/7e/788fc6566211e77c395ea272058eb71299c65cc5e55b6214d479c6c2ec9a/supervisor-3.3.3.tar.gz";
-      md5 = "0fe86dfec4e5c5d98324d24c4cf944bd";
+      url = "https://pypi.python.org/packages/44/60/698e54b4a4a9b956b2d709b4b7b676119c833d811d53ee2500f1b5e96dc3/supervisor-3.3.4.tar.gz";
+      md5 = "f1814d71d820ddfa8c86d46a72314cec";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   termcolor = super.buildPythonPackage {
     name = "termcolor-1.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz";
       md5 = "043e89644f8909d462fbbfa511c768df";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   testpath = super.buildPythonPackage {
     name = "testpath-0.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/f4/8b/b71e9ee10e5f751e9d959bc750ab122ba04187f5aa52aabdc4e63b0e31a7/testpath-0.3.1.tar.gz";
       md5 = "2cd5ed5522fda781bb497c9d80ae2fc9";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   traitlets = super.buildPythonPackage {
     name = "traitlets-4.3.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [ipython-genutils six decorator enum34];
     src = fetchurl {
       url = "https://pypi.python.org/packages/a5/98/7f5ef2fe9e9e071813aaf9cb91d1a732e0a68b6c44a32b38cb8e14c3f069/traitlets-4.3.2.tar.gz";
       md5 = "3068663f2f38fd939a9eb3a500ccc154";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   transifex-client = super.buildPythonPackage {
     name = "transifex-client-0.12.5";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [urllib3 six];
     src = fetchurl {
       url = "https://pypi.python.org/packages/7b/86/60f31a0c9b8d0b1266ce15b6c80b55f88522140c8acfc395d5aec5e23475/transifex-client-0.12.5.tar.gz";
       md5 = "e6e278117b23f60702c06e203b7e51ae";
     };
     meta = {
       license = [ pkgs.lib.licenses.gpl2 ];
     };
   };
   translationstring = super.buildPythonPackage {
     name = "translationstring-1.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/5e/eb/bee578cc150b44c653b63f5ebe258b5d0d812ddac12497e5f80fcad5d0b4/translationstring-1.3.tar.gz";
       md5 = "a4b62e0f3c189c783a1685b3027f7c90";
     };
     meta = {
       license = [ { fullName = "BSD-like (http://repoze.org/license.html)"; } ];
     };
   };
   trollius = super.buildPythonPackage {
     name = "trollius-1.0.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [futures];
     src = fetchurl {
       url = "https://pypi.python.org/packages/aa/e6/4141db437f55e6ee7a3fb69663239e3fde7841a811b4bef293145ad6c836/trollius-1.0.4.tar.gz";
       md5 = "3631a464d49d0cbfd30ab2918ef2b783";
     };
     meta = {
       license = [ pkgs.lib.licenses.asl20 ];
     };
   };
+  tzlocal = super.buildPythonPackage {
+    name = "tzlocal-1.5.1";
+    buildInputs = with self; [];
+    doCheck = false;
+    propagatedBuildInputs = with self; [pytz];
+    src = fetchurl {
+      url = "https://pypi.python.org/packages/cb/89/e3687d3ed99bc882793f82634e9824e62499fdfdc4b1ae39e211c5b05017/tzlocal-1.5.1.tar.gz";
+      md5 = "4553be891efa0812c4adfb0c6e818eec";
+    };
+    meta = {
+      license = [ pkgs.lib.licenses.mit ];
+    };
+  };
   uWSGI = super.buildPythonPackage {
     name = "uWSGI-2.0.15";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/bb/0a/45e5aa80dc135889594bb371c082d20fb7ee7303b174874c996888cc8511/uwsgi-2.0.15.tar.gz";
       md5 = "fc50bd9e83b7602fa474b032167010a7";
     };
     meta = {
       license = [ pkgs.lib.licenses.gpl2 ];
     };
   };
   urllib3 = super.buildPythonPackage {
     name = "urllib3-1.16";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3b/f0/e763169124e3f5db0926bc3dbfcd580a105f9ca44cf5d8e6c7a803c9f6b5/urllib3-1.16.tar.gz";
       md5 = "fcaab1c5385c57deeb7053d3d7d81d59";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   venusian = super.buildPythonPackage {
     name = "venusian-1.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/38/24/b4b470ab9e0a2e2e9b9030c7735828c8934b4c6b45befd1bb713ec2aeb2d/venusian-1.1.0.tar.gz";
       md5 = "56bc5e6756e4bda37bcdb94f74a72b8f";
     };
     meta = {
       license = [ { fullName = "BSD-derived (http://www.repoze.org/LICENSE.txt)"; } ];
     };
   };
   vine = super.buildPythonPackage {
     name = "vine-1.1.4";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/32/23/36284986e011f3c130d802c3c66abd8f1aef371eae110ddf80c5ae22e1ff/vine-1.1.4.tar.gz";
       md5 = "9fdb971e7fd15b181b84f3bfcf20d11c";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   waitress = super.buildPythonPackage {
     name = "waitress-1.1.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/3c/68/1c10dd5c556872ceebe88483b0436140048d39de83a84a06a8baa8136f4f/waitress-1.1.0.tar.gz";
       md5 = "0f1eb7fdfdbf2e6d18decbda1733045c";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   wcwidth = super.buildPythonPackage {
     name = "wcwidth-0.1.7";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/55/11/e4a2bb08bb450fdbd42cc709dd40de4ed2c472cf0ccb9e64af22279c5495/wcwidth-0.1.7.tar.gz";
       md5 = "b3b6a0a08f0c8a34d1de8cf44150a4ad";
     };
     meta = {
       license = [ pkgs.lib.licenses.mit ];
     };
   };
   webencodings = super.buildPythonPackage {
     name = "webencodings-0.5.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/0b/02/ae6ceac1baeda530866a85075641cec12989bd8d31af6d5ab4a3e8c92f47/webencodings-0.5.1.tar.gz";
       md5 = "32f6e261d52e57bf7e1c4d41546d15b8";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   ws4py = super.buildPythonPackage {
-    name = "ws4py-0.4.2";
+    name = "ws4py-0.4.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/b8/98/a90f1d96ffcb15dfc220af524ce23e0a5881258dafa197673357ce1683dd/ws4py-0.4.2.tar.gz";
-      md5 = "f0603ae376707a58d205bd87a67758a2";
+      url = "https://pypi.python.org/packages/fa/a1/33c43a4304ac3b4dc81deb93cbd329de9297dd034d75c47cce64fda806bc/ws4py-0.4.3.tar.gz";
+      md5 = "d5834cf7d3965bb0da31bbb02bd8513a";
     };
     meta = {
       license = [ pkgs.lib.licenses.bsdOriginal ];
     };
   };
   wsgiref = super.buildPythonPackage {
     name = "wsgiref-0.1.2";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [];
     src = fetchurl {
       url = "https://pypi.python.org/packages/41/9e/309259ce8dff8c596e8c26df86dbc4e848b9249fd36797fd60be456f03fc/wsgiref-0.1.2.zip";
       md5 = "29b146e6ebd0f9fb119fe321f7bcf6cb";
     };
     meta = {
       license = [ { fullName = "PSF or ZPL"; } ];
     };
   };
   zope.cachedescriptors = super.buildPythonPackage {
-    name = "zope.cachedescriptors-4.0.0";
+    name = "zope.cachedescriptors-4.3.1";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/40/33/694b6644c37f28553f4b9f20b3c3a20fb709a22574dff20b5bdffb09ecd5/zope.cachedescriptors-4.0.0.tar.gz";
-      md5 = "8d308de8c936792c8e758058fcb7d0f0";
+      url = "https://pypi.python.org/packages/2f/89/ebe1890cc6d3291ebc935558fa764d5fffe571018dbbee200e9db78762cb/zope.cachedescriptors-4.3.1.tar.gz";
+      md5 = "42f3693f43bc93f3b1eb86940f58acf3";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   zope.deprecation = super.buildPythonPackage {
-    name = "zope.deprecation-4.1.2";
+    name = "zope.deprecation-4.3.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/c1/d3/3919492d5e57d8dd01b36f30b34fc8404a30577392b1eb817c303499ad20/zope.deprecation-4.1.2.tar.gz";
-      md5 = "e9a663ded58f4f9f7881beb56cae2782";
+      url = "https://pypi.python.org/packages/a1/18/2dc5e6bfe64fdc3b79411b67464c55bb0b43b127051a20f7f492ab767758/zope.deprecation-4.3.0.tar.gz";
+      md5 = "2166b2cb7e0e96a21104e6f8f9b696bb";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   zope.event = super.buildPythonPackage {
-    name = "zope.event-4.0.3";
+    name = "zope.event-4.3.0";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/c1/29/91ba884d7d6d96691df592e9e9c2bfa57a47040ec1ff47eff18c85137152/zope.event-4.0.3.tar.gz";
-      md5 = "9a3780916332b18b8b85f522bcc3e249";
+      url = "https://pypi.python.org/packages/9e/d0/54ba59f19a0635f6591b74be259cf6fbf67e73f4edda27b5cd0cf4d26efa/zope.event-4.3.0.tar.gz";
+      md5 = "8ca737960741c6fd112972f3313303bd";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
   zope.interface = super.buildPythonPackage {
-    name = "zope.interface-4.1.3";
+    name = "zope.interface-4.4.3";
     buildInputs = with self; [];
     doCheck = false;
     propagatedBuildInputs = with self; [setuptools];
     src = fetchurl {
-      url = "https://pypi.python.org/packages/9d/81/2509ca3c6f59080123c1a8a97125eb48414022618cec0e64eb1313727bfe/zope.interface-4.1.3.tar.gz";
-      md5 = "9ae3d24c0c7415deb249dd1a132f0f79";
+      url = "https://pypi.python.org/packages/bd/d2/25349ed41f9dcff7b3baf87bd88a4c82396cf6e02f1f42bb68657a3132af/zope.interface-4.4.3.tar.gz";
+      md5 = "8700a4f527c1203b34b10c2b4e7a6912";
     };
     meta = {
       license = [ pkgs.lib.licenses.zpt21 ];
     };
   };
 
 ### Test requirements
 
   
 }

requirements.txt

@@ -1,131 +1,132 b''
 ## core
 setuptools==30.1.0
 setuptools-scm==1.15.6
 
 amqp==2.2.2
 authomatic==0.1.0.post1
 Babel==1.3
-Beaker==1.9.0
+Beaker==1.9.1
 celery==4.1.0
 Chameleon==2.24
 channelstream==0.5.2
 click==6.6
 colander==1.4.0
 configobj==5.0.6
 cssselect==1.0.1
 decorator==4.1.2
 deform==2.0.4
 docutils==0.14.0
 dogpile.cache==0.6.4
 dogpile.core==0.4.1
 ecdsa==0.13
 FormEncode==1.2.4
 future==0.14.3
 futures==3.0.2
 gnureadline==6.3.8
 infrae.cache==1.0.1
 iso8601==0.1.12
 itsdangerous==0.24
 Jinja2==2.9.6
 billiard==3.5.0.3
 kombu==4.1.0
 lxml==3.7.3
 Mako==1.0.7
-Markdown==2.6.9
+Markdown==2.6.11
 MarkupSafe==1.0.0
 msgpack-python==0.4.8
 MySQL-python==1.2.5
 objgraph==3.1.1
 packaging==15.2
 Paste==2.0.3
 PasteDeploy==1.5.2
 PasteScript==2.0.2
 pathlib2==2.3.0
 peppercorn==0.5
-psutil==5.4.0
+psutil==5.4.3
 psycopg2==2.7.3.2
 py-bcrypt==0.4
 pycrypto==2.6.1
 pycurl==7.19.5
 pyflakes==0.8.1
 pygments-markdown-lexer==0.1.0.dev39
 Pygments==2.2.0
 pyparsing==1.5.7
 pyramid-beaker==0.8
 pyramid-debugtoolbar==4.3.0
 pyramid-jinja2==2.7
 pyramid-mako==1.0.2
 pyramid==1.9.1
 pysqlite==2.8.3
 python-dateutil
 python-ldap==2.4.45
 python-memcached==1.58
 python-pam==1.8.2
-pytz==2017.3
+pytz==2018.3
+tzlocal==1.5.1
 pyzmq==14.6.0
 py-gfm==0.1.3
 recaptcha-client==1.0.6
 redis==2.10.6
 repoze.lru==0.7
 requests==2.9.1
 Routes==2.4.1
 setproctitle==1.1.10
 simplejson==3.11.1
 six==1.11.0
 SQLAlchemy==1.1.15
 sshpubkeys==2.2.0
 subprocess32==3.2.7
-supervisor==3.3.3
+supervisor==3.3.4
 Tempita==0.5.2
 translationstring==1.3
 trollius==1.0.4
 urllib3==1.16
 URLObject==2.4.0
 venusian==1.1.0
 WebError==0.10.3
 WebHelpers2==2.0
 WebHelpers==1.3
 WebOb==1.7.4
 Whoosh==2.7.4
 wsgiref==0.1.2
-zope.cachedescriptors==4.0.0
-zope.deprecation==4.1.2
-zope.event==4.0.3
-zope.interface==4.1.3
+zope.cachedescriptors==4.3.1
+zope.deprecation==4.3.0
+zope.event==4.3.0
+zope.interface==4.4.3
 
 
 # IPYTHON RENDERING
 # entrypoints backport, pypi version doesn't support egg installs
 https://code.rhodecode.com/upstream/entrypoints/archive/96e6d645684e1af3d7df5b5272f3fe85a546b233.tar.gz?md5=7db37771aea9ac9fefe093e5d6987313#egg=entrypoints==0.2.2.rhodecode-upstream1
 nbconvert==5.3.1
-bleach==2.1.1
+bleach==2.1.2
 nbformat==4.4.0
 jupyter_client==5.0.0
 
 ## cli tools
-alembic==0.9.6
+alembic==0.9.8
 invoke==0.13.0
 bumpversion==0.5.3
 transifex-client==0.12.5
 
 ## http servers
 gevent==1.2.2
-greenlet==0.4.12
+greenlet==0.4.13
 gunicorn==19.7.1
 waitress==1.1.0
 uWSGI==2.0.15
 
 ## debug
 ipdb==0.10.3
 ipython==5.1.0
 CProfileV==1.0.7
 bottle==0.12.13
 
 ## rhodecode-tools, special case
 https://code.rhodecode.com/rhodecode-tools-ce/archive/v0.14.1.tar.gz?md5=0b9c2caad160b68889f8172ea54af7b2#egg=rhodecode-tools==0.14.1
 
 ## appenlight
-appenlight-client==0.6.22
+appenlight-client==0.6.25
 
 ## test related requirements
 -r requirements_test.txt

rhodecode/VERSION

@@ -1,1 +1,1 b''
-4.11.6
  No newline at end of file
+4.12.0
  No newline at end of file

rhodecode/__init__.py

@@ -1,63 +1,63 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 """
 
 RhodeCode, a web based repository management software
 versioning implementation: http://www.python.org/dev/peps/pep-0386/
 """
 
 import os
 import sys
 import platform
 
 VERSION = tuple(open(os.path.join(
     os.path.dirname(__file__), 'VERSION')).read().split('.'))
 
 BACKENDS = {
     'hg': 'Mercurial repository',
     'git': 'Git repository',
     'svn': 'Subversion repository',
 }
 
 CELERY_ENABLED = False
 CELERY_EAGER = False
 
 # link to config for pyramid
 CONFIG = {}
 
 # Populated with the settings dictionary from application init in
 # rhodecode.conf.environment.load_pyramid_environment
 PYRAMID_SETTINGS = {}
 
 # Linked module for extensions
 EXTENSIONS = {}
 
 __version__ = ('.'.join((str(each) for each in VERSION[:3])))
-__dbversion__ = 85  # defines current db version for migrations
+__dbversion__ = 86  # defines current db version for migrations
 __platform__ = platform.system()
 __license__ = 'AGPLv3, and Commercial License'
 __author__ = 'RhodeCode GmbH'
 __url__ = 'https://code.rhodecode.com'
 
 is_windows = __platform__ in ['Windows']
 is_unix = not is_windows
 is_test = False
 disable_error_handler = False

rhodecode/api/tests/test_get_repo_changeset.py

@@ -1,140 +1,141 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import pytest
 
 from rhodecode.api.tests.utils import build_data, api_call, assert_error
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestGetRepoChangeset(object):
     @pytest.mark.parametrize("details", ['basic', 'extended', 'full'])
     def test_get_repo_changeset(self, details, backend):
         commit = backend.repo.get_commit(commit_idx=0)
         __, params = build_data(
             self.apikey, 'get_repo_changeset',
             repoid=backend.repo_name, revision=commit.raw_id,
             details=details,
         )
         response = api_call(self.app, params)
         result = response.json['result']
         assert result['revision'] == 0
         assert result['raw_id'] == commit.raw_id
 
         if details == 'full':
             assert result['refs']['bookmarks'] == getattr(
                 commit, 'bookmarks', [])
-            assert result['refs']['branches'] == [commit.branch]
+            branches = [commit.branch] if commit.branch else []
+            assert result['refs']['branches'] == branches
             assert result['refs']['tags'] == commit.tags
 
     @pytest.mark.parametrize("details", ['basic', 'extended', 'full'])
     def test_get_repo_changeset_bad_type(self, details, backend):
         id_, params = build_data(
             self.apikey, 'get_repo_changeset',
             repoid=backend.repo_name, revision=0,
             details=details,
         )
         response = api_call(self.app, params)
         expected = 'commit_id must be a string value'
         assert_error(id_, expected, given=response.body)
 
     @pytest.mark.parametrize("details", ['basic', 'extended', 'full'])
     def test_get_repo_changesets(self, details, backend):
         limit = 2
         commit = backend.repo.get_commit(commit_idx=0)
         __, params = build_data(
             self.apikey, 'get_repo_changesets',
             repoid=backend.repo_name, start_rev=commit.raw_id, limit=limit,
             details=details,
         )
         response = api_call(self.app, params)
         result = response.json['result']
         assert result
         assert len(result) == limit
         for x in xrange(limit):
             assert result[x]['revision'] == x
 
         if details == 'full':
             for x in xrange(limit):
                 assert 'bookmarks' in result[x]['refs']
                 assert 'branches' in result[x]['refs']
                 assert 'tags' in result[x]['refs']
 
     @pytest.mark.parametrize("details", ['basic', 'extended', 'full'])
     @pytest.mark.parametrize("start_rev, expected_revision", [
         ("0", 0),
         ("10", 10),
         ("20", 20),
         ])
     @pytest.mark.backends("hg", "git")
     def test_get_repo_changesets_commit_range(
             self, details, backend, start_rev, expected_revision):
         limit = 10
         __, params = build_data(
             self.apikey, 'get_repo_changesets',
             repoid=backend.repo_name, start_rev=start_rev, limit=limit,
             details=details,
         )
         response = api_call(self.app, params)
         result = response.json['result']
         assert result
         assert len(result) == limit
         for i in xrange(limit):
             assert result[i]['revision'] == int(expected_revision) + i
 
     @pytest.mark.parametrize("details", ['basic', 'extended', 'full'])
     @pytest.mark.parametrize("start_rev, expected_revision", [
         ("0", 0),
         ("10", 9),
         ("20", 19),
         ])
     def test_get_repo_changesets_commit_range_svn(
             self, details, backend_svn, start_rev, expected_revision):
 
         # TODO: johbo: SVN showed a problem here: The parameter "start_rev"
         # in our API allows to pass in a "Commit ID" as well as a
         # "Commit Index". In the case of Subversion it is not possible to
         # distinguish these cases. As a workaround we implemented this
         # behavior which gives a preference to see it as a "Commit ID".
 
         limit = 10
         __, params = build_data(
             self.apikey, 'get_repo_changesets',
             repoid=backend_svn.repo_name, start_rev=start_rev, limit=limit,
             details=details,
         )
         response = api_call(self.app, params)
         result = response.json['result']
         assert result
         assert len(result) == limit
         for i in xrange(limit):
             assert result[i]['revision'] == int(expected_revision) + i
 
     @pytest.mark.parametrize("details", ['basic', 'extended', 'full'])
     def test_get_repo_changesets_bad_type(self, details, backend):
         id_, params = build_data(
             self.apikey, 'get_repo_changesets',
             repoid=backend.repo_name, start_rev=0, limit=2,
             details=details,
         )
         response = api_call(self.app, params)
         expected = 'commit_id must be a string value'
         assert_error(id_, expected, given=response.body)

rhodecode/api/tests/test_pull.py

@@ -1,51 +1,53 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import os
 
 import pytest
 
 from rhodecode.tests import TESTS_TMP_PATH
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_ok, assert_error)
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestPull(object):
     @pytest.mark.backends("git", "hg")
     def test_api_pull(self, backend):
         r = backend.create_repo()
         repo_name = r.repo_name
-        r.clone_uri = os.path.join(TESTS_TMP_PATH, backend.repo_name)
+        clone_uri = os.path.join(TESTS_TMP_PATH, backend.repo_name)
+        r.clone_uri = clone_uri
 
         id_, params = build_data(self.apikey, 'pull', repoid=repo_name,)
         response = api_call(self.app, params)
-
-        expected = {'msg': 'Pulled from `%s`' % (repo_name,),
+        msg = 'Pulled from url `%s` on repo `%s`' % (
+            clone_uri, repo_name)
+        expected = {'msg': msg,
                     'repository': repo_name}
         assert_ok(id_, expected, given=response.body)
 
     def test_api_pull_error(self, backend):
         id_, params = build_data(
             self.apikey, 'pull', repoid=backend.repo_name)
         response = api_call(self.app, params)
 
-        expected = 'Unable to pull changes from `%s`' % (backend.repo_name,)
+        expected = 'Unable to pull changes from `None`'
         assert_error(id_, expected, given=response.body)

rhodecode/api/tests/test_update_repo.py

@@ -1,194 +1,197 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import mock
 import pytest
 
 from rhodecode.model.repo import RepoModel
 from rhodecode.tests import TEST_USER_ADMIN_LOGIN, TEST_USER_REGULAR_LOGIN
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_error, assert_ok, crash, jsonify)
 from rhodecode.tests.fixture import Fixture
 from rhodecode.tests.plugin import http_host_stub, http_host_only_stub
 
 fixture = Fixture()
 
 UPDATE_REPO_NAME = 'api_update_me'
 
 
 class SAME_AS_UPDATES(object):
     """ Constant used for tests below """
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestApiUpdateRepo(object):
 
     @pytest.mark.parametrize("updates, expected", [
         ({'owner': TEST_USER_REGULAR_LOGIN},
          SAME_AS_UPDATES),
 
         ({'description': 'new description'},
          SAME_AS_UPDATES),
 
         ({'clone_uri': 'http://foo.com/repo'},
          SAME_AS_UPDATES),
 
         ({'clone_uri': None},
          {'clone_uri': ''}),
 
         ({'clone_uri': ''},
          {'clone_uri': ''}),
 
+        ({'push_uri': ''},
+         {'push_uri': ''}),
+
         ({'landing_rev': 'rev:tip'},
          {'landing_rev': ['rev', 'tip']}),
 
         ({'enable_statistics': True},
          SAME_AS_UPDATES),
 
         ({'enable_locking': True},
          SAME_AS_UPDATES),
 
         ({'enable_downloads': True},
          SAME_AS_UPDATES),
 
         ({'repo_name': 'new_repo_name'},
          {
             'repo_name': 'new_repo_name',
             'url': 'http://{}/new_repo_name'.format(http_host_only_stub())
          }),
 
         ({'repo_name': 'test_group_for_update/{}'.format(UPDATE_REPO_NAME),
           '_group': 'test_group_for_update'},
          {
             'repo_name': 'test_group_for_update/{}'.format(UPDATE_REPO_NAME),
             'url': 'http://{}/test_group_for_update/{}'.format(
                 http_host_only_stub(), UPDATE_REPO_NAME)
          }),
     ])
     def test_api_update_repo(self, updates, expected, backend):
         repo_name = UPDATE_REPO_NAME
         repo = fixture.create_repo(repo_name, repo_type=backend.alias)
         if updates.get('_group'):
             fixture.create_repo_group(updates['_group'])
 
         expected_api_data = repo.get_api_data(include_secrets=True)
         if expected is SAME_AS_UPDATES:
             expected_api_data.update(updates)
         else:
             expected_api_data.update(expected)
 
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo_name, **updates)
 
         with mock.patch('rhodecode.model.validation_schema.validators.url_validator'):
             response = api_call(self.app, params)
 
         if updates.get('repo_name'):
             repo_name = updates['repo_name']
 
         try:
             expected = {
                 'msg': 'updated repo ID:%s %s' % (repo.repo_id, repo_name),
                 'repository': jsonify(expected_api_data)
             }
             assert_ok(id_, expected, given=response.body)
         finally:
             fixture.destroy_repo(repo_name)
             if updates.get('_group'):
                 fixture.destroy_repo_group(updates['_group'])
 
     def test_api_update_repo_fork_of_field(self, backend):
         master_repo = backend.create_repo()
         repo = backend.create_repo()
         updates = {
             'fork_of': master_repo.repo_name,
             'fork_of_id': master_repo.repo_id
         }
         expected_api_data = repo.get_api_data(include_secrets=True)
         expected_api_data.update(updates)
 
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo.repo_name, **updates)
         response = api_call(self.app, params)
         expected = {
             'msg': 'updated repo ID:%s %s' % (repo.repo_id, repo.repo_name),
             'repository': jsonify(expected_api_data)
         }
         assert_ok(id_, expected, given=response.body)
         result = response.json['result']['repository']
         assert result['fork_of'] == master_repo.repo_name
         assert result['fork_of_id'] == master_repo.repo_id
 
     def test_api_update_repo_fork_of_not_found(self, backend):
         master_repo_name = 'fake-parent-repo'
         repo = backend.create_repo()
         updates = {
             'fork_of': master_repo_name
         }
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo.repo_name, **updates)
         response = api_call(self.app, params)
         expected = {
             'repo_fork_of': 'Fork with id `{}` does not exists'.format(
                 master_repo_name)}
         assert_error(id_, expected, given=response.body)
 
     def test_api_update_repo_with_repo_group_not_existing(self):
         repo_name = 'admin_owned'
         fake_repo_group = 'test_group_for_update'
         fixture.create_repo(repo_name)
         updates = {'repo_name': '{}/{}'.format(fake_repo_group, repo_name)}
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo_name, **updates)
         response = api_call(self.app, params)
         try:
             expected = {
                 'repo_group': 'Repository group `{}` does not exist'.format(fake_repo_group)
             }
             assert_error(id_, expected, given=response.body)
         finally:
             fixture.destroy_repo(repo_name)
 
     def test_api_update_repo_regular_user_not_allowed(self):
         repo_name = 'admin_owned'
         fixture.create_repo(repo_name)
         updates = {'active': False}
         id_, params = build_data(
             self.apikey_regular, 'update_repo', repoid=repo_name, **updates)
         response = api_call(self.app, params)
         try:
             expected = 'repository `%s` does not exist' % (repo_name,)
             assert_error(id_, expected, given=response.body)
         finally:
             fixture.destroy_repo(repo_name)
 
     @mock.patch.object(RepoModel, 'update', crash)
     def test_api_update_repo_exception_occurred(self, backend):
         repo_name = UPDATE_REPO_NAME
         fixture.create_repo(repo_name, repo_type=backend.alias)
         id_, params = build_data(
             self.apikey, 'update_repo', repoid=repo_name,
             owner=TEST_USER_ADMIN_LOGIN,)
         response = api_call(self.app, params)
         try:
             expected = 'failed to update repo `%s`' % (repo_name,)
             assert_error(id_, expected, given=response.body)
         finally:
             fixture.destroy_repo(repo_name)

rhodecode/api/tests/test_update_user_group.py

@@ -1,110 +1,125 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import mock
 import pytest
 
 from rhodecode.model.user import UserModel
 from rhodecode.model.user_group import UserGroupModel
 from rhodecode.tests import TEST_USER_ADMIN_EMAIL
 from rhodecode.api.tests.utils import (
     build_data, api_call, assert_error, assert_ok, crash, jsonify)
 
 
 @pytest.mark.usefixtures("testuser_api", "app")
 class TestUpdateUserGroup(object):
     @pytest.mark.parametrize("changing_attr, updates", [
         ('group_name', {'group_name': 'new_group_name'}),
         ('group_name', {'group_name': 'test_group_for_update'}),
         # ('owner', {'owner': TEST_USER_REGULAR_LOGIN}),
         ('owner_email', {'owner_email': TEST_USER_ADMIN_EMAIL}),
         ('active', {'active': False}),
-        ('active', {'active': True})
+        ('active', {'active': True}),
+        ('sync', {'sync': False}),
+        ('sync', {'sync': True})
     ])
     def test_api_update_user_group(self, changing_attr, updates, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         expected_api_data = user_group.get_api_data()
         expected_api_data.update(updates)
 
         id_, params = build_data(
             self.apikey, 'update_user_group', usergroupid=group_name,
             **updates)
         response = api_call(self.app, params)
 
+        # special case for sync
+        if changing_attr == 'sync' and updates['sync'] is False:
+            expected_api_data['sync'] = None
+        elif changing_attr == 'sync' and updates['sync'] is True:
+            expected_api_data['sync'] = 'manual_api'
+
         expected = {
             'msg': 'updated user group ID:%s %s' % (
                 user_group.users_group_id, user_group.users_group_name),
             'user_group': jsonify(expected_api_data)
         }
         assert_ok(id_, expected, given=response.body)
 
     @pytest.mark.parametrize("changing_attr, updates", [
         # TODO: mikhail: decide if we need to test against the commented params
         # ('group_name', {'group_name': 'new_group_name'}),
         # ('group_name', {'group_name': 'test_group_for_update'}),
         # ('owner', {'owner': TEST_USER_REGULAR_LOGIN}),
         ('owner_email', {'owner_email': TEST_USER_ADMIN_EMAIL}),
         ('active', {'active': False}),
-        ('active', {'active': True})
+        ('active', {'active': True}),
+        ('sync', {'sync': False}),
+        ('sync', {'sync': True})
     ])
     def test_api_update_user_group_regular_user(
             self, changing_attr, updates, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         expected_api_data = user_group.get_api_data()
         expected_api_data.update(updates)
 
-
         # grant permission to this user
         user = UserModel().get_by_username(self.TEST_USER_LOGIN)
 
         user_util.grant_user_permission_to_user_group(
             user_group, user, 'usergroup.admin')
         id_, params = build_data(
             self.apikey_regular, 'update_user_group',
             usergroupid=group_name, **updates)
         response = api_call(self.app, params)
+        # special case for sync
+        if changing_attr == 'sync' and updates['sync'] is False:
+            expected_api_data['sync'] = None
+        elif changing_attr == 'sync' and updates['sync'] is True:
+            expected_api_data['sync'] = 'manual_api'
+
         expected = {
             'msg': 'updated user group ID:%s %s' % (
                 user_group.users_group_id, user_group.users_group_name),
             'user_group': jsonify(expected_api_data)
         }
         assert_ok(id_, expected, given=response.body)
 
     def test_api_update_user_group_regular_user_no_permission(self, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         id_, params = build_data(
             self.apikey_regular, 'update_user_group', usergroupid=group_name)
         response = api_call(self.app, params)
 
         expected = 'user group `%s` does not exist' % (group_name)
         assert_error(id_, expected, given=response.body)
 
     @mock.patch.object(UserGroupModel, 'update', crash)
     def test_api_update_user_group_exception_occurred(self, user_util):
         user_group = user_util.create_user_group()
         group_name = user_group.users_group_name
         id_, params = build_data(
             self.apikey, 'update_user_group', usergroupid=group_name)
         response = api_call(self.app, params)
         expected = 'failed to update user group `%s`' % (group_name,)
         assert_error(id_, expected, given=response.body)

rhodecode/api/views/repo_api.py

@@ -1,2046 +1,2064 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2011-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 import time
 
 import rhodecode
 from rhodecode.api import (
     jsonrpc_method, JSONRPCError, JSONRPCForbidden, JSONRPCValidationError)
 from rhodecode.api.utils import (
     has_superadmin_permission, Optional, OAttr, get_repo_or_error,
     get_user_group_or_error, get_user_or_error, validate_repo_permissions,
     get_perm_or_error, parse_args, get_origin, build_commit_data,
     validate_set_owner_permissions)
 from rhodecode.lib import audit_logger
 from rhodecode.lib import repo_maintenance
 from rhodecode.lib.auth import HasPermissionAnyApi, HasUserGroupPermissionAnyApi
 from rhodecode.lib.celerylib.utils import get_task_id
 from rhodecode.lib.utils2 import str2bool, time_to_datetime
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.exceptions import StatusChangeOnClosedPullRequestError
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import CommentsModel
 from rhodecode.model.db import (
     Session, ChangesetStatus, RepositoryField, Repository, RepoGroup,
     ChangesetComment)
 from rhodecode.model.repo import RepoModel
 from rhodecode.model.scm import ScmModel, RepoList
 from rhodecode.model.settings import SettingsModel, VcsSettingsModel
 from rhodecode.model import validation_schema
 from rhodecode.model.validation_schema.schemas import repo_schema
 
 log = logging.getLogger(__name__)
 
 
 @jsonrpc_method()
 def get_repo(request, apiuser, repoid, cache=Optional(True)):
     """
     Gets an existing repository by its name or repository_id.
 
     The members section so the output returns users groups or users
     associated with that repository.
 
     This command can only be run using an |authtoken| with admin rights,
     or users with at least read rights to the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository id.
     :type repoid: str or int
     :param cache: use the cached value for last changeset
     :type: cache: Optional(bool)
 
     Example output:
 
     .. code-block:: bash
 
         {
           "error": null,
           "id": <repo_id>,
           "result": {
             "clone_uri": null,
             "created_on": "timestamp",
             "description": "repo description",
             "enable_downloads": false,
             "enable_locking": false,
             "enable_statistics": false,
             "followers": [
               {
                 "active": true,
                 "admin": false,
                 "api_key": "****************************************",
                 "api_keys": [
                   "****************************************"
                 ],
                 "email": "[e-mail]",
                 "emails": [
                   "[e-mail]"
                 ],
                 "extern_name": "rhodecode",
                 "extern_type": "rhodecode",
                 "firstname": "username",
                 "ip_addresses": [],
                 "language": null,
                 "last_login": "2015-09-16T17:16:35.854",
                 "lastname": "surname",
                 "user_id": <user_id>,
                 "username": "name"
               }
             ],
             "fork_of": "parent-repo",
             "landing_rev": [
               "rev",
               "tip"
             ],
             "last_changeset": {
               "author": "User <[e-mail]>",
               "branch": "default",
               "date": "timestamp",
               "message": "last commit message",
               "parents": [
                 {
                   "raw_id": "commit-id"
                 }
               ],
               "raw_id": "commit-id",
               "revision": <revision number>,
               "short_id": "short id"
             },
             "lock_reason": null,
             "locked_by": null,
             "locked_date": null,
             "owner": "owner-name",
             "permissions": [
               {
                 "name": "super-admin-name",
                 "origin": "super-admin",
                 "permission": "repository.admin",
                 "type": "user"
               },
               {
                 "name": "owner-name",
                 "origin": "owner",
                 "permission": "repository.admin",
                 "type": "user"
               },
               {
                 "name": "user-group-name",
                 "origin": "permission",
                 "permission": "repository.write",
                 "type": "user_group"
               }
             ],
             "private": true,
             "repo_id": 676,
             "repo_name": "user-group/repo-name",
             "repo_type": "hg"
           }
         }
     """
 
     repo = get_repo_or_error(repoid)
     cache = Optional.extract(cache)
 
     include_secrets = False
     if has_superadmin_permission(apiuser):
         include_secrets = True
     else:
         # check if we have at least read permission for this repo !
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     permissions = []
     for _user in repo.permissions():
         user_data = {
             'name': _user.username,
             'permission': _user.permission,
             'origin': get_origin(_user),
             'type': "user",
         }
         permissions.append(user_data)
 
     for _user_group in repo.permission_user_groups():
         user_group_data = {
             'name': _user_group.users_group_name,
             'permission': _user_group.permission,
             'origin': get_origin(_user_group),
             'type': "user_group",
         }
         permissions.append(user_group_data)
 
     following_users = [
         user.user.get_api_data(include_secrets=include_secrets)
         for user in repo.followers]
 
     if not cache:
         repo.update_commit_cache()
     data = repo.get_api_data(include_secrets=include_secrets)
     data['permissions'] = permissions
     data['followers'] = following_users
     return data
 
 
 @jsonrpc_method()
 def get_repos(request, apiuser, root=Optional(None), traverse=Optional(True)):
     """
     Lists all existing repositories.
 
     This command can only be run using an |authtoken| with admin rights,
     or users with at least read rights to |repos|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param root: specify root repository group to fetch repositories.
         filters the returned repositories to be members of given root group.
     :type root: Optional(None)
     :param traverse: traverse given root into subrepositories. With this flag
         set to False, it will only return top-level repositories from `root`.
         if root is empty it will return just top-level repositories.
     :type traverse: Optional(True)
 
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: [
                   {
                     "repo_id" :          "<repo_id>",
                     "repo_name" :        "<reponame>"
                     "repo_type" :        "<repo_type>",
                     "clone_uri" :        "<clone_uri>",
                     "private": :         "<bool>",
                     "created_on" :       "<datetimecreated>",
                     "description" :      "<description>",
                     "landing_rev":       "<landing_rev>",
                     "owner":             "<repo_owner>",
                     "fork_of":           "<name_of_fork_parent>",
                     "enable_downloads":  "<bool>",
                     "enable_locking":    "<bool>",
                     "enable_statistics": "<bool>",
                   },
                   ...
                 ]
         error:  null
     """
 
     include_secrets = has_superadmin_permission(apiuser)
     _perms = ('repository.read', 'repository.write', 'repository.admin',)
     extras = {'user': apiuser}
 
     root = Optional.extract(root)
     traverse = Optional.extract(traverse, binary=True)
 
     if root:
         # verify parent existance, if it's empty return an error
         parent = RepoGroup.get_by_group_name(root)
         if not parent:
             raise JSONRPCError(
                 'Root repository group `{}` does not exist'.format(root))
 
         if traverse:
             repos = RepoModel().get_repos_for_root(root=root, traverse=traverse)
         else:
             repos = RepoModel().get_repos_for_root(root=parent)
     else:
         if traverse:
             repos = RepoModel().get_all()
         else:
             # return just top-level
             repos = RepoModel().get_repos_for_root(root=None)
 
     repo_list = RepoList(repos, perm_set=_perms, extra_kwargs=extras)
     return [repo.get_api_data(include_secrets=include_secrets)
             for repo in repo_list]
 
 
 @jsonrpc_method()
 def get_repo_changeset(request, apiuser, repoid, revision,
                        details=Optional('basic')):
     """
     Returns information about a changeset.
 
     Additionally parameters define the amount of details returned by
     this function.
 
     This command can only be run using an |authtoken| with admin rights,
     or users with at least read rights to the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository id
     :type repoid: str or int
     :param revision: revision for which listing should be done
     :type revision: str
     :param details: details can be 'basic|extended|full' full gives diff
         info details like the diff itself, and number of changed files etc.
     :type details: Optional(str)
 
     """
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     changes_details = Optional.extract(details)
     _changes_details_types = ['basic', 'extended', 'full']
     if changes_details not in _changes_details_types:
         raise JSONRPCError(
             'ret_type must be one of %s' % (
                 ','.join(_changes_details_types)))
 
     pre_load = ['author', 'branch', 'date', 'message', 'parents',
                 'status', '_commit', '_file_paths']
 
     try:
         cs = repo.get_commit(commit_id=revision, pre_load=pre_load)
     except TypeError as e:
         raise JSONRPCError(e.message)
     _cs_json = cs.__json__()
     _cs_json['diff'] = build_commit_data(cs, changes_details)
     if changes_details == 'full':
         _cs_json['refs'] = cs._get_refs()
     return _cs_json
 
 
 @jsonrpc_method()
 def get_repo_changesets(request, apiuser, repoid, start_rev, limit,
                         details=Optional('basic')):
     """
     Returns a set of commits limited by the number starting
     from the `start_rev` option.
 
     Additional parameters define the amount of details returned by this
     function.
 
     This command can only be run using an |authtoken| with admin rights,
     or users with at least read rights to |repos|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
     :param start_rev: The starting revision from where to get changesets.
     :type start_rev: str
     :param limit: Limit the number of commits to this amount
     :type limit: str or int
     :param details: Set the level of detail returned. Valid option are:
         ``basic``, ``extended`` and ``full``.
     :type details: Optional(str)
 
     .. note::
 
        Setting the parameter `details` to the value ``full`` is extensive
        and returns details like the diff itself, and the number
        of changed files.
 
     """
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     changes_details = Optional.extract(details)
     _changes_details_types = ['basic', 'extended', 'full']
     if changes_details not in _changes_details_types:
         raise JSONRPCError(
             'ret_type must be one of %s' % (
                 ','.join(_changes_details_types)))
 
     limit = int(limit)
     pre_load = ['author', 'branch', 'date', 'message', 'parents',
                 'status', '_commit', '_file_paths']
 
     vcs_repo = repo.scm_instance()
     # SVN needs a special case to distinguish its index and commit id
     if vcs_repo and vcs_repo.alias == 'svn' and (start_rev == '0'):
         start_rev = vcs_repo.commit_ids[0]
 
     try:
         commits = vcs_repo.get_commits(
             start_id=start_rev, pre_load=pre_load)
     except TypeError as e:
         raise JSONRPCError(e.message)
     except Exception:
         log.exception('Fetching of commits failed')
         raise JSONRPCError('Error occurred during commit fetching')
 
     ret = []
     for cnt, commit in enumerate(commits):
         if cnt >= limit != -1:
             break
         _cs_json = commit.__json__()
         _cs_json['diff'] = build_commit_data(commit, changes_details)
         if changes_details == 'full':
             _cs_json['refs'] = {
                 'branches': [commit.branch],
                 'bookmarks': getattr(commit, 'bookmarks', []),
                 'tags': commit.tags
             }
         ret.append(_cs_json)
     return ret
 
 
 @jsonrpc_method()
 def get_repo_nodes(request, apiuser, repoid, revision, root_path,
                    ret_type=Optional('all'), details=Optional('basic'),
                    max_file_bytes=Optional(None)):
     """
     Returns a list of nodes and children in a flat list for a given
     path at given revision.
 
     It's possible to specify ret_type to show only `files` or `dirs`.
 
     This command can only be run using an |authtoken| with admin rights,
     or users with at least read rights to |repos|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
     :param revision: The revision for which listing should be done.
     :type revision: str
     :param root_path: The path from which to start displaying.
     :type root_path: str
     :param ret_type: Set the return type. Valid options are
         ``all`` (default), ``files`` and ``dirs``.
     :type ret_type: Optional(str)
     :param details: Returns extended information about nodes, such as
         md5, binary, and or content.  The valid options are ``basic`` and
         ``full``.
     :type details: Optional(str)
     :param max_file_bytes: Only return file content under this file size bytes
     :type details: Optional(int)
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: [
                   {
                     "name" : "<name>"
                     "type" : "<type>",
                     "binary": "<true|false>" (only in extended mode)
                     "md5"  : "<md5 of file content>" (only in extended mode)
                   },
                   ...
                 ]
         error:  null
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = (
             'repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     ret_type = Optional.extract(ret_type)
     details = Optional.extract(details)
     _extended_types = ['basic', 'full']
     if details not in _extended_types:
         raise JSONRPCError(
             'ret_type must be one of %s' % (','.join(_extended_types)))
     extended_info = False
     content = False
     if details == 'basic':
         extended_info = True
 
     if details == 'full':
         extended_info = content = True
 
     _map = {}
     try:
         # check if repo is not empty by any chance, skip quicker if it is.
         _scm = repo.scm_instance()
         if _scm.is_empty():
             return []
 
         _d, _f = ScmModel().get_nodes(
             repo, revision, root_path, flat=False,
             extended_info=extended_info, content=content,
             max_file_bytes=max_file_bytes)
         _map = {
             'all': _d + _f,
             'files': _f,
             'dirs': _d,
         }
         return _map[ret_type]
     except KeyError:
         raise JSONRPCError(
             'ret_type must be one of %s' % (','.join(sorted(_map.keys()))))
     except Exception:
         log.exception("Exception occurred while trying to get repo nodes")
         raise JSONRPCError(
             'failed to get repo: `%s` nodes' % repo.repo_name
         )
 
 
 @jsonrpc_method()
 def get_repo_refs(request, apiuser, repoid):
     """
     Returns a dictionary of current references. It returns
     bookmarks, branches, closed_branches, and tags for given repository
 
     It's possible to specify ret_type to show only `files` or `dirs`.
 
     This command can only be run using an |authtoken| with admin rights,
     or users with at least read rights to |repos|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         "result": {
             "bookmarks": {
               "dev": "5611d30200f4040ba2ab4f3d64e5b06408a02188",
               "master": "367f590445081d8ec8c2ea0456e73ae1f1c3d6cf"
             },
             "branches": {
               "default": "5611d30200f4040ba2ab4f3d64e5b06408a02188",
               "stable": "367f590445081d8ec8c2ea0456e73ae1f1c3d6cf"
             },
             "branches_closed": {},
             "tags": {
               "tip": "5611d30200f4040ba2ab4f3d64e5b06408a02188",
               "v4.4.0": "1232313f9e6adac5ce5399c2a891dc1e72b79022",
               "v4.4.1": "cbb9f1d329ae5768379cdec55a62ebdd546c4e27",
               "v4.4.2": "24ffe44a27fcd1c5b6936144e176b9f6dd2f3a17",
             }
         }
         error:  null
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin', 'repository.write', 'repository.read',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         # check if repo is not empty by any chance, skip quicker if it is.
         vcs_instance = repo.scm_instance()
         refs = vcs_instance.refs()
         return refs
     except Exception:
         log.exception("Exception occurred while trying to get repo refs")
         raise JSONRPCError(
             'failed to get repo: `%s` references' % repo.repo_name
         )
 
 
 @jsonrpc_method()
 def create_repo(
         request, apiuser, repo_name, repo_type,
         owner=Optional(OAttr('apiuser')),
         description=Optional(''),
         private=Optional(False),
         clone_uri=Optional(None),
+        push_uri=Optional(None),
         landing_rev=Optional('rev:tip'),
         enable_statistics=Optional(False),
         enable_locking=Optional(False),
         enable_downloads=Optional(False),
         copy_permissions=Optional(False)):
     """
     Creates a repository.
 
     * If the repository name contains "/", repository will be created inside
       a repository group or nested repository groups
 
       For example "foo/bar/repo1" will create |repo| called "repo1" inside
       group "foo/bar". You have to have permissions to access and write to
       the last repository group ("bar" in this example)
 
     This command can only be run using an |authtoken| with at least
     permissions to create repositories, or write permissions to
     parent repository groups.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repo_name: Set the repository name.
     :type repo_name: str
     :param repo_type: Set the repository type; 'hg','git', or 'svn'.
     :type repo_type: str
     :param owner: user_id or username
     :type owner: Optional(str)
     :param description: Set the repository description.
     :type description: Optional(str)
     :param private: set repository as private
     :type private: bool
     :param clone_uri: set clone_uri
     :type clone_uri: str
+    :param push_uri: set push_uri
+    :type push_uri: str
     :param landing_rev: <rev_type>:<rev>
     :type landing_rev: str
     :param enable_locking:
     :type enable_locking: bool
     :param enable_downloads:
     :type enable_downloads: bool
     :param enable_statistics:
     :type enable_statistics: bool
     :param copy_permissions: Copy permission from group in which the
         repository is being created.
     :type copy_permissions: bool
 
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "msg": "Created new repository `<reponame>`",
                   "success": true,
                   "task": "<celery task id or None if done sync>"
                 }
         error:  null
 
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
          'failed to create repository `<repo_name>`'
       }
 
     """
 
     owner = validate_set_owner_permissions(apiuser, owner)
 
     description = Optional.extract(description)
     copy_permissions = Optional.extract(copy_permissions)
     clone_uri = Optional.extract(clone_uri)
+    push_uri = Optional.extract(push_uri)
     landing_commit_ref = Optional.extract(landing_rev)
 
     defs = SettingsModel().get_default_repo_settings(strip_prefix=True)
     if isinstance(private, Optional):
         private = defs.get('repo_private') or Optional.extract(private)
     if isinstance(repo_type, Optional):
         repo_type = defs.get('repo_type')
     if isinstance(enable_statistics, Optional):
         enable_statistics = defs.get('repo_enable_statistics')
     if isinstance(enable_locking, Optional):
         enable_locking = defs.get('repo_enable_locking')
     if isinstance(enable_downloads, Optional):
         enable_downloads = defs.get('repo_enable_downloads')
 
     schema = repo_schema.RepoSchema().bind(
         repo_type_options=rhodecode.BACKENDS.keys(),
         repo_type=repo_type,
         # user caller
         user=apiuser)
 
     try:
         schema_data = schema.deserialize(dict(
             repo_name=repo_name,
             repo_type=repo_type,
             repo_owner=owner.username,
             repo_description=description,
             repo_landing_commit_ref=landing_commit_ref,
             repo_clone_uri=clone_uri,
+            repo_push_uri=push_uri,
             repo_private=private,
             repo_copy_permissions=copy_permissions,
             repo_enable_statistics=enable_statistics,
             repo_enable_downloads=enable_downloads,
             repo_enable_locking=enable_locking))
     except validation_schema.Invalid as err:
         raise JSONRPCValidationError(colander_exc=err)
 
     try:
         data = {
             'owner': owner,
             'repo_name': schema_data['repo_group']['repo_name_without_group'],
             'repo_name_full': schema_data['repo_name'],
             'repo_group': schema_data['repo_group']['repo_group_id'],
             'repo_type': schema_data['repo_type'],
             'repo_description': schema_data['repo_description'],
             'repo_private': schema_data['repo_private'],
             'clone_uri': schema_data['repo_clone_uri'],
+            'push_uri': schema_data['repo_push_uri'],
             'repo_landing_rev': schema_data['repo_landing_commit_ref'],
             'enable_statistics': schema_data['repo_enable_statistics'],
             'enable_locking': schema_data['repo_enable_locking'],
             'enable_downloads': schema_data['repo_enable_downloads'],
             'repo_copy_permissions': schema_data['repo_copy_permissions'],
         }
 
-        task = RepoModel().create(form_data=data, cur_user=owner)
+        task = RepoModel().create(form_data=data, cur_user=owner.user_id)
         task_id = get_task_id(task)
         # no commit, it's done in RepoModel, or async via celery
         return {
             'msg': "Created new repository `%s`" % (schema_data['repo_name'],),
             'success': True,  # cannot return the repo data here since fork
             # can be done async
             'task': task_id
         }
     except Exception:
         log.exception(
             u"Exception while trying to create the repository %s",
             schema_data['repo_name'])
         raise JSONRPCError(
             'failed to create repository `%s`' % (schema_data['repo_name'],))
 
 
 @jsonrpc_method()
 def add_field_to_repo(request, apiuser, repoid, key, label=Optional(''),
                       description=Optional('')):
     """
     Adds an extra field to a repository.
 
     This command can only be run using an |authtoken| with at least
     write permissions to the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository id.
     :type repoid: str or int
     :param key: Create a unique field key for this repository.
     :type key: str
     :param label:
     :type label: Optional(str)
     :param description:
     :type description: Optional(str)
     """
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     label = Optional.extract(label) or key
     description = Optional.extract(description)
 
     field = RepositoryField.get_by_key_name(key, repo)
     if field:
         raise JSONRPCError('Field with key '
                            '`%s` exists for repo `%s`' % (key, repoid))
 
     try:
         RepoModel().add_repo_field(repo, key, field_label=label,
                                    field_desc=description)
         Session().commit()
         return {
             'msg': "Added new repository field `%s`" % (key,),
             'success': True,
         }
     except Exception:
         log.exception("Exception occurred while trying to add field to repo")
         raise JSONRPCError(
             'failed to create new field for repository `%s`' % (repoid,))
 
 
 @jsonrpc_method()
 def remove_field_from_repo(request, apiuser, repoid, key):
     """
     Removes an extra field from a repository.
 
     This command can only be run using an |authtoken| with at least
     write permissions to the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository ID.
     :type repoid: str or int
     :param key: Set the unique field key for this repository.
     :type key: str
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     field = RepositoryField.get_by_key_name(key, repo)
     if not field:
         raise JSONRPCError('Field with key `%s` does not '
                            'exists for repo `%s`' % (key, repoid))
 
     try:
         RepoModel().delete_repo_field(repo, field_key=key)
         Session().commit()
         return {
             'msg': "Deleted repository field `%s`" % (key,),
             'success': True,
         }
     except Exception:
         log.exception(
             "Exception occurred while trying to delete field from repo")
         raise JSONRPCError(
             'failed to delete field for repository `%s`' % (repoid,))
 
 
 @jsonrpc_method()
 def update_repo(
         request, apiuser, repoid, repo_name=Optional(None),
         owner=Optional(OAttr('apiuser')), description=Optional(''),
-        private=Optional(False), clone_uri=Optional(None),
+        private=Optional(False),
+        clone_uri=Optional(None), push_uri=Optional(None),
         landing_rev=Optional('rev:tip'), fork_of=Optional(None),
         enable_statistics=Optional(False),
         enable_locking=Optional(False),
         enable_downloads=Optional(False), fields=Optional('')):
     """
     Updates a repository with the given information.
 
     This command can only be run using an |authtoken| with at least
     admin permissions to the |repo|.
 
     * If the repository name contains "/", repository will be updated
       accordingly with a repository group or nested repository groups
 
       For example repoid=repo-test name="foo/bar/repo-test" will update |repo|
       called "repo-test" and place it inside group "foo/bar".
       You have to have permissions to access and write to the last repository
       group ("bar" in this example)
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: repository name or repository ID.
     :type repoid: str or int
     :param repo_name: Update the |repo| name, including the
         repository group it's in.
     :type repo_name: str
     :param owner: Set the |repo| owner.
     :type owner: str
     :param fork_of: Set the |repo| as fork of another |repo|.
     :type fork_of: str
     :param description: Update the |repo| description.
     :type description: str
     :param private: Set the |repo| as private. (True | False)
     :type private: bool
     :param clone_uri: Update the |repo| clone URI.
     :type clone_uri: str
     :param landing_rev: Set the |repo| landing revision. Default is ``rev:tip``.
     :type landing_rev: str
     :param enable_statistics: Enable statistics on the |repo|, (True | False).
     :type enable_statistics: bool
     :param enable_locking: Enable |repo| locking.
     :type enable_locking: bool
     :param enable_downloads: Enable downloads from the |repo|, (True | False).
     :type enable_downloads: bool
     :param fields: Add extra fields to the |repo|. Use the following
         example format: ``field_key=field_val,field_key2=fieldval2``.
         Escape ', ' with \,
     :type fields: str
     """
 
     repo = get_repo_or_error(repoid)
 
     include_secrets = False
     if not has_superadmin_permission(apiuser):
         validate_repo_permissions(apiuser, repoid, repo, ('repository.admin',))
     else:
         include_secrets = True
 
     updates = dict(
         repo_name=repo_name
         if not isinstance(repo_name, Optional) else repo.repo_name,
 
         fork_id=fork_of
         if not isinstance(fork_of, Optional) else repo.fork.repo_name if repo.fork else None,
 
         user=owner
         if not isinstance(owner, Optional) else repo.user.username,
 
         repo_description=description
         if not isinstance(description, Optional) else repo.description,
 
         repo_private=private
         if not isinstance(private, Optional) else repo.private,
 
         clone_uri=clone_uri
         if not isinstance(clone_uri, Optional) else repo.clone_uri,
 
+        push_uri=push_uri
+        if not isinstance(push_uri, Optional) else repo.push_uri,
+
         repo_landing_rev=landing_rev
         if not isinstance(landing_rev, Optional) else repo._landing_revision,
 
         repo_enable_statistics=enable_statistics
         if not isinstance(enable_statistics, Optional) else repo.enable_statistics,
 
         repo_enable_locking=enable_locking
         if not isinstance(enable_locking, Optional) else repo.enable_locking,
 
         repo_enable_downloads=enable_downloads
         if not isinstance(enable_downloads, Optional) else repo.enable_downloads)
 
     ref_choices, _labels = ScmModel().get_repo_landing_revs(
         request.translate, repo=repo)
 
     old_values = repo.get_api_data()
     repo_type = repo.repo_type
     schema = repo_schema.RepoSchema().bind(
         repo_type_options=rhodecode.BACKENDS.keys(),
         repo_ref_options=ref_choices,
         repo_type=repo_type,
         # user caller
         user=apiuser,
         old_values=old_values)
     try:
         schema_data = schema.deserialize(dict(
             # we save old value, users cannot change type
             repo_type=repo_type,
 
             repo_name=updates['repo_name'],
             repo_owner=updates['user'],
             repo_description=updates['repo_description'],
             repo_clone_uri=updates['clone_uri'],
+            repo_push_uri=updates['push_uri'],
             repo_fork_of=updates['fork_id'],
             repo_private=updates['repo_private'],
             repo_landing_commit_ref=updates['repo_landing_rev'],
             repo_enable_statistics=updates['repo_enable_statistics'],
             repo_enable_downloads=updates['repo_enable_downloads'],
             repo_enable_locking=updates['repo_enable_locking']))
     except validation_schema.Invalid as err:
         raise JSONRPCValidationError(colander_exc=err)
 
     # save validated data back into the updates dict
     validated_updates = dict(
         repo_name=schema_data['repo_group']['repo_name_without_group'],
         repo_group=schema_data['repo_group']['repo_group_id'],
 
         user=schema_data['repo_owner'],
         repo_description=schema_data['repo_description'],
         repo_private=schema_data['repo_private'],
         clone_uri=schema_data['repo_clone_uri'],
+        push_uri=schema_data['repo_push_uri'],
         repo_landing_rev=schema_data['repo_landing_commit_ref'],
         repo_enable_statistics=schema_data['repo_enable_statistics'],
         repo_enable_locking=schema_data['repo_enable_locking'],
         repo_enable_downloads=schema_data['repo_enable_downloads'],
     )
 
     if schema_data['repo_fork_of']:
         fork_repo = get_repo_or_error(schema_data['repo_fork_of'])
         validated_updates['fork_id'] = fork_repo.repo_id
 
     # extra fields
     fields = parse_args(Optional.extract(fields), key_prefix='ex_')
     if fields:
         validated_updates.update(fields)
 
     try:
         RepoModel().update(repo, **validated_updates)
         audit_logger.store_api(
             'repo.edit', action_data={'old_data': old_values},
             user=apiuser, repo=repo)
         Session().commit()
         return {
             'msg': 'updated repo ID:%s %s' % (repo.repo_id, repo.repo_name),
             'repository': repo.get_api_data(include_secrets=include_secrets)
         }
     except Exception:
         log.exception(
             u"Exception while trying to update the repository %s",
             repoid)
         raise JSONRPCError('failed to update repo `%s`' % repoid)
 
 
 @jsonrpc_method()
 def fork_repo(request, apiuser, repoid, fork_name,
               owner=Optional(OAttr('apiuser')),
               description=Optional(''),
               private=Optional(False),
               clone_uri=Optional(None),
               landing_rev=Optional('rev:tip'),
               copy_permissions=Optional(False)):
     """
     Creates a fork of the specified |repo|.
 
     * If the fork_name contains "/", fork will be created inside
       a repository group or nested repository groups
 
       For example "foo/bar/fork-repo" will create fork called "fork-repo"
       inside group "foo/bar". You have to have permissions to access and
       write to the last repository group ("bar" in this example)
 
     This command can only be run using an |authtoken| with minimum
     read permissions of the forked repo, create fork permissions for an user.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set repository name or repository ID.
     :type repoid: str or int
     :param fork_name: Set the fork name, including it's repository group membership.
     :type fork_name: str
     :param owner: Set the fork owner.
     :type owner: str
     :param description: Set the fork description.
     :type description: str
     :param copy_permissions: Copy permissions from parent |repo|. The
         default is False.
     :type copy_permissions: bool
     :param private: Make the fork private. The default is False.
     :type private: bool
     :param landing_rev: Set the landing revision. The default is tip.
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_for_response>
         api_key : "<api_key>"
         args:     {
                     "repoid" :          "<reponame or repo_id>",
                     "fork_name":        "<forkname>",
                     "owner":            "<username or user_id = Optional(=apiuser)>",
                     "description":      "<description>",
                     "copy_permissions": "<bool>",
                     "private":          "<bool>",
                     "landing_rev":      "<landing_rev>"
                   }
 
     Example error output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "msg": "Created fork of `<reponame>` as `<forkname>`",
                   "success": true,
                   "task": "<celery task id or None if done sync>"
                 }
         error:  null
 
     """
 
     repo = get_repo_or_error(repoid)
     repo_name = repo.repo_name
 
     if not has_superadmin_permission(apiuser):
         # check if we have at least read permission for
         # this repo that we fork !
         _perms = (
             'repository.admin', 'repository.write', 'repository.read')
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
         # check if the regular user has at least fork permissions as well
         if not HasPermissionAnyApi('hg.fork.repository')(user=apiuser):
             raise JSONRPCForbidden()
 
     # check if user can set owner parameter
     owner = validate_set_owner_permissions(apiuser, owner)
 
     description = Optional.extract(description)
     copy_permissions = Optional.extract(copy_permissions)
     clone_uri = Optional.extract(clone_uri)
     landing_commit_ref = Optional.extract(landing_rev)
     private = Optional.extract(private)
 
     schema = repo_schema.RepoSchema().bind(
         repo_type_options=rhodecode.BACKENDS.keys(),
         repo_type=repo.repo_type,
         # user caller
         user=apiuser)
 
     try:
         schema_data = schema.deserialize(dict(
             repo_name=fork_name,
             repo_type=repo.repo_type,
             repo_owner=owner.username,
             repo_description=description,
             repo_landing_commit_ref=landing_commit_ref,
             repo_clone_uri=clone_uri,
             repo_private=private,
             repo_copy_permissions=copy_permissions))
     except validation_schema.Invalid as err:
         raise JSONRPCValidationError(colander_exc=err)
 
     try:
         data = {
             'fork_parent_id': repo.repo_id,
 
             'repo_name': schema_data['repo_group']['repo_name_without_group'],
             'repo_name_full': schema_data['repo_name'],
             'repo_group': schema_data['repo_group']['repo_group_id'],
             'repo_type': schema_data['repo_type'],
             'description': schema_data['repo_description'],
             'private': schema_data['repo_private'],
             'copy_permissions': schema_data['repo_copy_permissions'],
             'landing_rev': schema_data['repo_landing_commit_ref'],
         }
 
-        task = RepoModel().create_fork(data, cur_user=owner)
+        task = RepoModel().create_fork(data, cur_user=owner.user_id)
         # no commit, it's done in RepoModel, or async via celery
         task_id = get_task_id(task)
 
         return {
             'msg': 'Created fork of `%s` as `%s`' % (
                 repo.repo_name, schema_data['repo_name']),
             'success': True,  # cannot return the repo data here since fork
             # can be done async
             'task': task_id
         }
     except Exception:
         log.exception(
             u"Exception while trying to create fork %s",
             schema_data['repo_name'])
         raise JSONRPCError(
             'failed to fork repository `%s` as `%s`' % (
                 repo_name, schema_data['repo_name']))
 
 
 @jsonrpc_method()
 def delete_repo(request, apiuser, repoid, forks=Optional('')):
     """
     Deletes a repository.
 
     * When the `forks` parameter is set it's possible to detach or delete
       forks of deleted repository.
 
     This command can only be run using an |authtoken| with admin
     permissions on the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository ID.
     :type repoid: str or int
     :param forks: Set to `detach` or `delete` forks from the |repo|.
     :type forks: Optional(str)
 
     Example error output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "msg": "Deleted repository `<reponame>`",
                   "success": true
                 }
         error:  null
     """
 
     repo = get_repo_or_error(repoid)
     repo_name = repo.repo_name
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         handle_forks = Optional.extract(forks)
         _forks_msg = ''
         _forks = [f for f in repo.forks]
         if handle_forks == 'detach':
             _forks_msg = ' ' + 'Detached %s forks' % len(_forks)
         elif handle_forks == 'delete':
             _forks_msg = ' ' + 'Deleted %s forks' % len(_forks)
         elif _forks:
             raise JSONRPCError(
                 'Cannot delete `%s` it still contains attached forks' %
                 (repo.repo_name,)
             )
         old_data = repo.get_api_data()
         RepoModel().delete(repo, forks=forks)
 
         repo = audit_logger.RepoWrap(repo_id=None,
                                      repo_name=repo.repo_name)
 
         audit_logger.store_api(
             'repo.delete', action_data={'old_data': old_data},
             user=apiuser, repo=repo)
 
         ScmModel().mark_for_invalidation(repo_name, delete=True)
         Session().commit()
         return {
             'msg': 'Deleted repository `%s`%s' % (repo_name, _forks_msg),
             'success': True
         }
     except Exception:
         log.exception("Exception occurred while trying to delete repo")
         raise JSONRPCError(
             'failed to delete repository `%s`' % (repo_name,)
         )
 
 
 #TODO: marcink, change name ?
 @jsonrpc_method()
 def invalidate_cache(request, apiuser, repoid, delete_keys=Optional(False)):
     """
     Invalidates the cache for the specified repository.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Sets the repository name or repository ID.
     :type repoid: str or int
     :param delete_keys: This deletes the invalidated keys instead of
         just flagging them.
     :type delete_keys: Optional(``True`` | ``False``)
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         'msg': Cache for repository `<repository name>` was invalidated,
         'repository': <repository name>
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error : {
          'Error occurred during cache invalidation action'
       }
 
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin', 'repository.write',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     delete = Optional.extract(delete_keys)
     try:
         ScmModel().mark_for_invalidation(repo.repo_name, delete=delete)
         return {
             'msg': 'Cache for repository `%s` was invalidated' % (repoid,),
             'repository': repo.repo_name
         }
     except Exception:
         log.exception(
             "Exception occurred while trying to invalidate repo cache")
         raise JSONRPCError(
             'Error occurred during cache invalidation action'
         )
 
 
 #TODO: marcink, change name ?
 @jsonrpc_method()
 def lock(request, apiuser, repoid, locked=Optional(None),
          userid=Optional(OAttr('apiuser'))):
     """
     Sets the lock state of the specified |repo| by the given user.
     From more information, see :ref:`repo-locking`.
 
     * If the ``userid`` option is not set, the repository is locked to the
       user who called the method.
     * If the ``locked`` parameter is not set, the current lock state of the
       repository is displayed.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Sets the repository name or repository ID.
     :type repoid: str or int
     :param locked: Sets the lock state.
     :type locked: Optional(``True`` | ``False``)
     :param userid: Set the repository lock to this user.
     :type userid: Optional(str or int)
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         'repo': '<reponame>',
         'locked': <bool: lock state>,
         'locked_since': <int: lock timestamp>,
         'locked_by': <username of person who made the lock>,
         'lock_reason': <str: reason for locking>,
         'lock_state_changed': <bool: True if lock state has been changed in this request>,
         'msg': 'Repo `<reponame>` locked by `<username>` on <timestamp>.'
         or
         'msg': 'Repo `<repository name>` not locked.'
         or
         'msg': 'User `<user name>` set lock state for repo `<repository name>` to `<new lock state>`'
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         'Error occurred locking repository `<reponame>`'
       }
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         # check if we have at least write permission for this repo !
         _perms = ('repository.admin', 'repository.write',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
         # make sure normal user does not pass someone else userid,
         # he is not allowed to do that
         if not isinstance(userid, Optional) and userid != apiuser.user_id:
             raise JSONRPCError('userid is not the same as your user')
 
     if isinstance(userid, Optional):
         userid = apiuser.user_id
 
     user = get_user_or_error(userid)
 
     if isinstance(locked, Optional):
         lockobj = repo.locked
 
         if lockobj[0] is None:
             _d = {
                 'repo': repo.repo_name,
                 'locked': False,
                 'locked_since': None,
                 'locked_by': None,
                 'lock_reason': None,
                 'lock_state_changed': False,
                 'msg': 'Repo `%s` not locked.' % repo.repo_name
             }
             return _d
         else:
             _user_id, _time, _reason = lockobj
             lock_user = get_user_or_error(userid)
             _d = {
                 'repo': repo.repo_name,
                 'locked': True,
                 'locked_since': _time,
                 'locked_by': lock_user.username,
                 'lock_reason': _reason,
                 'lock_state_changed': False,
                 'msg': ('Repo `%s` locked by `%s` on `%s`.'
                         % (repo.repo_name, lock_user.username,
                            json.dumps(time_to_datetime(_time))))
             }
             return _d
 
     # force locked state through a flag
     else:
         locked = str2bool(locked)
         lock_reason = Repository.LOCK_API
         try:
             if locked:
                 lock_time = time.time()
                 Repository.lock(repo, user.user_id, lock_time, lock_reason)
             else:
                 lock_time = None
                 Repository.unlock(repo)
             _d = {
                 'repo': repo.repo_name,
                 'locked': locked,
                 'locked_since': lock_time,
                 'locked_by': user.username,
                 'lock_reason': lock_reason,
                 'lock_state_changed': True,
                 'msg': ('User `%s` set lock state for repo `%s` to `%s`'
                         % (user.username, repo.repo_name, locked))
             }
             return _d
         except Exception:
             log.exception(
                 "Exception occurred while trying to lock repository")
             raise JSONRPCError(
                 'Error occurred locking repository `%s`' % repo.repo_name
             )
 
 
 @jsonrpc_method()
 def comment_commit(
         request, apiuser, repoid, commit_id, message, status=Optional(None),
         comment_type=Optional(ChangesetComment.COMMENT_TYPE_NOTE),
         resolves_comment_id=Optional(None),
         userid=Optional(OAttr('apiuser'))):
     """
     Set a commit comment, and optionally change the status of the commit.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository ID.
     :type repoid: str or int
     :param commit_id: Specify the commit_id for which to set a comment.
     :type commit_id: str
     :param message: The comment text.
     :type message: str
     :param status: (**Optional**) status of commit, one of: 'not_reviewed',
         'approved', 'rejected', 'under_review'
     :type status: str
     :param comment_type: Comment type, one of: 'note', 'todo'
     :type comment_type: Optional(str), default: 'note'
     :param userid: Set the user name of the comment creator.
     :type userid: Optional(str or int)
 
     Example error output:
 
     .. code-block:: bash
 
         {
             "id" : <id_given_in_input>,
             "result" : {
                 "msg": "Commented on commit `<commit_id>` for repository `<repoid>`",
                 "status_change": null or <status>,
                 "success": true
             },
             "error" :  null
         }
 
     """
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.read', 'repository.write', 'repository.admin')
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         commit_id = repo.scm_instance().get_commit(commit_id=commit_id).raw_id
     except Exception as e:
         log.exception('Failed to fetch commit')
         raise JSONRPCError(e.message)
 
     if isinstance(userid, Optional):
         userid = apiuser.user_id
 
     user = get_user_or_error(userid)
     status = Optional.extract(status)
     comment_type = Optional.extract(comment_type)
     resolves_comment_id = Optional.extract(resolves_comment_id)
 
     allowed_statuses = [x[0] for x in ChangesetStatus.STATUSES]
     if status and status not in allowed_statuses:
         raise JSONRPCError('Bad status, must be on '
                            'of %s got %s' % (allowed_statuses, status,))
 
     if resolves_comment_id:
         comment = ChangesetComment.get(resolves_comment_id)
         if not comment:
             raise JSONRPCError(
                 'Invalid resolves_comment_id `%s` for this commit.'
                 % resolves_comment_id)
         if comment.comment_type != ChangesetComment.COMMENT_TYPE_TODO:
             raise JSONRPCError(
                 'Comment `%s` is wrong type for setting status to resolved.'
                 % resolves_comment_id)
 
     try:
         rc_config = SettingsModel().get_all_settings()
         renderer = rc_config.get('rhodecode_markup_renderer', 'rst')
         status_change_label = ChangesetStatus.get_status_lbl(status)
         comment = CommentsModel().create(
             message, repo, user, commit_id=commit_id,
             status_change=status_change_label,
             status_change_type=status,
             renderer=renderer,
             comment_type=comment_type,
             resolves_comment_id=resolves_comment_id
         )
         if status:
             # also do a status change
             try:
                 ChangesetStatusModel().set_status(
                     repo, status, user, comment, revision=commit_id,
                     dont_allow_on_closed_pull_request=True
                 )
             except StatusChangeOnClosedPullRequestError:
                 log.exception(
                     "Exception occurred while trying to change repo commit status")
                 msg = ('Changing status on a changeset associated with '
                        'a closed pull request is not allowed')
                 raise JSONRPCError(msg)
 
         Session().commit()
         return {
             'msg': (
                 'Commented on commit `%s` for repository `%s`' % (
                     comment.revision, repo.repo_name)),
             'status_change': status,
             'success': True,
         }
     except JSONRPCError:
         # catch any inside errors, and re-raise them to prevent from
         # below global catch to silence them
         raise
     except Exception:
         log.exception("Exception occurred while trying to comment on commit")
         raise JSONRPCError(
             'failed to set comment on repository `%s`' % (repo.repo_name,)
         )
 
 
 @jsonrpc_method()
 def grant_user_permission(request, apiuser, repoid, userid, perm):
     """
     Grant permissions for the specified user on the given repository,
     or update existing permissions if found.
 
     This command can only be run using an |authtoken| with admin
     permissions on the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository ID.
     :type repoid: str or int
     :param userid: Set the user name.
     :type userid: str
     :param perm: Set the user permissions, using the following format
         ``(repository.(none|read|write|admin))``
     :type perm: str
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "msg" : "Granted perm: `<perm>` for user: `<username>` in repo: `<reponame>`",
                   "success": true
                 }
         error:  null
     """
 
     repo = get_repo_or_error(repoid)
     user = get_user_or_error(userid)
     perm = get_perm_or_error(perm)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
 
         RepoModel().grant_user_permission(repo=repo, user=user, perm=perm)
 
         Session().commit()
         return {
             'msg': 'Granted perm: `%s` for user: `%s` in repo: `%s`' % (
                 perm.permission_name, user.username, repo.repo_name
             ),
             'success': True
         }
     except Exception:
         log.exception(
             "Exception occurred while trying edit permissions for repo")
         raise JSONRPCError(
             'failed to edit permission for user: `%s` in repo: `%s`' % (
                 userid, repoid
             )
         )
 
 
 @jsonrpc_method()
 def revoke_user_permission(request, apiuser, repoid, userid):
     """
     Revoke permission for a user on the specified repository.
 
     This command can only be run using an |authtoken| with admin
     permissions on the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository ID.
     :type repoid: str or int
     :param userid: Set the user name of revoked user.
     :type userid: str or int
 
     Example error output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "msg" : "Revoked perm for user: `<username>` in repo: `<reponame>`",
                   "success": true
                 }
         error:  null
     """
 
     repo = get_repo_or_error(repoid)
     user = get_user_or_error(userid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         RepoModel().revoke_user_permission(repo=repo, user=user)
         Session().commit()
         return {
             'msg': 'Revoked perm for user: `%s` in repo: `%s`' % (
                 user.username, repo.repo_name
             ),
             'success': True
         }
     except Exception:
         log.exception(
             "Exception occurred while trying revoke permissions to repo")
         raise JSONRPCError(
             'failed to edit permission for user: `%s` in repo: `%s`' % (
                 userid, repoid
             )
         )
 
 
 @jsonrpc_method()
 def grant_user_group_permission(request, apiuser, repoid, usergroupid, perm):
     """
     Grant permission for a user group on the specified repository,
     or update existing permissions.
 
     This command can only be run using an |authtoken| with admin
     permissions on the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository ID.
     :type repoid: str or int
     :param usergroupid: Specify the ID of the user group.
     :type usergroupid: str or int
     :param perm: Set the user group permissions using the following
         format: (repository.(none|read|write|admin))
     :type perm: str
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg" : "Granted perm: `<perm>` for group: `<usersgroupname>` in repo: `<reponame>`",
         "success": true
 
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "failed to edit permission for user group: `<usergroup>` in repo `<repo>`'
       }
 
     """
 
     repo = get_repo_or_error(repoid)
     perm = get_perm_or_error(perm)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
         # check if we have at least read permission for this user group !
         _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
     try:
         RepoModel().grant_user_group_permission(
             repo=repo, group_name=user_group, perm=perm)
 
         Session().commit()
         return {
             'msg': 'Granted perm: `%s` for user group: `%s` in '
                    'repo: `%s`' % (
                        perm.permission_name, user_group.users_group_name,
                        repo.repo_name
                    ),
             'success': True
         }
     except Exception:
         log.exception(
             "Exception occurred while trying change permission on repo")
         raise JSONRPCError(
             'failed to edit permission for user group: `%s` in '
             'repo: `%s`' % (
                 usergroupid, repo.repo_name
             )
         )
 
 
 @jsonrpc_method()
 def revoke_user_group_permission(request, apiuser, repoid, usergroupid):
     """
     Revoke the permissions of a user group on a given repository.
 
     This command can only be run using an |authtoken| with admin
     permissions on the |repo|.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: Set the repository name or repository ID.
     :type repoid: str or int
     :param usergroupid: Specify the user group ID.
     :type usergroupid: str or int
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "msg" : "Revoked perm for group: `<usersgroupname>` in repo: `<reponame>`",
                   "success": true
                 }
         error:  null
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
         # check if we have at least read permission for this user group !
         _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
     try:
         RepoModel().revoke_user_group_permission(
             repo=repo, group_name=user_group)
 
         Session().commit()
         return {
             'msg': 'Revoked perm for user group: `%s` in repo: `%s`' % (
                 user_group.users_group_name, repo.repo_name
             ),
             'success': True
         }
     except Exception:
         log.exception("Exception occurred while trying revoke "
                       "user group permission on repo")
         raise JSONRPCError(
             'failed to edit permission for user group: `%s` in '
             'repo: `%s`' % (
                 user_group.users_group_name, repo.repo_name
             )
         )
 
 
 @jsonrpc_method()
-def pull(request, apiuser, repoid):
+def pull(request, apiuser, repoid, remote_uri=Optional(None)):
     """
     Triggers a pull on the given repository from a remote location. You
     can use this to keep remote repositories up-to-date.
 
     This command can only be run using an |authtoken| with admin
     rights to the specified repository. For more information,
     see :ref:`config-token-ref`.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
+    :param remote_uri: Optional remote URI to pass in for pull
+    :type remote_uri: str
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
-        "msg": "Pulled from `<repository name>`"
+        "msg": "Pulled from url `<remote_url>` on repo `<repository name>`"
         "repository": "<repository name>"
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
-        "Unable to pull changes from `<reponame>`"
+        "Unable to push changes from `<remote_url>`"
       }
 
     """
 
     repo = get_repo_or_error(repoid)
+    remote_uri = Optional.extract(remote_uri)
+    remote_uri_display = remote_uri or repo.clone_uri_hidden
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
-        ScmModel().pull_changes(repo.repo_name, apiuser.username)
+        ScmModel().pull_changes(
+            repo.repo_name, apiuser.username, remote_uri=remote_uri)
         return {
-            'msg': 'Pulled from `%s`' % repo.repo_name,
+            'msg': 'Pulled from url `%s` on repo `%s`' % (
+                remote_uri_display, repo.repo_name),
             'repository': repo.repo_name
         }
     except Exception:
         log.exception("Exception occurred while trying to "
                       "pull changes from remote location")
         raise JSONRPCError(
-            'Unable to pull changes from `%s`' % repo.repo_name
+            'Unable to pull changes from `%s`' % remote_uri_display
         )
 
 
 @jsonrpc_method()
 def strip(request, apiuser, repoid, revision, branch):
     """
     Strips the given revision from the specified repository.
 
     * This will remove the revision and all of its decendants.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
     :param revision: The revision you wish to strip.
     :type revision: str
     :param branch: The branch from which to strip the revision.
     :type branch: str
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": "'Stripped commit <commit_hash> from repo `<repository name>`'"
         "repository": "<repository name>"
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "Unable to strip commit <commit_hash> from repo `<repository name>`"
       }
 
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         ScmModel().strip(repo, revision, branch)
         audit_logger.store_api(
             'repo.commit.strip', action_data={'commit_id': revision},
             repo=repo,
             user=apiuser, commit=True)
 
         return {
             'msg': 'Stripped commit %s from repo `%s`' % (
                 revision, repo.repo_name),
             'repository': repo.repo_name
         }
     except Exception:
         log.exception("Exception while trying to strip")
         raise JSONRPCError(
             'Unable to strip commit %s from repo `%s`' % (
                 revision, repo.repo_name)
         )
 
 
 @jsonrpc_method()
 def get_repo_settings(request, apiuser, repoid, key=Optional(None)):
     """
     Returns all settings for a repository. If key is given it only returns the
     setting identified by the key or null.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository id.
     :type repoid: str or int
     :param key: Key of the setting to return.
     :type: key: Optional(str)
 
     Example output:
 
     .. code-block:: bash
 
         {
             "error": null,
             "id": 237,
             "result": {
                 "extensions_largefiles": true,
                 "extensions_evolve": true,
                 "hooks_changegroup_push_logger": true,
                 "hooks_changegroup_repo_size": false,
                 "hooks_outgoing_pull_logger": true,
                 "phases_publish": "True",
                 "rhodecode_hg_use_rebase_for_merging": true,
                 "rhodecode_pr_merge_enabled": true,
                 "rhodecode_use_outdated_comments": true
             }
         }
     """
 
     # Restrict access to this api method to admins only.
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     try:
         repo = get_repo_or_error(repoid)
         settings_model = VcsSettingsModel(repo=repo)
         settings = settings_model.get_global_settings()
         settings.update(settings_model.get_repo_settings())
 
         # If only a single setting is requested fetch it from all settings.
         key = Optional.extract(key)
         if key is not None:
             settings = settings.get(key, None)
     except Exception:
         msg = 'Failed to fetch settings for repository `{}`'.format(repoid)
         log.exception(msg)
         raise JSONRPCError(msg)
 
     return settings
 
 
 @jsonrpc_method()
 def set_repo_settings(request, apiuser, repoid, settings):
     """
     Update repository settings. Returns true on success.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository id.
     :type repoid: str or int
     :param settings: The new settings for the repository.
     :type: settings: dict
 
     Example output:
 
     .. code-block:: bash
 
         {
             "error": null,
             "id": 237,
             "result": true
         }
     """
     # Restrict access to this api method to admins only.
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     if type(settings) is not dict:
         raise JSONRPCError('Settings have to be a JSON Object.')
 
     try:
         settings_model = VcsSettingsModel(repo=repoid)
 
         # Merge global, repo and incoming settings.
         new_settings = settings_model.get_global_settings()
         new_settings.update(settings_model.get_repo_settings())
         new_settings.update(settings)
 
         # Update the settings.
         inherit_global_settings = new_settings.get(
             'inherit_global_settings', False)
         settings_model.create_or_update_repo_settings(
             new_settings, inherit_global_settings=inherit_global_settings)
         Session().commit()
     except Exception:
         msg = 'Failed to update settings for repository `{}`'.format(repoid)
         log.exception(msg)
         raise JSONRPCError(msg)
 
     # Indicate success.
     return True
 
 
 @jsonrpc_method()
 def maintenance(request, apiuser, repoid):
     """
     Triggers a maintenance on the given repository.
 
     This command can only be run using an |authtoken| with admin
     rights to the specified repository. For more information,
     see :ref:`config-token-ref`.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param repoid: The repository name or repository ID.
     :type repoid: str or int
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": "executed maintenance command",
         "executed_actions": [
            <action_message>, <action_message2>...
         ],
         "repository": "<repository name>"
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "Unable to execute maintenance on `<reponame>`"
       }
 
     """
 
     repo = get_repo_or_error(repoid)
     if not has_superadmin_permission(apiuser):
         _perms = ('repository.admin',)
         validate_repo_permissions(apiuser, repoid, repo, _perms)
 
     try:
         maintenance = repo_maintenance.RepoMaintenance()
         executed_actions = maintenance.execute(repo)
 
         return {
             'msg': 'executed maintenance command',
             'executed_actions': executed_actions,
             'repository': repo.repo_name
         }
     except Exception:
         log.exception("Exception occurred while trying to run maintenance")
         raise JSONRPCError(
             'Unable to execute maintenance on `%s`' % repo.repo_name)

rhodecode/api/views/user_group_api.py

@@ -1,829 +1,895 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2011-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 
 from rhodecode.api import (
     jsonrpc_method, JSONRPCError, JSONRPCForbidden, JSONRPCValidationError)
 from rhodecode.api.utils import (
     Optional, OAttr, store_update, has_superadmin_permission, get_origin,
     get_user_or_error, get_user_group_or_error, get_perm_or_error)
 from rhodecode.lib import audit_logger
 from rhodecode.lib.auth import HasUserGroupPermissionAnyApi, HasPermissionAnyApi
 from rhodecode.lib.exceptions import UserGroupAssignedException
 from rhodecode.model.db import Session
 from rhodecode.model.scm import UserGroupList
 from rhodecode.model.user_group import UserGroupModel
 from rhodecode.model import validation_schema
 from rhodecode.model.validation_schema.schemas import user_group_schema
 
 log = logging.getLogger(__name__)
 
 
 @jsonrpc_method()
 def get_user_group(request, apiuser, usergroupid):
     """
     Returns the data of an existing user group.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Set the user group from which to return data.
     :type usergroupid: str or int
 
     Example error output:
 
     .. code-block:: bash
 
         {
           "error": null,
           "id": <id>,
           "result": {
             "active": true,
             "group_description": "group description",
             "group_name": "group name",
             "permissions": [
               {
                 "name": "owner-name",
                 "origin": "owner",
                 "permission": "usergroup.admin",
                 "type": "user"
               },
               {
               {
                 "name": "user name",
                 "origin": "permission",
                 "permission": "usergroup.admin",
                 "type": "user"
               },
               {
                 "name": "user group name",
                 "origin": "permission",
                 "permission": "usergroup.write",
                 "type": "user_group"
               }
             ],
             "permissions_summary": {
               "repositories": {
                 "aa-root-level-repo-1": "repository.admin"
               },
               "repositories_groups": {}
             },
             "owner": "owner name",
             "users": [],
             "users_group_id": 2
           }
         }
 
     """
 
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
         # check if we have at least read permission for this user group !
         _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError('user group `%s` does not exist' % (
                 usergroupid,))
 
     permissions = []
     for _user in user_group.permissions():
         user_data = {
             'name': _user.username,
             'permission': _user.permission,
             'origin': get_origin(_user),
             'type': "user",
         }
         permissions.append(user_data)
 
     for _user_group in user_group.permission_user_groups():
         user_group_data = {
             'name': _user_group.users_group_name,
             'permission': _user_group.permission,
             'origin': get_origin(_user_group),
             'type': "user_group",
         }
         permissions.append(user_group_data)
 
     data = user_group.get_api_data()
     data["permissions"] = permissions
     data["permissions_summary"] = UserGroupModel().get_perms_summary(
         user_group.users_group_id)
     return data
 
 
 @jsonrpc_method()
 def get_user_groups(request, apiuser):
     """
     Lists all the existing user groups within RhodeCode.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
 
     Example error output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result : [<user_group_obj>,...]
         error : null
     """
 
     include_secrets = has_superadmin_permission(apiuser)
 
     result = []
     _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)
     extras = {'user': apiuser}
     for user_group in UserGroupList(UserGroupModel().get_all(),
                                     perm_set=_perms, extra_kwargs=extras):
         result.append(
             user_group.get_api_data(include_secrets=include_secrets))
     return result
 
 
 @jsonrpc_method()
 def create_user_group(
         request, apiuser, group_name, description=Optional(''),
-        owner=Optional(OAttr('apiuser')), active=Optional(True)):
+        owner=Optional(OAttr('apiuser')), active=Optional(True),
+        sync=Optional(None)):
     """
     Creates a new user group.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param group_name: Set the name of the new user group.
     :type group_name: str
     :param description: Give a description of the new user group.
     :type description: str
     :param owner: Set the owner of the new user group.
         If not set, the owner is the |authtoken| user.
     :type owner: Optional(str or int)
     :param active: Set this group as active.
     :type active: Optional(``True`` | ``False``)
+    :param sync: Set enabled or disabled the automatically sync from
+        external authentication types like ldap.
+    :type sync: Optional(``True`` | ``False``)
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "msg": "created new user group `<groupname>`",
                   "user_group": <user_group_object>
                 }
         error:  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "user group `<group name>` already exist"
         or
         "failed to create group `<group name>`"
       }
 
     """
 
     if not has_superadmin_permission(apiuser):
         if not HasPermissionAnyApi('hg.usergroup.create.true')(user=apiuser):
             raise JSONRPCForbidden()
 
     if UserGroupModel().get_by_name(group_name):
         raise JSONRPCError("user group `%s` already exist" % (group_name,))
 
     if isinstance(owner, Optional):
         owner = apiuser.user_id
 
     owner = get_user_or_error(owner)
     active = Optional.extract(active)
     description = Optional.extract(description)
+    sync = Optional.extract(sync)
+
+    # set the sync option based on group_data
+    group_data = None
+    if sync:
+        group_data = {
+            'extern_type': 'manual_api',
+            'extern_type_set_by': apiuser.username
+        }
 
     schema = user_group_schema.UserGroupSchema().bind(
         # user caller
         user=apiuser)
     try:
         schema_data = schema.deserialize(dict(
             user_group_name=group_name,
             user_group_description=description,
             user_group_owner=owner.username,
             user_group_active=active,
             ))
     except validation_schema.Invalid as err:
         raise JSONRPCValidationError(colander_exc=err)
 
     try:
         user_group = UserGroupModel().create(
             name=schema_data['user_group_name'],
             description=schema_data['user_group_description'],
             owner=owner,
-            active=schema_data['user_group_active'])
+            active=schema_data['user_group_active'], group_data=group_data)
         Session().flush()
         creation_data = user_group.get_api_data()
         audit_logger.store_api(
             'user_group.create', action_data={'data': creation_data},
             user=apiuser)
         Session().commit()
         return {
             'msg': 'created new user group `%s`' % group_name,
             'user_group': creation_data
         }
     except Exception:
         log.exception("Error occurred during creation of user group")
         raise JSONRPCError('failed to create group `%s`' % (group_name,))
 
 
 @jsonrpc_method()
 def update_user_group(request, apiuser, usergroupid, group_name=Optional(''),
                       description=Optional(''), owner=Optional(None),
-                      active=Optional(True)):
+                      active=Optional(True), sync=Optional(None)):
     """
     Updates the specified `user group` with the details provided.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Set the id of the `user group` to update.
     :type usergroupid: str or int
     :param group_name: Set the new name the `user group`
     :type group_name: str
     :param description: Give a description for the `user group`
     :type description: str
     :param owner: Set the owner of the `user group`.
     :type owner: Optional(str or int)
     :param active: Set the group as active.
     :type active: Optional(``True`` | ``False``)
+    :param sync: Set enabled or disabled the automatically sync from
+        external authentication types like ldap.
+    :type sync: Optional(``True`` | ``False``)
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": 'updated user group ID:<user group id> <user group name>',
         "user_group": <user_group_object>
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "failed to update user group `<user group name>`"
       }
 
     """
 
     user_group = get_user_group_or_error(usergroupid)
     include_secrets = False
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
     else:
         include_secrets = True
 
     if not isinstance(owner, Optional):
         owner = get_user_or_error(owner)
 
     old_data = user_group.get_api_data()
     updates = {}
     store_update(updates, group_name, 'users_group_name')
     store_update(updates, description, 'user_group_description')
     store_update(updates, owner, 'user')
     store_update(updates, active, 'users_group_active')
+
+    sync = Optional.extract(sync)
+    group_data = None
+    if sync is True:
+        group_data = {
+            'extern_type': 'manual_api',
+            'extern_type_set_by': apiuser.username
+        }
+    if sync is False:
+        group_data = user_group.group_data
+        if group_data and "extern_type" in group_data:
+            del group_data["extern_type"]
+
     try:
-        UserGroupModel().update(user_group, updates)
+        UserGroupModel().update(user_group, updates, group_data=group_data)
         audit_logger.store_api(
             'user_group.edit', action_data={'old_data': old_data},
             user=apiuser)
         Session().commit()
         return {
             'msg': 'updated user group ID:%s %s' % (
                 user_group.users_group_id, user_group.users_group_name),
             'user_group': user_group.get_api_data(
                 include_secrets=include_secrets)
         }
     except Exception:
         log.exception("Error occurred during update of user group")
         raise JSONRPCError(
             'failed to update user group `%s`' % (usergroupid,))
 
 
 @jsonrpc_method()
 def delete_user_group(request, apiuser, usergroupid):
     """
     Deletes the specified `user group`.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: filled automatically from apikey
     :type apiuser: AuthUser
     :param usergroupid:
     :type usergroupid: int
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": "deleted user group ID:<user_group_id> <user_group_name>"
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "failed to delete user group ID:<user_group_id> <user_group_name>"
         or
         "RepoGroup assigned to <repo_groups_list>"
       }
 
     """
 
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
     old_data = user_group.get_api_data()
     try:
         UserGroupModel().delete(user_group)
         audit_logger.store_api(
             'user_group.delete', action_data={'old_data': old_data},
             user=apiuser)
         Session().commit()
         return {
             'msg': 'deleted user group ID:%s %s' % (
                 user_group.users_group_id, user_group.users_group_name),
             'user_group': None
         }
     except UserGroupAssignedException as e:
         log.exception("UserGroupAssigned error")
         raise JSONRPCError(str(e))
     except Exception:
         log.exception("Error occurred during deletion of user group")
         raise JSONRPCError(
             'failed to delete user group ID:%s %s' %(
                 user_group.users_group_id, user_group.users_group_name))
 
 
 @jsonrpc_method()
 def add_user_to_user_group(request, apiuser, usergroupid, userid):
     """
     Adds a user to a `user group`. If the user already exists in the group
     this command will return false.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified user group.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Set the name of the `user group` to which a
         user will be added.
     :type usergroupid: int
     :param userid: Set the `user_id` of the user to add to the group.
     :type userid: int
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
           "success": True|False # depends on if member is in group
           "msg": "added member `<username>` to user group `<groupname>` |
                   User is already in that group"
 
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         "failed to add member to user group `<user_group_name>`"
       }
 
     """
 
     user = get_user_or_error(userid)
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError('user group `%s` does not exist' % (
                 usergroupid,))
 
     old_values = user_group.get_api_data()
     try:
         ugm = UserGroupModel().add_user_to_group(user_group, user)
         success = True if ugm is not True else False
         msg = 'added member `%s` to user group `%s`' % (
             user.username, user_group.users_group_name
         )
         msg = msg if success else 'User is already in that group'
         if success:
             user_data = user.get_api_data()
             audit_logger.store_api(
                 'user_group.edit.member.add',
                 action_data={'user': user_data, 'old_data': old_values},
                 user=apiuser)
 
         Session().commit()
 
         return {
             'success': success,
             'msg': msg
         }
     except Exception:
         log.exception("Error occurred during adding a member to user group")
         raise JSONRPCError(
             'failed to add member to user group `%s`' % (
                 user_group.users_group_name,
             )
         )
 
 
 @jsonrpc_method()
 def remove_user_from_user_group(request, apiuser, usergroupid, userid):
     """
     Removes a user from a user group.
 
     * If the specified user is not in the group, this command will return
       `false`.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified user group.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Sets the user group name.
     :type usergroupid: str or int
     :param userid: The user you wish to remove from |RCE|.
     :type userid: str or int
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result: {
                   "success":  True|False,  # depends on if member is in group
                   "msg": "removed member <username> from user group <groupname> |
                           User wasn't in group"
                 }
         error:  null
 
     """
 
     user = get_user_or_error(userid)
     user_group = get_user_group_or_error(usergroupid)
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
     old_values = user_group.get_api_data()
     try:
         success = UserGroupModel().remove_user_from_group(user_group, user)
         msg = 'removed member `%s` from user group `%s`' % (
             user.username, user_group.users_group_name
         )
         msg = msg if success else "User wasn't in group"
         if success:
             user_data = user.get_api_data()
             audit_logger.store_api(
                 'user_group.edit.member.delete',
                 action_data={'user': user_data, 'old_data': old_values},
                 user=apiuser)
 
         Session().commit()
         return {'success': success, 'msg': msg}
     except Exception:
         log.exception("Error occurred during removing an member from user group")
         raise JSONRPCError(
             'failed to remove member from user group `%s`' % (
                 user_group.users_group_name,
             )
         )
 
 
 @jsonrpc_method()
 def grant_user_permission_to_user_group(
         request, apiuser, usergroupid, userid, perm):
     """
     Set permissions for a user in a user group.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Set the user group to edit permissions on.
     :type usergroupid: str or int
     :param userid: Set the user from whom you wish to set permissions.
     :type userid: str
     :param perm: (usergroup.(none|read|write|admin))
     :type perm: str
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": "Granted perm: `<perm_name>` for user: `<username>` in user group: `<user_group_name>`",
         "success": true
       }
       error :  null
     """
 
     user_group = get_user_group_or_error(usergroupid)
 
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
     user = get_user_or_error(userid)
     perm = get_perm_or_error(perm, prefix='usergroup.')
 
     try:
-        UserGroupModel().grant_user_permission(
+        changes = UserGroupModel().grant_user_permission(
             user_group=user_group, user=user, perm=perm)
+
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'user_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
         return {
             'msg':
                 'Granted perm: `%s` for user: `%s` in user group: `%s`' % (
                     perm.permission_name, user.username,
                     user_group.users_group_name
                 ),
             'success': True
         }
     except Exception:
         log.exception("Error occurred during editing permissions "
                       "for user in user group")
         raise JSONRPCError(
             'failed to edit permission for user: '
             '`%s` in user group: `%s`' % (
                 userid, user_group.users_group_name))
 
 
 @jsonrpc_method()
 def revoke_user_permission_from_user_group(
         request, apiuser, usergroupid, userid):
     """
     Revoke a users permissions in a user group.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Set the user group from which to revoke the user
         permissions.
     :type: usergroupid: str or int
     :param userid: Set the userid of the user whose permissions will be
         revoked.
     :type userid: str
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": "Revoked perm for user: `<username>` in user group: `<user_group_name>`",
         "success": true
       }
       error :  null
     """
 
     user_group = get_user_group_or_error(usergroupid)
 
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (usergroupid,))
 
     user = get_user_or_error(userid)
 
     try:
-        UserGroupModel().revoke_user_permission(
+        changes = UserGroupModel().revoke_user_permission(
             user_group=user_group, user=user)
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'user_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
         return {
             'msg': 'Revoked perm for user: `%s` in user group: `%s`' % (
                 user.username, user_group.users_group_name
             ),
             'success': True
         }
     except Exception:
         log.exception("Error occurred during editing permissions "
                       "for user in user group")
         raise JSONRPCError(
             'failed to edit permission for user: `%s` in user group: `%s`'
             % (userid, user_group.users_group_name))
 
 
 @jsonrpc_method()
 def grant_user_group_permission_to_user_group(
         request, apiuser, usergroupid, sourceusergroupid, perm):
     """
     Give one user group permissions to another user group.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Set the user group on which to edit permissions.
     :type usergroupid: str or int
     :param sourceusergroupid: Set the source user group to which
         access/permissions will be granted.
     :type sourceusergroupid: str or int
     :param perm: (usergroup.(none|read|write|admin))
     :type perm: str
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": "Granted perm: `<perm_name>` for user group: `<source_user_group_name>` in user group: `<user_group_name>`",
         "success": true
       }
       error :  null
     """
 
     user_group = get_user_group_or_error(sourceusergroupid)
     target_user_group = get_user_group_or_error(usergroupid)
     perm = get_perm_or_error(perm, prefix='usergroup.')
 
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser,
                 user_group_name=target_user_group.users_group_name):
             raise JSONRPCError(
                 'to user group `%s` does not exist' % (usergroupid,))
 
         # check if we have at least read permission for source user group !
         _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (sourceusergroupid,))
 
     try:
-        UserGroupModel().grant_user_group_permission(
+        changes = UserGroupModel().grant_user_group_permission(
             target_user_group=target_user_group,
             user_group=user_group, perm=perm)
+
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'user_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
-
         return {
             'msg': 'Granted perm: `%s` for user group: `%s` '
                    'in user group: `%s`' % (
                        perm.permission_name, user_group.users_group_name,
                        target_user_group.users_group_name
                    ),
             'success': True
         }
     except Exception:
         log.exception("Error occurred during editing permissions "
                       "for user group in user group")
         raise JSONRPCError(
             'failed to edit permission for user group: `%s` in '
             'user group: `%s`' % (
                 sourceusergroupid, target_user_group.users_group_name
             )
         )
 
 
 @jsonrpc_method()
 def revoke_user_group_permission_from_user_group(
         request, apiuser, usergroupid, sourceusergroupid):
     """
     Revoke the permissions that one user group has to another.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param usergroupid: Set the user group on which to edit permissions.
     :type usergroupid: str or int
     :param sourceusergroupid: Set the user group from which permissions
         are revoked.
     :type sourceusergroupid: str or int
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         "msg": "Revoked perm for user group: `<user_group_name>` in user group: `<target_user_group_name>`",
         "success": true
       }
       error :  null
     """
 
     user_group = get_user_group_or_error(sourceusergroupid)
     target_user_group = get_user_group_or_error(usergroupid)
 
     if not has_superadmin_permission(apiuser):
         # check if we have admin permission for this user group !
         _perms = ('usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser,
                 user_group_name=target_user_group.users_group_name):
             raise JSONRPCError(
                 'to user group `%s` does not exist' % (usergroupid,))
 
         # check if we have at least read permission
         # for the source user group !
         _perms = ('usergroup.read', 'usergroup.write', 'usergroup.admin',)
         if not HasUserGroupPermissionAnyApi(*_perms)(
                 user=apiuser, user_group_name=user_group.users_group_name):
             raise JSONRPCError(
                 'user group `%s` does not exist' % (sourceusergroupid,))
 
     try:
-        UserGroupModel().revoke_user_group_permission(
+        changes = UserGroupModel().revoke_user_group_permission(
             target_user_group=target_user_group, user_group=user_group)
+        action_data = {
+            'added': changes['added'],
+            'updated': changes['updated'],
+            'deleted': changes['deleted'],
+        }
+        audit_logger.store_api(
+            'user_group.edit.permissions', action_data=action_data,
+            user=apiuser)
+
         Session().commit()
 
         return {
             'msg': 'Revoked perm for user group: '
                    '`%s` in user group: `%s`' % (
                        user_group.users_group_name,
                        target_user_group.users_group_name
                    ),
             'success': True
         }
     except Exception:
         log.exception("Error occurred during editing permissions "
                       "for user group in user group")
         raise JSONRPCError(
             'failed to edit permission for user group: '
             '`%s` in user group: `%s`' % (
                 sourceusergroupid, target_user_group.users_group_name
             )
         )

rhodecode/apps/_base/__init__.py

@@ -1,564 +1,641 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import time
 import logging
 import operator
 
-from pyramid.httpexceptions import HTTPFound
+from pyramid.httpexceptions import HTTPFound, HTTPForbidden
 
-from rhodecode.lib import helpers as h
+from rhodecode.lib import helpers as h, diffs
 from rhodecode.lib.utils2 import StrictAttributeDict, safe_int, datetime_to_time
 from rhodecode.lib.vcs.exceptions import RepositoryRequirementError
 from rhodecode.model import repo
 from rhodecode.model import repo_group
 from rhodecode.model import user_group
 from rhodecode.model import user
 from rhodecode.model.db import User
 from rhodecode.model.scm import ScmModel
+from rhodecode.model.settings import VcsSettingsModel
 
 log = logging.getLogger(__name__)
 
 
 ADMIN_PREFIX = '/_admin'
 STATIC_FILE_PREFIX = '/_static'
 
 URL_NAME_REQUIREMENTS = {
     # group name can have a slash in them, but they must not end with a slash
     'group_name': r'.*?[^/]',
     'repo_group_name': r'.*?[^/]',
     # repo names can have a slash in them, but they must not end with a slash
     'repo_name': r'.*?[^/]',
     # file path eats up everything at the end
     'f_path': r'.*',
     # reference types
     'source_ref_type': '(branch|book|tag|rev|\%\(source_ref_type\)s)',
     'target_ref_type': '(branch|book|tag|rev|\%\(target_ref_type\)s)',
 }
 
 
 def add_route_with_slash(config,name, pattern, **kw):
     config.add_route(name, pattern, **kw)
     if not pattern.endswith('/'):
         config.add_route(name + '_slash', pattern + '/', **kw)
 
 
 def add_route_requirements(route_path, requirements=URL_NAME_REQUIREMENTS):
     """
     Adds regex requirements to pyramid routes using a mapping dict
     e.g::
         add_route_requirements('{repo_name}/settings')
     """
     for key, regex in requirements.items():
         route_path = route_path.replace('{%s}' % key, '{%s:%s}' % (key, regex))
     return route_path
 
 
 def get_format_ref_id(repo):
     """Returns a `repo` specific reference formatter function"""
     if h.is_svn(repo):
         return _format_ref_id_svn
     else:
         return _format_ref_id
 
 
 def _format_ref_id(name, raw_id):
     """Default formatting of a given reference `name`"""
     return name
 
 
 def _format_ref_id_svn(name, raw_id):
     """Special way of formatting a reference for Subversion including path"""
     return '%s@%s' % (name, raw_id)
 
 
 class TemplateArgs(StrictAttributeDict):
     pass
 
 
 class BaseAppView(object):
 
     def __init__(self, context, request):
         self.request = request
         self.context = context
         self.session = request.session
         self._rhodecode_user = request.user  # auth user
         self._rhodecode_db_user = self._rhodecode_user.get_instance()
         self._maybe_needs_password_change(
             request.matched_route.name, self._rhodecode_db_user)
 
     def _maybe_needs_password_change(self, view_name, user_obj):
         log.debug('Checking if user %s needs password change on view %s',
                   user_obj, view_name)
         skip_user_views = [
             'logout', 'login',
             'my_account_password', 'my_account_password_update'
         ]
 
         if not user_obj:
             return
 
         if user_obj.username == User.DEFAULT_USER:
             return
 
         now = time.time()
         should_change = user_obj.user_data.get('force_password_change')
         change_after = safe_int(should_change) or 0
         if should_change and now > change_after:
             log.debug('User %s requires password change', user_obj)
             h.flash('You are required to change your password', 'warning',
                     ignore_duplicate=True)
 
             if view_name not in skip_user_views:
                 raise HTTPFound(
                     self.request.route_path('my_account_password'))
 
     def _log_creation_exception(self, e, repo_name):
         _ = self.request.translate
         reason = None
         if len(e.args) == 2:
             reason = e.args[1]
 
         if reason == 'INVALID_CERTIFICATE':
             log.exception(
                 'Exception creating a repository: invalid certificate')
             msg = (_('Error creating repository %s: invalid certificate')
                    % repo_name)
         else:
             log.exception("Exception creating a repository")
             msg = (_('Error creating repository %s')
                    % repo_name)
         return msg
 
     def _get_local_tmpl_context(self, include_app_defaults=True):
         c = TemplateArgs()
         c.auth_user = self.request.user
         # TODO(marcink): migrate the usage of c.rhodecode_user to c.auth_user
         c.rhodecode_user = self.request.user
 
         if include_app_defaults:
             from rhodecode.lib.base import attach_context_attributes
             attach_context_attributes(c, self.request, self.request.user.user_id)
 
         return c
 
     def _get_template_context(self, tmpl_args, **kwargs):
 
         local_tmpl_args = {
             'defaults': {},
             'errors': {},
             'c': tmpl_args
         }
         local_tmpl_args.update(kwargs)
         return local_tmpl_args
 
     def load_default_context(self):
         """
         example:
 
         def load_default_context(self):
             c = self._get_local_tmpl_context()
             c.custom_var = 'foobar'
 
             return c
         """
         raise NotImplementedError('Needs implementation in view class')
 
 
 class RepoAppView(BaseAppView):
 
     def __init__(self, context, request):
         super(RepoAppView, self).__init__(context, request)
         self.db_repo = request.db_repo
         self.db_repo_name = self.db_repo.repo_name
         self.db_repo_pull_requests = ScmModel().get_pull_requests(self.db_repo)
 
     def _handle_missing_requirements(self, error):
         log.error(
             'Requirements are missing for repository %s: %s',
             self.db_repo_name, error.message)
 
     def _get_local_tmpl_context(self, include_app_defaults=True):
         _ = self.request.translate
         c = super(RepoAppView, self)._get_local_tmpl_context(
             include_app_defaults=include_app_defaults)
 
         # register common vars for this type of view
         c.rhodecode_db_repo = self.db_repo
         c.repo_name = self.db_repo_name
         c.repository_pull_requests = self.db_repo_pull_requests
+        self.path_filter = PathFilter(None)
 
-        c.repository_requirements_missing = False
+        c.repository_requirements_missing = {}
         try:
             self.rhodecode_vcs_repo = self.db_repo.scm_instance()
+            if self.rhodecode_vcs_repo:
+                path_perms = self.rhodecode_vcs_repo.get_path_permissions(
+                    c.auth_user.username)
+                self.path_filter = PathFilter(path_perms)
         except RepositoryRequirementError as e:
-            c.repository_requirements_missing = True
+            c.repository_requirements_missing = {'error': str(e)}
             self._handle_missing_requirements(e)
             self.rhodecode_vcs_repo = None
 
-        if (not c.repository_requirements_missing
-            and self.rhodecode_vcs_repo is None):
+        c.path_filter = self.path_filter  # used by atom_feed_entry.mako
+
+        if self.rhodecode_vcs_repo is None:
             # unable to fetch this repo as vcs instance, report back to user
             h.flash(_(
                 "The repository `%(repo_name)s` cannot be loaded in filesystem. "
                 "Please check if it exist, or is not damaged.") %
                     {'repo_name': c.repo_name},
                     category='error', ignore_duplicate=True)
+            if c.repository_requirements_missing:
+                route = self.request.matched_route.name
+                if route.startswith(('edit_repo', 'repo_summary')):
+                    # allow summary and edit repo on missing requirements
+                    return c
+
+                raise HTTPFound(
+                    h.route_path('repo_summary', repo_name=self.db_repo_name))
+
+            else:  # redirect if we don't show missing requirements
             raise HTTPFound(h.route_path('home'))
 
         return c
 
-    def _get_f_path(self, matchdict, default=None):
+    def _get_f_path_unchecked(self, matchdict, default=None):
+        """
+        Should only be used by redirects, everything else should call _get_f_path
+        """
         f_path = matchdict.get('f_path')
         if f_path:
             # fix for multiple initial slashes that causes errors for GIT
             return f_path.lstrip('/')
 
         return default
 
+    def _get_f_path(self, matchdict, default=None):
+        f_path_match = self._get_f_path_unchecked(matchdict, default)
+        return self.path_filter.assert_path_permissions(f_path_match)
+
+    def _get_general_setting(self, target_repo, settings_key, default=False):
+        settings_model = VcsSettingsModel(repo=target_repo)
+        settings = settings_model.get_general_settings()
+        return settings.get(settings_key, default)
+
+
+class PathFilter(object):
+
+    # Expects and instance of BasePathPermissionChecker or None
+    def __init__(self, permission_checker):
+        self.permission_checker = permission_checker
+
+    def assert_path_permissions(self, path):
+        if path and self.permission_checker and not self.permission_checker.has_access(path):
+            raise HTTPForbidden()
+        return path
+
+    def filter_patchset(self, patchset):
+        if not self.permission_checker or not patchset:
+            return patchset, False
+        had_filtered = False
+        filtered_patchset = []
+        for patch in patchset:
+            filename = patch.get('filename', None)
+            if not filename or self.permission_checker.has_access(filename):
+                filtered_patchset.append(patch)
+            else:
+                had_filtered = True
+        if had_filtered:
+            if isinstance(patchset, diffs.LimitedDiffContainer):
+                filtered_patchset = diffs.LimitedDiffContainer(patchset.diff_limit, patchset.cur_diff_size, filtered_patchset)
+            return filtered_patchset, True
+        else:
+            return patchset, False
+
+    def render_patchset_filtered(self, diffset, patchset, source_ref=None, target_ref=None):
+        filtered_patchset, has_hidden_changes = self.filter_patchset(patchset)
+        result = diffset.render_patchset(filtered_patchset, source_ref=source_ref, target_ref=target_ref)
+        result.has_hidden_changes = has_hidden_changes
+        return result
+
+    def get_raw_patch(self, diff_processor):
+        if self.permission_checker is None:
+            return diff_processor.as_raw()
+        elif self.permission_checker.has_full_access:
+            return diff_processor.as_raw()
+        else:
+            return '# Repository has user-specific filters, raw patch generation is disabled.'
+
+    @property
+    def is_enabled(self):
+        return self.permission_checker is not None
+
 
 class RepoGroupAppView(BaseAppView):
     def __init__(self, context, request):
         super(RepoGroupAppView, self).__init__(context, request)
         self.db_repo_group = request.db_repo_group
         self.db_repo_group_name = self.db_repo_group.group_name
 
     def _revoke_perms_on_yourself(self, form_result):
         _updates = filter(lambda u: self._rhodecode_user.user_id == int(u[0]),
                           form_result['perm_updates'])
         _additions = filter(lambda u: self._rhodecode_user.user_id == int(u[0]),
                             form_result['perm_additions'])
         _deletions = filter(lambda u: self._rhodecode_user.user_id == int(u[0]),
                             form_result['perm_deletions'])
         admin_perm = 'group.admin'
         if _updates and _updates[0][1] != admin_perm or \
            _additions and _additions[0][1] != admin_perm or \
            _deletions and _deletions[0][1] != admin_perm:
             return True
         return False
 
 
 class UserGroupAppView(BaseAppView):
     def __init__(self, context, request):
         super(UserGroupAppView, self).__init__(context, request)
         self.db_user_group = request.db_user_group
         self.db_user_group_name = self.db_user_group.users_group_name
 
 
 class UserAppView(BaseAppView):
     def __init__(self, context, request):
         super(UserAppView, self).__init__(context, request)
         self.db_user = request.db_user
         self.db_user_id = self.db_user.user_id
 
         _ = self.request.translate
         if not request.db_user_supports_default:
             if self.db_user.username == User.DEFAULT_USER:
                 h.flash(_("Editing user `{}` is disabled.".format(
                     User.DEFAULT_USER)), category='warning')
                 raise HTTPFound(h.route_path('users'))
 
 
 class DataGridAppView(object):
     """
     Common class to have re-usable grid rendering components
     """
 
     def _extract_ordering(self, request, column_map=None):
         column_map = column_map or {}
         column_index = safe_int(request.GET.get('order[0][column]'))
         order_dir = request.GET.get(
             'order[0][dir]', 'desc')
         order_by = request.GET.get(
             'columns[%s][data][sort]' % column_index, 'name_raw')
 
         # translate datatable to DB columns
         order_by = column_map.get(order_by) or order_by
 
         search_q = request.GET.get('search[value]')
         return search_q, order_by, order_dir
 
     def _extract_chunk(self, request):
         start = safe_int(request.GET.get('start'), 0)
         length = safe_int(request.GET.get('length'), 25)
         draw = safe_int(request.GET.get('draw'))
         return draw, start, length
 
     def _get_order_col(self, order_by, model):
         if isinstance(order_by, basestring):
             try:
                 return operator.attrgetter(order_by)(model)
             except AttributeError:
                 return None
         else:
             return order_by
 
 
 class BaseReferencesView(RepoAppView):
     """
     Base for reference view for branches, tags and bookmarks.
     """
     def load_default_context(self):
         c = self._get_local_tmpl_context()
 
 
         return c
 
     def load_refs_context(self, ref_items, partials_template):
         _render = self.request.get_partial_renderer(partials_template)
         pre_load = ["author", "date", "message"]
 
         is_svn = h.is_svn(self.rhodecode_vcs_repo)
         is_hg = h.is_hg(self.rhodecode_vcs_repo)
 
         format_ref_id = get_format_ref_id(self.rhodecode_vcs_repo)
 
         closed_refs = {}
         if is_hg:
             closed_refs = self.rhodecode_vcs_repo.branches_closed
 
         data = []
         for ref_name, commit_id in ref_items:
             commit = self.rhodecode_vcs_repo.get_commit(
                 commit_id=commit_id, pre_load=pre_load)
             closed = ref_name in closed_refs
 
             # TODO: johbo: Unify generation of reference links
             use_commit_id = '/' in ref_name or is_svn
 
             if use_commit_id:
                 files_url = h.route_path(
                     'repo_files',
                     repo_name=self.db_repo_name,
                     f_path=ref_name if is_svn else '',
                     commit_id=commit_id)
 
             else:
                 files_url = h.route_path(
                     'repo_files',
                     repo_name=self.db_repo_name,
                     f_path=ref_name if is_svn else '',
                     commit_id=ref_name,
                     _query=dict(at=ref_name))
 
             data.append({
                 "name": _render('name', ref_name, files_url, closed),
                 "name_raw": ref_name,
                 "date": _render('date', commit.date),
                 "date_raw": datetime_to_time(commit.date),
                 "author": _render('author', commit.author),
                 "commit": _render(
                     'commit', commit.message, commit.raw_id, commit.idx),
                 "commit_raw": commit.idx,
                 "compare": _render(
                     'compare', format_ref_id(ref_name, commit.raw_id)),
             })
 
         return data
 
 
 class RepoRoutePredicate(object):
     def __init__(self, val, config):
         self.val = val
 
     def text(self):
         return 'repo_route = %s' % self.val
 
     phash = text
 
     def __call__(self, info, request):
 
         if hasattr(request, 'vcs_call'):
             # skip vcs calls
             return
 
         repo_name = info['match']['repo_name']
         repo_model = repo.RepoModel()
         by_name_match = repo_model.get_by_repo_name(repo_name, cache=True)
 
         def redirect_if_creating(db_repo):
             if db_repo.repo_state in [repo.Repository.STATE_PENDING]:
                 raise HTTPFound(
                     request.route_path('repo_creating',
                                        repo_name=db_repo.repo_name))
 
         if by_name_match:
             # register this as request object we can re-use later
             request.db_repo = by_name_match
             redirect_if_creating(by_name_match)
             return True
 
         by_id_match = repo_model.get_repo_by_id(repo_name)
         if by_id_match:
             request.db_repo = by_id_match
             redirect_if_creating(by_id_match)
             return True
 
         return False
 
 
 class RepoTypeRoutePredicate(object):
     def __init__(self, val, config):
         self.val = val or ['hg', 'git', 'svn']
 
     def text(self):
         return 'repo_accepted_type = %s' % self.val
 
     phash = text
 
     def __call__(self, info, request):
         if hasattr(request, 'vcs_call'):
             # skip vcs calls
             return
 
         rhodecode_db_repo = request.db_repo
 
         log.debug(
             '%s checking repo type for %s in %s',
             self.__class__.__name__, rhodecode_db_repo.repo_type, self.val)
 
         if rhodecode_db_repo.repo_type in self.val:
             return True
         else:
             log.warning('Current view is not supported for repo type:%s',
                         rhodecode_db_repo.repo_type)
             #
             # h.flash(h.literal(
             #     _('Action not supported for %s.' % rhodecode_repo.alias)),
             #     category='warning')
             # return redirect(
             #     route_path('repo_summary', repo_name=cls.rhodecode_db_repo.repo_name))
 
             return False
 
 
 class RepoGroupRoutePredicate(object):
     def __init__(self, val, config):
         self.val = val
 
     def text(self):
         return 'repo_group_route = %s' % self.val
 
     phash = text
 
     def __call__(self, info, request):
         if hasattr(request, 'vcs_call'):
             # skip vcs calls
             return
 
         repo_group_name = info['match']['repo_group_name']
         repo_group_model = repo_group.RepoGroupModel()
         by_name_match = repo_group_model.get_by_group_name(
             repo_group_name, cache=True)
 
         if by_name_match:
             # register this as request object we can re-use later
             request.db_repo_group = by_name_match
             return True
 
         return False
 
 
 class UserGroupRoutePredicate(object):
     def __init__(self, val, config):
         self.val = val
 
     def text(self):
         return 'user_group_route = %s' % self.val
 
     phash = text
 
     def __call__(self, info, request):
         if hasattr(request, 'vcs_call'):
             # skip vcs calls
             return
 
         user_group_id = info['match']['user_group_id']
         user_group_model = user_group.UserGroup()
         by_id_match = user_group_model.get(
             user_group_id, cache=True)
 
         if by_id_match:
             # register this as request object we can re-use later
             request.db_user_group = by_id_match
             return True
 
         return False
 
 
 class UserRoutePredicateBase(object):
     supports_default = None
 
     def __init__(self, val, config):
         self.val = val
 
     def text(self):
         raise NotImplementedError()
 
     def __call__(self, info, request):
         if hasattr(request, 'vcs_call'):
             # skip vcs calls
             return
 
         user_id = info['match']['user_id']
         user_model = user.User()
         by_id_match = user_model.get(
             user_id, cache=True)
 
         if by_id_match:
             # register this as request object we can re-use later
             request.db_user = by_id_match
             request.db_user_supports_default = self.supports_default
             return True
 
         return False
 
 
 class UserRoutePredicate(UserRoutePredicateBase):
     supports_default = False
 
     def text(self):
         return 'user_route = %s' % self.val
 
     phash = text
 
 
 class UserRouteWithDefaultPredicate(UserRoutePredicateBase):
     supports_default = True
 
     def text(self):
         return 'user_with_default_route = %s' % self.val
 
     phash = text
 
 
 def includeme(config):
     config.add_route_predicate(
         'repo_route', RepoRoutePredicate)
     config.add_route_predicate(
         'repo_accepted_types', RepoTypeRoutePredicate)
     config.add_route_predicate(
         'repo_group_route', RepoGroupRoutePredicate)
     config.add_route_predicate(
         'user_group_route', UserGroupRoutePredicate)
     config.add_route_predicate(
         'user_route_with_default', UserRouteWithDefaultPredicate)
     config.add_route_predicate(
         'user_route', UserRoutePredicate)
  No newline at end of file

rhodecode/apps/admin/__init__.py

@@ -1,409 +1,414 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 from rhodecode.apps._base import ADMIN_PREFIX
 
 
 def admin_routes(config):
     """
     Admin prefixed routes
     """
 
     config.add_route(
         name='admin_audit_logs',
         pattern='/audit_logs')
 
     config.add_route(
         name='admin_audit_log_entry',
         pattern='/audit_logs/{audit_log_id}')
 
     config.add_route(
         name='pull_requests_global_0',  # backward compat
         pattern='/pull_requests/{pull_request_id:\d+}')
     config.add_route(
         name='pull_requests_global_1',  # backward compat
         pattern='/pull-requests/{pull_request_id:\d+}')
     config.add_route(
         name='pull_requests_global',
         pattern='/pull-request/{pull_request_id:\d+}')
 
     config.add_route(
         name='admin_settings_open_source',
         pattern='/settings/open_source')
     config.add_route(
         name='admin_settings_vcs_svn_generate_cfg',
         pattern='/settings/vcs/svn_generate_cfg')
 
     config.add_route(
         name='admin_settings_system',
         pattern='/settings/system')
     config.add_route(
         name='admin_settings_system_update',
         pattern='/settings/system/updates')
 
     config.add_route(
         name='admin_settings_sessions',
         pattern='/settings/sessions')
     config.add_route(
         name='admin_settings_sessions_cleanup',
         pattern='/settings/sessions/cleanup')
 
     config.add_route(
         name='admin_settings_process_management',
         pattern='/settings/process_management')
     config.add_route(
         name='admin_settings_process_management_data',
         pattern='/settings/process_management/data')
     config.add_route(
         name='admin_settings_process_management_signal',
         pattern='/settings/process_management/signal')
     config.add_route(
         name='admin_settings_process_management_master_signal',
         pattern='/settings/process_management/master_signal')
 
     # default settings
     config.add_route(
         name='admin_defaults_repositories',
         pattern='/defaults/repositories')
     config.add_route(
         name='admin_defaults_repositories_update',
         pattern='/defaults/repositories/update')
 
     # admin settings
 
     config.add_route(
         name='admin_settings',
         pattern='/settings')
     config.add_route(
         name='admin_settings_update',
         pattern='/settings/update')
 
     config.add_route(
         name='admin_settings_global',
         pattern='/settings/global')
     config.add_route(
         name='admin_settings_global_update',
         pattern='/settings/global/update')
 
     config.add_route(
         name='admin_settings_vcs',
         pattern='/settings/vcs')
     config.add_route(
         name='admin_settings_vcs_update',
         pattern='/settings/vcs/update')
     config.add_route(
         name='admin_settings_vcs_svn_pattern_delete',
         pattern='/settings/vcs/svn_pattern_delete')
 
     config.add_route(
         name='admin_settings_mapping',
         pattern='/settings/mapping')
     config.add_route(
         name='admin_settings_mapping_update',
         pattern='/settings/mapping/update')
 
     config.add_route(
         name='admin_settings_visual',
         pattern='/settings/visual')
     config.add_route(
         name='admin_settings_visual_update',
         pattern='/settings/visual/update')
 
 
     config.add_route(
         name='admin_settings_issuetracker',
         pattern='/settings/issue-tracker')
     config.add_route(
         name='admin_settings_issuetracker_update',
         pattern='/settings/issue-tracker/update')
     config.add_route(
         name='admin_settings_issuetracker_test',
         pattern='/settings/issue-tracker/test')
     config.add_route(
         name='admin_settings_issuetracker_delete',
         pattern='/settings/issue-tracker/delete')
 
     config.add_route(
         name='admin_settings_email',
         pattern='/settings/email')
     config.add_route(
         name='admin_settings_email_update',
         pattern='/settings/email/update')
 
     config.add_route(
         name='admin_settings_hooks',
         pattern='/settings/hooks')
     config.add_route(
         name='admin_settings_hooks_update',
         pattern='/settings/hooks/update')
     config.add_route(
         name='admin_settings_hooks_delete',
         pattern='/settings/hooks/delete')
 
     config.add_route(
         name='admin_settings_search',
         pattern='/settings/search')
 
     config.add_route(
         name='admin_settings_labs',
         pattern='/settings/labs')
     config.add_route(
         name='admin_settings_labs_update',
         pattern='/settings/labs/update')
 
+    # Automation EE feature
+    config.add_route(
+        'admin_settings_automation',
+        pattern=ADMIN_PREFIX + '/settings/automation')
+
     # global permissions
 
     config.add_route(
         name='admin_permissions_application',
         pattern='/permissions/application')
     config.add_route(
         name='admin_permissions_application_update',
         pattern='/permissions/application/update')
 
     config.add_route(
         name='admin_permissions_global',
         pattern='/permissions/global')
     config.add_route(
         name='admin_permissions_global_update',
         pattern='/permissions/global/update')
 
     config.add_route(
         name='admin_permissions_object',
         pattern='/permissions/object')
     config.add_route(
         name='admin_permissions_object_update',
         pattern='/permissions/object/update')
 
     config.add_route(
         name='admin_permissions_ips',
         pattern='/permissions/ips')
 
     config.add_route(
         name='admin_permissions_overview',
         pattern='/permissions/overview')
 
     config.add_route(
         name='admin_permissions_auth_token_access',
         pattern='/permissions/auth_token_access')
 
     config.add_route(
         name='admin_permissions_ssh_keys',
         pattern='/permissions/ssh_keys')
     config.add_route(
         name='admin_permissions_ssh_keys_data',
         pattern='/permissions/ssh_keys/data')
     config.add_route(
         name='admin_permissions_ssh_keys_update',
         pattern='/permissions/ssh_keys/update')
 
     # users admin
     config.add_route(
         name='users',
         pattern='/users')
 
     config.add_route(
         name='users_data',
         pattern='/users_data')
 
     config.add_route(
         name='users_create',
         pattern='/users/create')
 
     config.add_route(
         name='users_new',
         pattern='/users/new')
 
     # user management
     config.add_route(
         name='user_edit',
         pattern='/users/{user_id:\d+}/edit',
         user_route=True)
     config.add_route(
         name='user_edit_advanced',
         pattern='/users/{user_id:\d+}/edit/advanced',
         user_route=True)
     config.add_route(
         name='user_edit_global_perms',
         pattern='/users/{user_id:\d+}/edit/global_permissions',
         user_route=True)
     config.add_route(
         name='user_edit_global_perms_update',
         pattern='/users/{user_id:\d+}/edit/global_permissions/update',
         user_route=True)
     config.add_route(
         name='user_update',
         pattern='/users/{user_id:\d+}/update',
         user_route=True)
     config.add_route(
         name='user_delete',
         pattern='/users/{user_id:\d+}/delete',
         user_route=True)
     config.add_route(
         name='user_force_password_reset',
         pattern='/users/{user_id:\d+}/password_reset',
         user_route=True)
     config.add_route(
         name='user_create_personal_repo_group',
         pattern='/users/{user_id:\d+}/create_repo_group',
         user_route=True)
 
     # user auth tokens
     config.add_route(
         name='edit_user_auth_tokens',
         pattern='/users/{user_id:\d+}/edit/auth_tokens',
         user_route=True)
     config.add_route(
         name='edit_user_auth_tokens_add',
         pattern='/users/{user_id:\d+}/edit/auth_tokens/new',
         user_route=True)
     config.add_route(
         name='edit_user_auth_tokens_delete',
         pattern='/users/{user_id:\d+}/edit/auth_tokens/delete',
         user_route=True)
 
     # user ssh keys
     config.add_route(
         name='edit_user_ssh_keys',
         pattern='/users/{user_id:\d+}/edit/ssh_keys',
         user_route=True)
     config.add_route(
         name='edit_user_ssh_keys_generate_keypair',
         pattern='/users/{user_id:\d+}/edit/ssh_keys/generate',
         user_route=True)
     config.add_route(
         name='edit_user_ssh_keys_add',
         pattern='/users/{user_id:\d+}/edit/ssh_keys/new',
         user_route=True)
     config.add_route(
         name='edit_user_ssh_keys_delete',
         pattern='/users/{user_id:\d+}/edit/ssh_keys/delete',
         user_route=True)
 
     # user emails
     config.add_route(
         name='edit_user_emails',
         pattern='/users/{user_id:\d+}/edit/emails',
         user_route=True)
     config.add_route(
         name='edit_user_emails_add',
         pattern='/users/{user_id:\d+}/edit/emails/new',
         user_route=True)
     config.add_route(
         name='edit_user_emails_delete',
         pattern='/users/{user_id:\d+}/edit/emails/delete',
         user_route=True)
 
     # user IPs
     config.add_route(
         name='edit_user_ips',
         pattern='/users/{user_id:\d+}/edit/ips',
         user_route=True)
     config.add_route(
         name='edit_user_ips_add',
         pattern='/users/{user_id:\d+}/edit/ips/new',
         user_route_with_default=True)  # enabled for default user too
     config.add_route(
         name='edit_user_ips_delete',
         pattern='/users/{user_id:\d+}/edit/ips/delete',
         user_route_with_default=True)  # enabled for default user too
 
     # user perms
     config.add_route(
         name='edit_user_perms_summary',
         pattern='/users/{user_id:\d+}/edit/permissions_summary',
         user_route=True)
     config.add_route(
         name='edit_user_perms_summary_json',
         pattern='/users/{user_id:\d+}/edit/permissions_summary/json',
         user_route=True)
 
     # user user groups management
     config.add_route(
         name='edit_user_groups_management',
         pattern='/users/{user_id:\d+}/edit/groups_management',
         user_route=True)
 
     config.add_route(
         name='edit_user_groups_management_updates',
         pattern='/users/{user_id:\d+}/edit/edit_user_groups_management/updates',
         user_route=True)
 
     # user audit logs
     config.add_route(
         name='edit_user_audit_logs',
         pattern='/users/{user_id:\d+}/edit/audit', user_route=True)
 
     # user-groups admin
     config.add_route(
         name='user_groups',
         pattern='/user_groups')
 
     config.add_route(
         name='user_groups_data',
         pattern='/user_groups_data')
 
     config.add_route(
         name='user_groups_new',
         pattern='/user_groups/new')
 
     config.add_route(
         name='user_groups_create',
         pattern='/user_groups/create')
 
     # repos admin
     config.add_route(
         name='repos',
         pattern='/repos')
 
     config.add_route(
         name='repo_new',
         pattern='/repos/new')
 
     config.add_route(
         name='repo_create',
         pattern='/repos/create')
 
     # repo groups admin
     config.add_route(
         name='repo_groups',
         pattern='/repo_groups')
 
     config.add_route(
         name='repo_group_new',
         pattern='/repo_group/new')
 
     config.add_route(
         name='repo_group_create',
         pattern='/repo_group/create')
 
 
 def includeme(config):
     from rhodecode.apps.admin.navigation import includeme as nav_includeme
 
     # Create admin navigation registry and add it to the pyramid registry.
     nav_includeme(config)
 
     # main admin routes
     config.add_route(name='admin_home', pattern=ADMIN_PREFIX)
     config.include(admin_routes, route_prefix=ADMIN_PREFIX)
 
     # Scan module for configuration decorators.
     config.scan('.views', ignore='.tests')

rhodecode/apps/admin/navigation.py

@@ -1,143 +1,144 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import logging
 import collections
 
 from zope.interface import implementer
 
 from rhodecode.apps.admin.interfaces import IAdminNavigationRegistry
 from rhodecode.lib.utils2 import str2bool
 from rhodecode.translation import _
 
 
 log = logging.getLogger(__name__)
 
 NavListEntry = collections.namedtuple(
     'NavListEntry', ['key', 'name', 'url', 'active_list'])
 
 
 class NavEntry(object):
     """
     Represents an entry in the admin navigation.
 
     :param key: Unique identifier used to store reference in an OrderedDict.
     :param name: Display name, usually a translation string.
     :param view_name: Name of the view, used generate the URL.
     :param active_list: list of urls that we select active for this element
     """
 
     def __init__(self, key, name, view_name, active_list=None):
         self.key = key
         self.name = name
         self.view_name = view_name
         self._active_list = active_list or []
 
     def generate_url(self, request):
         return request.route_path(self.view_name)
 
     def get_localized_name(self, request):
         return request.translate(self.name)
 
     @property
     def active_list(self):
         active_list = [self.key]
         if self._active_list:
             active_list = self._active_list
         return active_list
 
 
 @implementer(IAdminNavigationRegistry)
 class NavigationRegistry(object):
 
     _base_entries = [
         NavEntry('global', _('Global'),
                  'admin_settings_global'),
         NavEntry('vcs', _('VCS'),
                  'admin_settings_vcs'),
         NavEntry('visual', _('Visual'),
                  'admin_settings_visual'),
         NavEntry('mapping', _('Remap and Rescan'),
                  'admin_settings_mapping'),
         NavEntry('issuetracker', _('Issue Tracker'),
                  'admin_settings_issuetracker'),
         NavEntry('email', _('Email'),
                  'admin_settings_email'),
         NavEntry('hooks', _('Hooks'),
                  'admin_settings_hooks'),
         NavEntry('search', _('Full Text Search'),
                  'admin_settings_search'),
         NavEntry('integrations', _('Integrations'),
                  'global_integrations_home'),
         NavEntry('system', _('System Info'),
                  'admin_settings_system'),
         NavEntry('process_management', _('Processes'),
                  'admin_settings_process_management'),
         NavEntry('sessions', _('User Sessions'),
                  'admin_settings_sessions'),
         NavEntry('open_source', _('Open Source Licenses'),
                  'admin_settings_open_source'),
-
+        NavEntry('automation', _('Automation'),
+                 'admin_settings_automation')
     ]
 
     _labs_entry = NavEntry('labs', _('Labs'),
                            'admin_settings_labs')
 
     def __init__(self, labs_active=False):
         self._registered_entries = collections.OrderedDict()
         for item in self.__class__._base_entries:
             self._registered_entries[item.key] = item
 
         if labs_active:
             self.add_entry(self._labs_entry)
 
     def add_entry(self, entry):
         self._registered_entries[entry.key] = entry
 
     def get_navlist(self, request):
         navlist = [NavListEntry(i.key, i.get_localized_name(request),
                                 i.generate_url(request), i.active_list)
                    for i in self._registered_entries.values()]
         return navlist
 
 
 def navigation_registry(request, registry=None):
     """
     Helper that returns the admin navigation registry.
     """
     pyramid_registry = registry or request.registry
     nav_registry = pyramid_registry.queryUtility(IAdminNavigationRegistry)
     return nav_registry
 
 
 def navigation_list(request):
     """
     Helper that returns the admin navigation as list of NavListEntry objects.
     """
     return navigation_registry(request).get_navlist(request)
 
 
 def includeme(config):
     # Create admin navigation registry and add it to the pyramid registry.
     settings = config.get_settings()
     labs_active = str2bool(settings.get('labs_settings_active', False))
     navigation_registry = NavigationRegistry(labs_active=labs_active)
     config.registry.registerUtility(navigation_registry)
  No newline at end of file

rhodecode/apps/admin/tests/test_admin_auth_settings.py

@@ -1,201 +1,202 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import pytest
 
 from rhodecode.tests import assert_session_flash
 from rhodecode.tests.utils import AssertResponse
 from rhodecode.model.db import Session
 from rhodecode.model.settings import SettingsModel
 
 
 def assert_auth_settings_updated(response):
     assert response.status_int == 302, 'Expected response HTTP Found 302'
     assert_session_flash(response, 'Auth settings updated successfully')
 
 
 @pytest.mark.usefixtures("autologin_user", "app")
 class TestAuthSettingsView(object):
 
     def _enable_plugins(self, plugins_list, csrf_token, override=None,
                         verify_response=False):
         test_url = '/_admin/auth'
         params = {
             'auth_plugins': plugins_list,
             'csrf_token': csrf_token,
         }
         if override:
             params.update(override)
         _enabled_plugins = []
         for plugin in plugins_list.split(','):
             plugin_name = plugin.partition('#')[-1]
             enabled_plugin = '%s_enabled' % plugin_name
             cache_ttl = '%s_cache_ttl' % plugin_name
 
             # default params that are needed for each plugin,
             # `enabled` and `cache_ttl`
             params.update({
                 enabled_plugin: True,
                 cache_ttl: 0
             })
             _enabled_plugins.append(enabled_plugin)
 
         # we need to clean any enabled plugin before, since they require
         # form params to be present
         db_plugin = SettingsModel().get_setting_by_name('auth_plugins')
         db_plugin.app_settings_value = \
             'egg:rhodecode-enterprise-ce#rhodecode'
         Session().add(db_plugin)
         Session().commit()
         for _plugin in _enabled_plugins:
             db_plugin = SettingsModel().get_setting_by_name(_plugin)
             if db_plugin:
                 Session().delete(db_plugin)
         Session().commit()
 
         response = self.app.post(url=test_url, params=params)
 
         if verify_response:
             assert_auth_settings_updated(response)
         return params
 
     def _post_ldap_settings(self, params, override=None, force=False):
 
         params.update({
             'filter': 'user',
             'user_member_of': '',
             'user_search_base': '',
             'user_search_filter': 'test_filter',
 
             'host': 'dc.example.com',
             'port': '999',
+            'timeout': 3600,
             'tls_kind': 'PLAIN',
             'tls_reqcert': 'NEVER',
 
             'dn_user': 'test_user',
             'dn_pass': 'test_pass',
             'base_dn': 'test_base_dn',
             'search_scope': 'BASE',
             'attr_login': 'test_attr_login',
             'attr_firstname': 'ima',
             'attr_lastname': 'tester',
             'attr_email': '[e-mail]',
             'cache_ttl': '0',
         })
         if force:
             params = {}
         params.update(override or {})
 
         test_url = '/_admin/auth/ldap/'
 
         response = self.app.post(url=test_url, params=params)
         return response
 
     def test_index(self):
         response = self.app.get('/_admin/auth')
         response.mustcontain('Authentication Plugins')
 
     @pytest.mark.parametrize("disable_plugin, needs_import", [
         ('egg:rhodecode-enterprise-ce#headers', None),
         ('egg:rhodecode-enterprise-ce#crowd', None),
         ('egg:rhodecode-enterprise-ce#jasig_cas', None),
         ('egg:rhodecode-enterprise-ce#ldap', None),
         ('egg:rhodecode-enterprise-ce#pam',  "pam"),
     ])
     def test_disable_plugin(self, csrf_token, disable_plugin, needs_import):
         # TODO: johbo: "pam" is currently not available on darwin,
         # although the docs state that it should work on darwin.
         if needs_import:
             pytest.importorskip(needs_import)
 
         self._enable_plugins(
             'egg:rhodecode-enterprise-ce#rhodecode,' + disable_plugin,
             csrf_token, verify_response=True)
 
         self._enable_plugins(
             'egg:rhodecode-enterprise-ce#rhodecode', csrf_token,
             verify_response=True)
 
     def test_ldap_save_settings(self, csrf_token):
         params = self._enable_plugins(
             'egg:rhodecode-enterprise-ce#rhodecode,'
             'egg:rhodecode-enterprise-ce#ldap',
             csrf_token)
         response = self._post_ldap_settings(params)
         assert_auth_settings_updated(response)
 
         new_settings = SettingsModel().get_auth_settings()
         assert new_settings['auth_ldap_host'] == u'dc.example.com', \
             'fail db write compare'
 
     def test_ldap_error_form_wrong_port_number(self, csrf_token):
         params = self._enable_plugins(
             'egg:rhodecode-enterprise-ce#rhodecode,'
             'egg:rhodecode-enterprise-ce#ldap',
             csrf_token)
         invalid_port_value = 'invalid-port-number'
         response = self._post_ldap_settings(params, override={
             'port': invalid_port_value,
         })
         assertr = AssertResponse(response)
         assertr.element_contains(
             '.form .field #port ~ .error-message',
             invalid_port_value)
 
     def test_ldap_error_form(self, csrf_token):
         params = self._enable_plugins(
             'egg:rhodecode-enterprise-ce#rhodecode,'
             'egg:rhodecode-enterprise-ce#ldap',
             csrf_token)
         response = self._post_ldap_settings(params, override={
             'attr_login': '',
         })
         response.mustcontain("""<span class="error-message">The LDAP Login"""
                              """ attribute of the CN must be specified""")
 
     def test_post_ldap_group_settings(self, csrf_token):
         params = self._enable_plugins(
             'egg:rhodecode-enterprise-ce#rhodecode,'
             'egg:rhodecode-enterprise-ce#ldap',
             csrf_token)
 
         response = self._post_ldap_settings(params, override={
             'host': 'dc-legacy.example.com',
             'port': '999',
             'tls_kind': 'PLAIN',
             'tls_reqcert': 'NEVER',
             'dn_user': 'test_user',
             'dn_pass': 'test_pass',
             'base_dn': 'test_base_dn',
             'filter': 'test_filter',
             'search_scope': 'BASE',
             'attr_login': 'test_attr_login',
             'attr_firstname': 'ima',
             'attr_lastname': 'tester',
             'attr_email': '[e-mail]',
             'cache_ttl': '60',
             'csrf_token': csrf_token,
             }
         )
         assert_auth_settings_updated(response)
 
         new_settings = SettingsModel().get_auth_settings()
         assert new_settings['auth_ldap_host'] == u'dc-legacy.example.com', \
             'fail db write compare'

rhodecode/apps/admin/tests/test_admin_users_ssh_keys.py

@@ -1,173 +1,176 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import pytest
 
 from rhodecode.model.db import User, UserSshKeys
 
 from rhodecode.tests import TestController, assert_session_flash
 from rhodecode.tests.fixture import Fixture
 
 fixture = Fixture()
 
 
 def route_path(name, params=None, **kwargs):
     import urllib
     from rhodecode.apps._base import ADMIN_PREFIX
 
     base_url = {
         'edit_user_ssh_keys':
             ADMIN_PREFIX + '/users/{user_id}/edit/ssh_keys',
         'edit_user_ssh_keys_generate_keypair':
             ADMIN_PREFIX + '/users/{user_id}/edit/ssh_keys/generate',
         'edit_user_ssh_keys_add':
             ADMIN_PREFIX + '/users/{user_id}/edit/ssh_keys/new',
         'edit_user_ssh_keys_delete':
             ADMIN_PREFIX + '/users/{user_id}/edit/ssh_keys/delete',
 
     }[name].format(**kwargs)
 
     if params:
         base_url = '{}?{}'.format(base_url, urllib.urlencode(params))
     return base_url
 
 
 class TestAdminUsersSshKeysView(TestController):
     INVALID_KEY = """\
             ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDk+77sjDzVeB6vevJsuZds1iNU5
             LANOa5CU5G/9JYIA6RYsWWMO7mbsR82IUckdqOHmxSykfR1D1TdluyIpQLrwgH5kb
             n8FkVI8zBMCKakxowvN67B0R7b1BT4PPzW2JlOXei/m9W12ZY484VTow6/B+kf2Q8
             cP8tmCJmKWZma5Em7OTUhvjyQVNz3v7HfeY5Hq0Ci4ECJ59hepFDabJvtAXg9XrI6
             jvdphZTc30I4fG8+hBHzpeFxUGvSGNtXPUbwaAY8j/oHYrTpMgkj6pUEFsiKfC5zP
             qPFR5HyKTCHW0nFUJnZsbyFT5hMiF/hZkJc9A0ZbdSvJwCRQ/g3bmdL 
             [e-mail]      
         """
     VALID_KEY = 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDk+77sjDzVeB6vev' \
                    'JsuZds1iNU5LANOa5CU5G/9JYIA6RYsWWMO7mbsR82IUckdqOHmxSy' \
                    'kfR1D1TdluyIpQLrwgH5kbn8FkVI8zBMCKakxowvN67B0R7b1BT4PP' \
                    'zW2JlOXei/m9W12ZY484VTow6/B+kf2Q8cP8tmCJmKWZma5Em7OTUh' \
                    'vjyQVNz3v7HfeY5Hq0Ci4ECJ59hepFDabJvtAXg9XrI6jvdphZTc30' \
                    'I4fG8+hBHzpeFxUGvSGNtXPUbwaAY8j/oHYrTpMgkj6pUEFsiKfC5zPq' \
                    'PFR5HyKTCHW0nFUJnZsbyFT5hMiF/hZkJc9A0ZbdSvJwCRQ/g3bmdL ' \
                    '[e-mail]'
+    FINGERPRINT = 'MD5:01:4f:ad:29:22:6e:01:37:c9:d2:52:26:52:b0:2d:93'
 
     def test_ssh_keys_default_user(self):
         self.log_user()
         user = User.get_default_user()
         self.app.get(
             route_path('edit_user_ssh_keys', user_id=user.user_id),
             status=302)
 
     def test_add_ssh_key_error(self, user_util):
         self.log_user()
         user = user_util.create_user()
         user_id = user.user_id
 
         key_data = self.INVALID_KEY
 
         desc = 'MY SSH KEY'
         response = self.app.post(
             route_path('edit_user_ssh_keys_add', user_id=user_id),
             {'description': desc, 'key_data': key_data,
              'csrf_token': self.csrf_token})
         assert_session_flash(response, 'An error occurred during ssh '
                                        'key saving: Unable to decode the key')
 
     def test_ssh_key_duplicate(self, user_util):
         self.log_user()
         user = user_util.create_user()
         user_id = user.user_id
 
         key_data = self.VALID_KEY
 
         desc = 'MY SSH KEY'
         response = self.app.post(
             route_path('edit_user_ssh_keys_add', user_id=user_id),
             {'description': desc, 'key_data': key_data,
              'csrf_token': self.csrf_token})
         assert_session_flash(response, 'Ssh Key successfully created')
         response.follow()  # flush session flash
 
         # add the same key AGAIN
         desc = 'MY SSH KEY'
         response = self.app.post(
             route_path('edit_user_ssh_keys_add', user_id=user_id),
             {'description': desc, 'key_data': key_data,
              'csrf_token': self.csrf_token})
+
+        err = 'Such key with fingerprint `{}` already exists, ' \
+              'please use a different one'.format(self.FINGERPRINT)
         assert_session_flash(response, 'An error occurred during ssh key '
-                                       'saving: Such key already exists, '
-                                       'please use a different one')
+                                       'saving: {}'.format(err))
 
     def test_add_ssh_key(self, user_util):
         self.log_user()
         user = user_util.create_user()
         user_id = user.user_id
 
         key_data = self.VALID_KEY
 
         desc = 'MY SSH KEY'
         response = self.app.post(
             route_path('edit_user_ssh_keys_add', user_id=user_id),
             {'description': desc, 'key_data': key_data,
              'csrf_token': self.csrf_token})
         assert_session_flash(response, 'Ssh Key successfully created')
 
         response = response.follow()
         response.mustcontain(desc)
 
     def test_delete_ssh_key(self, user_util):
         self.log_user()
         user = user_util.create_user()
         user_id = user.user_id
 
         key_data = self.VALID_KEY
 
         desc = 'MY SSH KEY'
         response = self.app.post(
             route_path('edit_user_ssh_keys_add', user_id=user_id),
             {'description': desc, 'key_data': key_data,
              'csrf_token': self.csrf_token})
         assert_session_flash(response, 'Ssh Key successfully created')
         response = response.follow()  # flush the Session flash
 
         # now delete our key
         keys = UserSshKeys.query().filter(UserSshKeys.user_id == user_id).all()
         assert 1 == len(keys)
 
         response = self.app.post(
             route_path('edit_user_ssh_keys_delete', user_id=user_id),
             {'del_ssh_key': keys[0].ssh_key_id,
              'csrf_token': self.csrf_token})
 
         assert_session_flash(response, 'Ssh key successfully deleted')
         keys = UserSshKeys.query().filter(UserSshKeys.user_id == user_id).all()
         assert 0 == len(keys)
 
     def test_generate_keypair(self, user_util):
         self.log_user()
         user = user_util.create_user()
         user_id = user.user_id
 
         response = self.app.get(
             route_path('edit_user_ssh_keys_generate_keypair', user_id=user_id))
 
         response.mustcontain('Private key')
         response.mustcontain('Public key')
         response.mustcontain('-----BEGIN RSA PRIVATE KEY-----')

rhodecode/apps/admin/views/process_management.py

@@ -1,133 +1,171 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 
 import psutil
 import signal
 from pyramid.view import view_config
 
 from rhodecode.apps._base import BaseAppView
 from rhodecode.apps.admin.navigation import navigation_list
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator, CSRFRequired)
-from rhodecode.lib.utils2 import safe_int
+from rhodecode.lib.utils2 import safe_int, StrictAttributeDict
 
 log = logging.getLogger(__name__)
 
 
 class AdminProcessManagementView(BaseAppView):
     def load_default_context(self):
         c = self._get_local_tmpl_context()
+        return c
 
-        return c
+    def _format_proc(self, proc, with_children=False):
+        try:
+            mem = proc.memory_info()
+            proc_formatted = StrictAttributeDict({
+                'pid': proc.pid,
+                'name': proc.name(),
+                'mem_rss': mem.rss,
+                'mem_vms': mem.vms,
+                'cpu_percent': proc.cpu_percent(),
+                'create_time': proc.create_time(),
+                'cmd': ' '.join(proc.cmdline()),
+            })
+
+            if with_children:
+                proc_formatted.update({
+                    'children': [self._format_proc(x)
+                                 for x in proc.children(recursive=True)]
+                })
+        except Exception:
+            log.exception('Failed to load proc')
+            proc_formatted = None
+        return proc_formatted
+
+    def get_processes(self):
+        proc_list = []
+        for p in psutil.process_iter():
+            if 'gunicorn' in p.name():
+                proc = self._format_proc(p, with_children=True)
+                if proc:
+                    proc_list.append(proc)
+
+        return proc_list
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_process_management', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def process_management(self):
         _ = self.request.translate
         c = self.load_default_context()
 
         c.active = 'process_management'
         c.navlist = navigation_list(self.request)
-        c.gunicorn_processes = (
-            p for p in psutil.process_iter() if 'gunicorn' in p.name())
+        c.gunicorn_processes = self.get_processes()
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_process_management_data', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings_process_management_data.mako')
     def process_management_data(self):
         _ = self.request.translate
         c = self.load_default_context()
-        c.gunicorn_processes = (
-            p for p in psutil.process_iter() if 'gunicorn' in p.name())
+        c.gunicorn_processes = self.get_processes()
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_process_management_signal',
         request_method='POST', renderer='json_ext')
     def process_management_signal(self):
         pids = self.request.json.get('pids', [])
         result = []
+
         def on_terminate(proc):
             msg = "process `PID:{}` terminated with exit code {}".format(
-                proc.pid, proc.returncode)
+                proc.pid, proc.returncode or 0)
             result.append(msg)
 
         procs = []
         for pid in pids:
             pid = safe_int(pid)
             if pid:
                 try:
                     proc = psutil.Process(pid)
                 except psutil.NoSuchProcess:
                     continue
 
                 children = proc.children(recursive=True)
                 if children:
-                    print('Wont kill Master Process')
+                    log.warning('Wont kill Master Process')
                 else:
                     procs.append(proc)
 
         for p in procs:
+            try:
             p.terminate()
+            except psutil.AccessDenied as e:
+                log.warning('Access denied: {}'.format(e))
+
         gone, alive = psutil.wait_procs(procs, timeout=10, callback=on_terminate)
         for p in alive:
+            try:
             p.kill()
+            except psutil.AccessDenied as e:
+                log.warning('Access denied: {}'.format(e))
 
         return {'result': result}
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_process_management_master_signal',
         request_method='POST', renderer='json_ext')
     def process_management_master_signal(self):
         pid_data = self.request.json.get('pid_data', {})
         pid = safe_int(pid_data['pid'])
         action = pid_data['action']
         if pid:
             try:
                 proc = psutil.Process(pid)
             except psutil.NoSuchProcess:
                 return {'result': 'failure_no_such_process'}
 
             children = proc.children(recursive=True)
             if children:
                 # master process
                 if action == '+' and len(children) <= 20:
                     proc.send_signal(signal.SIGTTIN)
                 elif action == '-' and len(children) >= 2:
                     proc.send_signal(signal.SIGTTOU)
                 else:
                     return {'result': 'failure_wrong_action'}
                 return {'result': 'success'}
 
         return {'result': 'failure_not_master'}

rhodecode/apps/admin/views/settings.py

@@ -1,764 +1,781 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2010-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import logging
 import collections
 
 import datetime
 import formencode
 import formencode.htmlfill
 
 import rhodecode
 from pyramid.view import view_config
 from pyramid.httpexceptions import HTTPFound, HTTPNotFound
 from pyramid.renderers import render
 from pyramid.response import Response
 
 from rhodecode.apps._base import BaseAppView
 from rhodecode.apps.admin.navigation import navigation_list
 from rhodecode.apps.svn_support.config_keys import generate_config
 from rhodecode.lib import helpers as h
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator, CSRFRequired)
 from rhodecode.lib.celerylib import tasks, run_task
 from rhodecode.lib.utils import repo2db_mapper
 from rhodecode.lib.utils2 import str2bool, safe_unicode, AttributeDict
 from rhodecode.lib.index import searcher_from_config
 
 from rhodecode.model.db import RhodeCodeUi, Repository
 from rhodecode.model.forms import (ApplicationSettingsForm,
     ApplicationUiSettingsForm, ApplicationVisualisationForm,
     LabsSettingsForm, IssueTrackerPatternsForm)
 from rhodecode.model.repo_group import RepoGroupModel
 
 from rhodecode.model.scm import ScmModel
 from rhodecode.model.notification import EmailNotificationModel
 from rhodecode.model.meta import Session
 from rhodecode.model.settings import (
     IssueTrackerSettingsModel, VcsSettingsModel, SettingNotFound,
     SettingsModel)
 
 
 log = logging.getLogger(__name__)
 
 
 class AdminSettingsView(BaseAppView):
 
     def load_default_context(self):
         c = self._get_local_tmpl_context()
         c.labs_active = str2bool(
             rhodecode.CONFIG.get('labs_settings_active', 'true'))
         c.navlist = navigation_list(self.request)
 
         return c
 
     @classmethod
     def _get_ui_settings(cls):
         ret = RhodeCodeUi.query().all()
 
         if not ret:
             raise Exception('Could not get application ui settings !')
         settings = {}
         for each in ret:
             k = each.ui_key
             v = each.ui_value
             if k == '/':
                 k = 'root_path'
 
             if k in ['push_ssl', 'publish', 'enabled']:
                 v = str2bool(v)
 
             if k.find('.') != -1:
                 k = k.replace('.', '_')
 
             if each.ui_section in ['hooks', 'extensions']:
                 v = each.ui_active
 
             settings[each.ui_section + '_' + k] = v
         return settings
 
     @classmethod
     def _form_defaults(cls):
         defaults = SettingsModel().get_all_settings()
         defaults.update(cls._get_ui_settings())
 
         defaults.update({
             'new_svn_branch': '',
             'new_svn_tag': '',
         })
         return defaults
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_vcs', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_vcs(self):
         c = self.load_default_context()
         c.active = 'vcs'
         model = VcsSettingsModel()
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
 
         settings = self.request.registry.settings
         c.svn_proxy_generate_config = settings[generate_config]
 
         defaults = self._form_defaults()
 
         model.create_largeobjects_dirs_if_needed(defaults['paths_root_path'])
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_vcs_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_vcs_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'vcs'
 
         model = VcsSettingsModel()
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
 
         settings = self.request.registry.settings
         c.svn_proxy_generate_config = settings[generate_config]
 
         application_form = ApplicationUiSettingsForm(self.request.translate)()
 
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _("Some form inputs contain invalid data."),
                 category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         try:
             if c.visual.allow_repo_location_change:
                 model.update_global_path_setting(
                     form_result['paths_root_path'])
 
             model.update_global_ssl_setting(form_result['web_push_ssl'])
             model.update_global_hook_settings(form_result)
 
             model.create_or_update_global_svn_settings(form_result)
             model.create_or_update_global_hg_settings(form_result)
             model.create_or_update_global_git_settings(form_result)
             model.create_or_update_global_pr_settings(form_result)
         except Exception:
             log.exception("Exception while updating settings")
             h.flash(_('Error occurred during updating '
                       'application settings'), category='error')
         else:
             Session().commit()
             h.flash(_('Updated VCS settings'), category='success')
             raise HTTPFound(h.route_path('admin_settings_vcs'))
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_vcs_svn_pattern_delete', request_method='POST',
         renderer='json_ext', xhr=True)
     def settings_vcs_delete_svn_pattern(self):
         delete_pattern_id = self.request.POST.get('delete_svn_pattern')
         model = VcsSettingsModel()
         try:
             model.delete_global_svn_pattern(delete_pattern_id)
         except SettingNotFound:
             log.exception(
                 'Failed to delete svn_pattern with id %s', delete_pattern_id)
             raise HTTPNotFound()
 
         Session().commit()
         return True
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_mapping', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_mapping(self):
         c = self.load_default_context()
         c.active = 'mapping'
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_mapping_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_mapping_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'mapping'
         rm_obsolete = self.request.POST.get('destroy', False)
         invalidate_cache = self.request.POST.get('invalidate', False)
         log.debug(
             'rescanning repo location with destroy obsolete=%s', rm_obsolete)
 
         if invalidate_cache:
             log.debug('invalidating all repositories cache')
             for repo in Repository.get_all():
                 ScmModel().mark_for_invalidation(repo.repo_name, delete=True)
 
         filesystem_repos = ScmModel().repo_scan()
         added, removed = repo2db_mapper(filesystem_repos, rm_obsolete)
         _repr = lambda l: ', '.join(map(safe_unicode, l)) or '-'
         h.flash(_('Repositories successfully '
                   'rescanned added: %s ; removed: %s') %
                 (_repr(added), _repr(removed)),
                 category='success')
         raise HTTPFound(h.route_path('admin_settings_mapping'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     @view_config(
         route_name='admin_settings_global', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_global(self):
         c = self.load_default_context()
         c.active = 'global'
         c.personal_repo_group_default_pattern = RepoGroupModel()\
             .get_personal_group_name_pattern()
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     @view_config(
         route_name='admin_settings_global_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_global_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'global'
         c.personal_repo_group_default_pattern = RepoGroupModel()\
             .get_personal_group_name_pattern()
         application_form = ApplicationSettingsForm(self.request.translate)()
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
+            h.flash(
+                _("Some form inputs contain invalid data."),
+                category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         settings = [
             ('title', 'rhodecode_title', 'unicode'),
             ('realm', 'rhodecode_realm', 'unicode'),
             ('pre_code', 'rhodecode_pre_code', 'unicode'),
             ('post_code', 'rhodecode_post_code', 'unicode'),
             ('captcha_public_key', 'rhodecode_captcha_public_key', 'unicode'),
             ('captcha_private_key', 'rhodecode_captcha_private_key', 'unicode'),
             ('create_personal_repo_group', 'rhodecode_create_personal_repo_group', 'bool'),
             ('personal_repo_group_pattern', 'rhodecode_personal_repo_group_pattern', 'unicode'),
         ]
         try:
             for setting, form_key, type_ in settings:
                 sett = SettingsModel().create_or_update_setting(
                     setting, form_result[form_key], type_)
                 Session().add(sett)
 
             Session().commit()
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated application settings'), category='success')
         except Exception:
             log.exception("Exception while updating application settings")
             h.flash(
                 _('Error occurred during updating application settings'),
                 category='error')
 
         raise HTTPFound(h.route_path('admin_settings_global'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_visual', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_visual(self):
         c = self.load_default_context()
         c.active = 'visual'
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_visual_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_visual_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'visual'
         application_form = ApplicationVisualisationForm(self.request.translate)()
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
+            h.flash(
+                _("Some form inputs contain invalid data."),
+                category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         try:
             settings = [
                 ('show_public_icon', 'rhodecode_show_public_icon', 'bool'),
                 ('show_private_icon', 'rhodecode_show_private_icon', 'bool'),
                 ('stylify_metatags', 'rhodecode_stylify_metatags', 'bool'),
                 ('repository_fields', 'rhodecode_repository_fields', 'bool'),
                 ('dashboard_items', 'rhodecode_dashboard_items', 'int'),
                 ('admin_grid_items', 'rhodecode_admin_grid_items', 'int'),
                 ('show_version', 'rhodecode_show_version', 'bool'),
                 ('use_gravatar', 'rhodecode_use_gravatar', 'bool'),
                 ('markup_renderer', 'rhodecode_markup_renderer', 'unicode'),
                 ('gravatar_url', 'rhodecode_gravatar_url', 'unicode'),
                 ('clone_uri_tmpl', 'rhodecode_clone_uri_tmpl', 'unicode'),
                 ('clone_uri_ssh_tmpl', 'rhodecode_clone_uri_ssh_tmpl', 'unicode'),
                 ('support_url', 'rhodecode_support_url', 'unicode'),
                 ('show_revision_number', 'rhodecode_show_revision_number', 'bool'),
                 ('show_sha_length', 'rhodecode_show_sha_length', 'int'),
             ]
             for setting, form_key, type_ in settings:
                 sett = SettingsModel().create_or_update_setting(
                     setting, form_result[form_key], type_)
                 Session().add(sett)
 
             Session().commit()
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated visualisation settings'), category='success')
         except Exception:
             log.exception("Exception updating visualization settings")
             h.flash(_('Error occurred during updating '
                       'visualisation settings'),
                     category='error')
 
         raise HTTPFound(h.route_path('admin_settings_visual'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_issuetracker', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_issuetracker(self):
         c = self.load_default_context()
         c.active = 'issuetracker'
         defaults = SettingsModel().get_all_settings()
 
         entry_key = 'rhodecode_issuetracker_pat_'
 
         c.issuetracker_entries = {}
         for k, v in defaults.items():
             if k.startswith(entry_key):
                 uid = k[len(entry_key):]
                 c.issuetracker_entries[uid] = None
 
         for uid in c.issuetracker_entries:
             c.issuetracker_entries[uid] = AttributeDict({
                 'pat': defaults.get('rhodecode_issuetracker_pat_' + uid),
                 'url': defaults.get('rhodecode_issuetracker_url_' + uid),
                 'pref': defaults.get('rhodecode_issuetracker_pref_' + uid),
                 'desc': defaults.get('rhodecode_issuetracker_desc_' + uid),
             })
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_issuetracker_test', request_method='POST',
         renderer='string', xhr=True)
     def settings_issuetracker_test(self):
         return h.urlify_commit_message(
             self.request.POST.get('test_text', ''),
             'repo_group/test_repo1')
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_issuetracker_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_issuetracker_update(self):
         _ = self.request.translate
         self.load_default_context()
         settings_model = IssueTrackerSettingsModel()
 
         try:
             form = IssueTrackerPatternsForm(self.request.translate)()
             data = form.to_python(self.request.POST)
         except formencode.Invalid as errors:
             log.exception('Failed to add new pattern')
             error = errors
             h.flash(_('Invalid issue tracker pattern: {}'.format(error)),
                     category='error')
             raise HTTPFound(h.route_path('admin_settings_issuetracker'))
 
         if data:
             for uid in data.get('delete_patterns', []):
                 settings_model.delete_entries(uid)
 
             for pattern in data.get('patterns', []):
                 for setting, value, type_ in pattern:
                     sett = settings_model.create_or_update_setting(
                         setting, value, type_)
                     Session().add(sett)
 
                 Session().commit()
 
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated issue tracker entries'), category='success')
         raise HTTPFound(h.route_path('admin_settings_issuetracker'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_issuetracker_delete', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_issuetracker_delete(self):
         _ = self.request.translate
         self.load_default_context()
         uid = self.request.POST.get('uid')
         try:
             IssueTrackerSettingsModel().delete_entries(uid)
         except Exception:
             log.exception('Failed to delete issue tracker setting %s', uid)
             raise HTTPNotFound()
         h.flash(_('Removed issue tracker entry'), category='success')
         raise HTTPFound(h.route_path('admin_settings_issuetracker'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_email', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_email(self):
         c = self.load_default_context()
         c.active = 'email'
         c.rhodecode_ini = rhodecode.CONFIG
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_email_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_email_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'email'
 
         test_email = self.request.POST.get('test_email')
 
         if not test_email:
             h.flash(_('Please enter email address'), category='error')
             raise HTTPFound(h.route_path('admin_settings_email'))
 
         email_kwargs = {
             'date': datetime.datetime.now(),
             'user': c.rhodecode_user,
             'rhodecode_version': c.rhodecode_version
         }
 
         (subject, headers, email_body,
          email_body_plaintext) = EmailNotificationModel().render_email(
             EmailNotificationModel.TYPE_EMAIL_TEST, **email_kwargs)
 
         recipients = [test_email] if test_email else None
 
         run_task(tasks.send_email, recipients, subject,
                  email_body_plaintext, email_body)
 
         h.flash(_('Send email task created'), category='success')
         raise HTTPFound(h.route_path('admin_settings_email'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_hooks', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_hooks(self):
         c = self.load_default_context()
         c.active = 'hooks'
 
         model = SettingsModel()
         c.hooks = model.get_builtin_hooks()
         c.custom_hooks = model.get_custom_hooks()
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_hooks_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     @view_config(
         route_name='admin_settings_hooks_delete', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_hooks_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'hooks'
         if c.visual.allow_custom_hooks_settings:
             ui_key = self.request.POST.get('new_hook_ui_key')
             ui_value = self.request.POST.get('new_hook_ui_value')
 
             hook_id = self.request.POST.get('hook_id')
             new_hook = False
 
             model = SettingsModel()
             try:
                 if ui_value and ui_key:
                     model.create_or_update_hook(ui_key, ui_value)
                     h.flash(_('Added new hook'), category='success')
                     new_hook = True
                 elif hook_id:
                     RhodeCodeUi.delete(hook_id)
                     Session().commit()
 
                 # check for edits
                 update = False
                 _d = self.request.POST.dict_of_lists()
                 for k, v in zip(_d.get('hook_ui_key', []),
                                 _d.get('hook_ui_value_new', [])):
                     model.create_or_update_hook(k, v)
                     update = True
 
                 if update and not new_hook:
                     h.flash(_('Updated hooks'), category='success')
                 Session().commit()
             except Exception:
                 log.exception("Exception during hook creation")
                 h.flash(_('Error occurred during hook creation'),
                         category='error')
 
         raise HTTPFound(h.route_path('admin_settings_hooks'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_search', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_search(self):
         c = self.load_default_context()
         c.active = 'search'
 
         searcher = searcher_from_config(self.request.registry.settings)
         c.statistics = searcher.statistics(self.request.translate)
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
+        route_name='admin_settings_automation', request_method='GET',
+        renderer='rhodecode:templates/admin/settings/settings.mako')
+    def settings_automation(self):
+        c = self.load_default_context()
+        c.active = 'automation'
+
+        return self._get_template_context(c)
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    @view_config(
         route_name='admin_settings_labs', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_labs(self):
         c = self.load_default_context()
         if not c.labs_active:
             raise HTTPFound(h.route_path('admin_settings'))
 
         c.active = 'labs'
         c.lab_settings = _LAB_SETTINGS
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     @view_config(
         route_name='admin_settings_labs_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_labs_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'labs'
 
         application_form = LabsSettingsForm(self.request.translate)()
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
             h.flash(
-                _('Some form inputs contain invalid data.'),
+                _("Some form inputs contain invalid data."),
                 category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.error_dict or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         try:
             session = Session()
             for setting in _LAB_SETTINGS:
                 setting_name = setting.key[len('rhodecode_'):]
                 sett = SettingsModel().create_or_update_setting(
                     setting_name, form_result[setting.key], setting.type)
                 session.add(sett)
 
         except Exception:
             log.exception('Exception while updating lab settings')
             h.flash(_('Error occurred during updating labs settings'),
                     category='error')
         else:
             Session().commit()
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated Labs settings'), category='success')
             raise HTTPFound(h.route_path('admin_settings_labs'))
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
 
 # :param key: name of the setting including the 'rhodecode_' prefix
 # :param type: the RhodeCodeSetting type to use.
 # :param group: the i18ned group in which we should dispaly this setting
 # :param label: the i18ned label we should display for this setting
 # :param help: the i18ned help we should dispaly for this setting
 LabSetting = collections.namedtuple(
     'LabSetting', ('key', 'type', 'group', 'label', 'help'))
 
 
 # This list has to be kept in sync with the form
 # rhodecode.model.forms.LabsSettingsForm.
 _LAB_SETTINGS = [
 
 ]

rhodecode/apps/admin/views/system_info.py

@@ -1,197 +1,200 b''
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2016-2018 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 import urllib2
 
 from pyramid.view import view_config
 
 import rhodecode
 from rhodecode.apps._base import BaseAppView
 from rhodecode.apps.admin.navigation import navigation_list
 from rhodecode.lib import helpers as h
 from rhodecode.lib.auth import (LoginRequired, HasPermissionAllDecorator)
 from rhodecode.lib.utils2 import str2bool
 from rhodecode.lib import system_info
 from rhodecode.model.update import UpdateModel
 
 log = logging.getLogger(__name__)
 
 
 class AdminSystemInfoSettingsView(BaseAppView):
     def load_default_context(self):
         c = self._get_local_tmpl_context()
         return c
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_system', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
     def settings_system_info(self):
         _ = self.request.translate
         c = self.load_default_context()
 
         c.active = 'system'
         c.navlist = navigation_list(self.request)
 
         # TODO(marcink), figure out how to allow only selected users to do this
         c.allowed_to_snapshot = self._rhodecode_user.admin
 
         snapshot = str2bool(self.request.params.get('snapshot'))
 
         c.rhodecode_update_url = UpdateModel().get_update_url()
         server_info = system_info.get_system_info(self.request.environ)
 
         for key, val in server_info.items():
             setattr(c, key, val)
 
         def val(name, subkey='human_value'):
             return server_info[name][subkey]
 
         def state(name):
             return server_info[name]['state']
 
         def val2(name):
             val = server_info[name]['human_value']
             state = server_info[name]['state']
             return val, state
 
         update_info_msg = _('Note: please make sure this server can '
                             'access `${url}` for the update link to work',
                             mapping=dict(url=c.rhodecode_update_url))
         version = UpdateModel().get_stored_version()
         is_outdated = UpdateModel().is_outdated(
             rhodecode.__version__, version)
         update_state = {
             'type': 'warning',
             'message': 'New version available: {}'.format(version)
             } \
             if is_outdated else {}
         c.data_items = [
             # update info
             (_('Update info'), h.literal(
                 '<span class="link" id="check_for_update" >%s.</span>' % (
                 _('Check for updates')) +
                 '<br/> <span >%s.</span>' % (update_info_msg)
             ), ''),
 
             # RhodeCode specific
             (_('RhodeCode Version'), val('rhodecode_app')['text'], state('rhodecode_app')),
             (_('Latest version'), version, update_state),
             (_('RhodeCode Server IP'), val('server')['server_ip'], state('server')),
             (_('RhodeCode Server ID'), val('server')['server_id'], state('server')),
             (_('RhodeCode Configuration'), val('rhodecode_config')['path'], state('rhodecode_config')),
             (_('RhodeCode Certificate'), val('rhodecode_config')['cert_path'], state('rhodecode_config')),
             (_('Workers'), val('rhodecode_config')['config']['server:main'].get('workers', '?'), state('rhodecode_config')),
             (_('Worker Type'), val('rhodecode_config')['config']['server:main'].get('worker_class', 'sync'), state('rhodecode_config')),
             ('', '', ''),  # spacer
 
             # Database
             (_('Database'), val('database')['url'], state('database')),
             (_('Database version'), val('database')['version'], state('database')),
             ('', '', ''),  # spacer
 
             # Platform/Python
             (_('Platform'), val('platform')['name'], state('platform')),
             (_('Platform UUID'), val('platform')['uuid'], state('platform')),
             (_('Python version'), val('python')['version'], state('python')),
             (_('Python path'), val('python')['executable'], state('python')),
             ('', '', ''),  # spacer
 
             # Systems stats
             (_('CPU'), val('cpu')['text'], state('cpu')),
             (_('Load'), val('load')['text'], state('load')),
             (_('Memory'), val('memory')['text'], state('memory')),
             (_('Uptime'), val('uptime')['text'], state('uptime')),
             ('', '', ''),  # spacer
 
+            # ulimit
+            (_('Ulimit'), val('ulimit')['text'], state('ulimit')),
+
             # Repo storage
             (_('Storage location'), val('storage')['path'], state('storage')),
             (_('Storage info'), val('storage')['text'], state('storage')),
             (_('Storage inodes'), val('storage_inodes')['text'], state('storage_inodes')),
 
             (_('Gist storage location'), val('storage_gist')['path'], state('storage_gist')),
             (_('Gist storage info'), val('storage_gist')['text'], state('storage_gist')),
 
             (_('Archive cache storage location'), val('storage_archive')['path'], state('storage_archive')),
             (_('Archive cache info'), val('storage_archive')['text'], state('storage_archive')),
 
             (_('Temp storage location'), val('storage_temp')['path'], state('storage_temp')),
             (_('Temp storage info'), val('storage_temp')['text'], state('storage_temp')),
 
             (_('Search info'), val('search')['text'], state('search')),
             (_('Search location'), val('search')['location'], state('search')),
             ('', '', ''),  # spacer
 
             # VCS specific
             (_('VCS Backends'), val('vcs_backends'), state('vcs_backends')),
             (_('VCS Server'), val('vcs_server')['text'], state('vcs_server')),
             (_('GIT'), val('git'), state('git')),
             (_('HG'), val('hg'), state('hg')),
             (_('SVN'), val('svn'), state('svn')),
 
         ]
 
         if snapshot:
             if c.allowed_to_snapshot:
                 c.data_items.pop(0)  # remove server info
                 self.request.override_renderer = 'admin/settings/settings_system_snapshot.mako'
             else:
                 h.flash('You are not allowed to do this', category='warning')
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @view_config(
         route_name='admin_settings_system_update', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings_system_update.mako')
     def settings_system_info_check_update(self):
         _ = self.request.translate
         c = self.load_default_context()
 
         update_url = UpdateModel().get_update_url()
 
         _err = lambda s: '<div style="color:#ff8888; padding:4px 0px">{}</div>'.format(s)
         try:
             data = UpdateModel().get_update_data(update_url)
         except urllib2.URLError as e:
             log.exception("Exception contacting upgrade server")
             self.request.override_renderer = 'string'
             return _err('Failed to contact upgrade server: %r' % e)
         except ValueError as e:
             log.exception("Bad data sent from update server")
             self.request.override_renderer = 'string'
             return _err('Bad data sent from update server')
 
         latest = data['versions'][0]
 
         c.update_url = update_url
         c.latest_data = latest
         c.latest_ver = latest['version']
         c.cur_ver = rhodecode.__version__
         c.should_upgrade = False
 
         is_oudated = UpdateModel().is_outdated(c.cur_ver, c.latest_ver)
         if is_oudated:
             c.should_upgrade = True
         c.important_notices = latest['general']
         UpdateModel().store_version(latest['version'])
         return self._get_template_context(c)