Resolve error occurring during recursive group creation in API create-repo function

docs/changelog.rst

@@ -1,584 +1,586
 .. _changelog:
 
 =========
 Changelog
 =========
 
 
 1.3.4 (**2012-XX-XX**)
 ----------------------
 
 :status: in-progress
 :branch: beta
 
 news
 ++++
 
 - Whoosh logging is now controlled by the .ini files logging setup
 - added clone-url into edit form on /settings page
 - added help text into repo add/edit forms
 - created rcextensions module with additional mappings (ref #322) and
   post push/pull/create repo hooks callbacks
 
 fixes
 +++++
 
 - fixed #390 cache invalidation problems on repos inside group
 - fixed #385 clone by ID url was loosing proxy prefix in URL
 - fixed some unicode problems with waitress
 - fixed issue with escaping < and > in changeset commits
+- fixed error occurring during recursive group creation in API 
+  create_repo function
 
 1.3.3 (**2012-03-02**)
 ----------------------
 
 news
 ++++
 
 
 fixes
 +++++
 
 - fixed some python2.5 compatibility issues 
 - fixed issues with removed repos was accidentally added as groups, after
   full rescan of paths
 - fixes #376 Cannot edit user (using container auth)
 - fixes #378 Invalid image urls on changeset screen with proxy-prefix 
   configuration
 - fixed initial sorting of repos inside repo group
 - fixes issue when user tried to resubmit same permission into user/user_groups
 - bumped beaker version that fixes #375 leap error bug
 - fixed raw_changeset for git. It was generated with hg patch headers
 - fixed vcs issue with last_changeset for filenodes
 - fixed missing commit after hook delete
 - fixed #372 issues with git operation detection that caused a security issue 
   for git repos
 
 1.3.2 (**2012-02-28**)
 ----------------------
 
 news
 ++++
 
 
 fixes
 +++++
 
 - fixed git protocol issues with repos-groups
 - fixed git remote repos validator that prevented from cloning remote git repos
 - fixes #370 ending slashes fixes for repo and groups
 - fixes #368 improved git-protocol detection to handle other clients
 - fixes #366 When Setting Repository Group To Blank Repo Group Wont Be 
   Moved To Root
 - fixes #371 fixed issues with beaker/sqlalchemy and non-ascii cache keys 
 - fixed #373 missing cascade drop on user_group_to_perm table
 
 1.3.1 (**2012-02-27**)
 ----------------------
 
 news
 ++++
 
 
 fixes
 +++++
 
 - redirection loop occurs when remember-me wasn't checked during login
 - fixes issues with git blob history generation 
 - don't fetch branch for git in file history dropdown. Causes unneeded slowness
 
 1.3.0 (**2012-02-26**)
 ----------------------
 
 news
 ++++
 
 - code review, inspired by github code-comments 
 - #215 rst and markdown README files support
 - #252 Container-based and proxy pass-through authentication support
 - #44 branch browser. Filtering of changelog by branches
 - mercurial bookmarks support
 - new hover top menu, optimized to add maximum size for important views
 - configurable clone url template with possibility to specify  protocol like 
   ssh:// or http:// and also manually alter other parts of clone_url.
 - enabled largefiles extension by default
 - optimized summary file pages and saved a lot of unused space in them
 - #239 option to manually mark repository as fork
 - #320 mapping of commit authors to RhodeCode users
 - #304 hashes are displayed using monospace font    
 - diff configuration, toggle white lines and context lines
 - #307 configurable diffs, whitespace toggle, increasing context lines
 - sorting on branches, tags and bookmarks using YUI datatable
 - improved file filter on files page
 - implements #330 api method for listing nodes ar particular revision
 - #73 added linking issues in commit messages to chosen issue tracker url
   based on user defined regular expression
 - added linking of changesets in commit messages  
 - new compact changelog with expandable commit messages
 - firstname and lastname are optional in user creation
 - #348 added post-create repository hook
 - #212 global encoding settings is now configurable from .ini files 
 - #227 added repository groups permissions
 - markdown gets codehilite extensions
 - new API methods, delete_repositories, grante/revoke permissions for groups 
   and repos
   
     
 fixes
 +++++
 
 - rewrote dbsession management for atomic operations, and better error handling
 - fixed sorting of repo tables
 - #326 escape of special html entities in diffs
 - normalized user_name => username in api attributes
 - fixes #298 ldap created users with mixed case emails created conflicts 
   on saving a form
 - fixes issue when owner of a repo couldn't revoke permissions for users 
   and groups
 - fixes #271 rare JSON serialization problem with statistics
 - fixes #337 missing validation check for conflicting names of a group with a
   repositories group
 - #340 fixed session problem for mysql and celery tasks
 - fixed #331 RhodeCode mangles repository names if the a repository group 
   contains the "full path" to the repositories
 - #355 RhodeCode doesn't store encrypted LDAP passwords
 
 1.2.5 (**2012-01-28**)
 ----------------------
 
 news
 ++++
 
 fixes
 +++++
 
 - #340 Celery complains about MySQL server gone away, added session cleanup
   for celery tasks
 - #341 "scanning for repositories in None" log message during Rescan was missing
   a parameter
 - fixed creating archives with subrepos. Some hooks were triggered during that
   operation leading to crash.
 - fixed missing email in account page.
 - Reverted Mercurial to 2.0.1 for windows due to bug in Mercurial that makes
   forking on windows impossible 
 
 1.2.4 (**2012-01-19**)
 ----------------------
 
 news
 ++++
 
 - RhodeCode is bundled with mercurial series 2.0.X by default, with
   full support to largefiles extension. Enabled by default in new installations
 - #329 Ability to Add/Remove Groups to/from a Repository via AP
 - added requires.txt file with requirements
      
 fixes
 +++++
 
 - fixes db session issues with celery when emailing admins
 - #331 RhodeCode mangles repository names if the a repository group 
   contains the "full path" to the repositories
 - #298 Conflicting e-mail addresses for LDAP and RhodeCode users
 - DB session cleanup after hg protocol operations, fixes issues with
   `mysql has gone away` errors
 - #333 doc fixes for get_repo api function
 - #271 rare JSON serialization problem with statistics enabled
 - #337 Fixes issues with validation of repository name conflicting with 
   a group name. A proper message is now displayed.
 - #292 made ldap_dn in user edit readonly, to get rid of confusion that field
   doesn't work   
 - #316 fixes issues with web description in hgrc files 
 
 1.2.3 (**2011-11-02**)
 ----------------------
 
 news
 ++++
 
 - added option to manage repos group for non admin users
 - added following API methods for get_users, create_user, get_users_groups, 
   get_users_group, create_users_group, add_user_to_users_groups, get_repos, 
   get_repo, create_repo, add_user_to_repo
 - implements #237 added password confirmation for my account 
   and admin edit user.
 - implements #291 email notification for global events are now sent to all
   administrator users, and global config email.
      
 fixes
 +++++
 
 - added option for passing auth method for smtp mailer
 - #276 issue with adding a single user with id>10 to usergroups
 - #277 fixes windows LDAP settings in which missing values breaks the ldap auth 
 - #288 fixes managing of repos in a group for non admin user
 
 1.2.2 (**2011-10-17**)
 ----------------------
 
 news
 ++++
 
 - #226 repo groups are available by path instead of numerical id
  
 fixes
 +++++
 
 - #259 Groups with the same name but with different parent group
 - #260 Put repo in group, then move group to another group -> repo becomes unavailable
 - #258 RhodeCode 1.2 assumes egg folder is writable (lockfiles problems)
 - #265 ldap save fails sometimes on converting attributes to booleans, 
   added getter and setter into model that will prevent from this on db model level
 - fixed problems with timestamps issues #251 and #213
 - fixes #266 RhodeCode allows to create repo with the same name and in 
   the same parent as group
 - fixes #245 Rescan of the repositories on Windows
 - fixes #248 cannot edit repos inside a group on windows
 - fixes #219 forking problems on windows
 
 1.2.1 (**2011-10-08**)
 ----------------------
 
 news
 ++++
 
 
 fixes
 +++++
 
 - fixed problems with basic auth and push problems 
 - gui fixes
 - fixed logger
 
 1.2.0 (**2011-10-07**)
 ----------------------
 
 news
 ++++
 
 - implemented #47 repository groups
 - implemented #89 Can setup google analytics code from settings menu
 - implemented #91 added nicer looking archive urls with more download options
   like tags, branches
 - implemented #44 into file browsing, and added follow branch option
 - implemented #84 downloads can be enabled/disabled for each repository
 - anonymous repository can be cloned without having to pass default:default
   into clone url
 - fixed #90 whoosh indexer can index chooses repositories passed in command 
   line
 - extended journal with day aggregates and paging
 - implemented #107 source code lines highlight ranges
 - implemented #93 customizable changelog on combined revision ranges - 
   equivalent of githubs compare view 
 - implemented #108 extended and more powerful LDAP configuration
 - implemented #56 users groups
 - major code rewrites optimized codes for speed and memory usage
 - raw and diff downloads are now in git format
 - setup command checks for write access to given path
 - fixed many issues with international characters and unicode. It uses utf8
   decode with replace to provide less errors even with non utf8 encoded strings
 - #125 added API KEY access to feeds
 - #109 Repository can be created from external Mercurial link (aka. remote 
   repository, and manually updated (via pull) from admin panel
 - beta git support - push/pull server + basic view for git repos
 - added followers page and forks page
 - server side file creation (with binary file upload interface) 
   and edition with commits powered by codemirror 
 - #111 file browser file finder, quick lookup files on whole file tree 
 - added quick login sliding menu into main page
 - changelog uses lazy loading of affected files details, in some scenarios 
   this can improve speed of changelog page dramatically especially for 
   larger repositories.
 - implements #214 added support for downloading subrepos in download menu.
 - Added basic API for direct operations on rhodecode via JSON
 - Implemented advanced hook management
 
 fixes
 +++++
 
 - fixed file browser bug, when switching into given form revision the url was 
   not changing
 - fixed propagation to error controller on simplehg and simplegit middlewares
 - fixed error when trying to make a download on empty repository
 - fixed problem with '[' chars in commit messages in journal
 - fixed #99 Unicode errors, on file node paths with non utf-8 characters
 - journal fork fixes
 - removed issue with space inside renamed repository after deletion
 - fixed strange issue on formencode imports
 - fixed #126 Deleting repository on Windows, rename used incompatible chars. 
 - #150 fixes for errors on repositories mapped in db but corrupted in 
   filesystem
 - fixed problem with ascendant characters in realm #181
 - fixed problem with sqlite file based database connection pool
 - whoosh indexer and code stats share the same dynamic extensions map
 - fixes #188 - relationship delete of repo_to_perm entry on user removal
 - fixes issue #189 Trending source files shows "show more" when no more exist
 - fixes issue #197 Relative paths for pidlocks
 - fixes issue #198 password will require only 3 chars now for login form
 - fixes issue #199 wrong redirection for non admin users after creating a repository
 - fixes issues #202, bad db constraint made impossible to attach same group 
   more than one time. Affects only mysql/postgres
 - fixes #218 os.kill patch for windows was missing sig param
 - improved rendering of dag (they are not trimmed anymore when number of 
   heads exceeds 5)
     
 1.1.8 (**2011-04-12**)
 ----------------------
 
 news
 ++++
 
 - improved windows support
 
 fixes
 +++++
 
 - fixed #140 freeze of python dateutil library, since new version is python2.x
   incompatible
 - setup-app will check for write permission in given path
 - cleaned up license info issue #149
 - fixes for issues #137,#116 and problems with unicode and accented characters.
 - fixes crashes on gravatar, when passed in email as unicode
 - fixed tooltip flickering problems
 - fixed came_from redirection on windows
 - fixed logging modules, and sql formatters
 - windows fixes for os.kill issue #133
 - fixes path splitting for windows issues #148
 - fixed issue #143 wrong import on migration to 1.1.X
 - fixed problems with displaying binary files, thanks to Thomas Waldmann
 - removed name from archive files since it's breaking ui for long repo names
 - fixed issue with archive headers sent to browser, thanks to Thomas Waldmann
 - fixed compatibility for 1024px displays, and larger dpi settings, thanks to 
   Thomas Waldmann
 - fixed issue #166 summary pager was skipping 10 revisions on second page
 
 
 1.1.7 (**2011-03-23**)
 ----------------------
 
 news
 ++++
 
 fixes
 +++++
 
 - fixed (again) #136 installation support for FreeBSD
 
 
 1.1.6 (**2011-03-21**)
 ----------------------
 
 news
 ++++
 
 fixes
 +++++
 
 - fixed #136 installation support for FreeBSD
 - RhodeCode will check for python version during installation
 
 1.1.5 (**2011-03-17**)
 ----------------------
 
 news
 ++++
 
 - basic windows support, by exchanging pybcrypt into sha256 for windows only
   highly inspired by idea of mantis406
 
 fixes
 +++++
 
 - fixed sorting by author in main page
 - fixed crashes with diffs on binary files
 - fixed #131 problem with boolean values for LDAP
 - fixed #122 mysql problems thanks to striker69 
 - fixed problem with errors on calling raw/raw_files/annotate functions 
   with unknown revisions
 - fixed returned rawfiles attachment names with international character
 - cleaned out docs, big thanks to Jason Harris
 
 1.1.4 (**2011-02-19**)
 ----------------------
 
 news
 ++++
 
 fixes
 +++++
 
 - fixed formencode import problem on settings page, that caused server crash
   when that page was accessed as first after server start
 - journal fixes
 - fixed option to access repository just by entering http://server/<repo_name> 
 
 1.1.3 (**2011-02-16**)
 ----------------------
 
 news
 ++++
 
 - implemented #102 allowing the '.' character in username
 - added option to access repository just by entering http://server/<repo_name>
 - celery task ignores result for better performance
 
 fixes
 +++++
 
 - fixed ehlo command and non auth mail servers on smtp_lib. Thanks to 
   apollo13 and Johan Walles
 - small fixes in journal
 - fixed problems with getting setting for celery from .ini files
 - registration, password reset and login boxes share the same title as main 
   application now
 - fixed #113: to high permissions to fork repository
 - fixed problem with '[' chars in commit messages in journal
 - removed issue with space inside renamed repository after deletion
 - db transaction fixes when filesystem repository creation failed
 - fixed #106 relation issues on databases different than sqlite
 - fixed static files paths links to use of url() method
 
 1.1.2 (**2011-01-12**)
 ----------------------
 
 news
 ++++
 
 
 fixes
 +++++
 
 - fixes #98 protection against float division of percentage stats
 - fixed graph bug
 - forced webhelpers version since it was making troubles during installation 
 
 1.1.1 (**2011-01-06**)
 ----------------------
  
 news
 ++++
 
 - added force https option into ini files for easier https usage (no need to
   set server headers with this options)
 - small css updates
 
 fixes
 +++++
 
 - fixed #96 redirect loop on files view on repositories without changesets
 - fixed #97 unicode string passed into server header in special cases (mod_wsgi)
   and server crashed with errors
 - fixed large tooltips problems on main page
 - fixed #92 whoosh indexer is more error proof
 
 1.1.0 (**2010-12-18**)
 ----------------------
 
 news
 ++++
 
 - rewrite of internals for vcs >=0.1.10
 - uses mercurial 1.7 with dotencode disabled for maintaining compatibility 
   with older clients
 - anonymous access, authentication via ldap
 - performance upgrade for cached repos list - each repository has its own 
   cache that's invalidated when needed.
 - performance upgrades on repositories with large amount of commits (20K+)
 - main page quick filter for filtering repositories
 - user dashboards with ability to follow chosen repositories actions
 - sends email to admin on new user registration
 - added cache/statistics reset options into repository settings
 - more detailed action logger (based on hooks) with pushed changesets lists
   and options to disable those hooks from admin panel
 - introduced new enhanced changelog for merges that shows more accurate results
 - new improved and faster code stats (based on pygments lexers mapping tables, 
   showing up to 10 trending sources for each repository. Additionally stats
   can be disabled in repository settings.
 - gui optimizations, fixed application width to 1024px
 - added cut off (for large files/changesets) limit into config files
 - whoosh, celeryd, upgrade moved to paster command
 - other than sqlite database backends can be used
 
 fixes
 +++++
 
 - fixes #61 forked repo was showing only after cache expired
 - fixes #76 no confirmation on user deletes
 - fixes #66 Name field misspelled
 - fixes #72 block user removal when he owns repositories
 - fixes #69 added password confirmation fields
 - fixes #87 RhodeCode crashes occasionally on updating repository owner
 - fixes #82 broken annotations on files with more than 1 blank line at the end
 - a lot of fixes and tweaks for file browser
 - fixed detached session issues
 - fixed when user had no repos he would see all repos listed in my account
 - fixed ui() instance bug when global hgrc settings was loaded for server 
   instance and all hgrc options were merged with our db ui() object
 - numerous small bugfixes
  
 (special thanks for TkSoh for detailed feedback)
 
 
 1.0.2 (**2010-11-12**)
 ----------------------
 
 news
 ++++
 
 - tested under python2.7
 - bumped sqlalchemy and celery versions
 
 fixes
 +++++
 
 - fixed #59 missing graph.js
 - fixed repo_size crash when repository had broken symlinks
 - fixed python2.5 crashes.
 
 
 1.0.1 (**2010-11-10**)
 ----------------------
 
 news
 ++++
 
 - small css updated
 
 fixes
 +++++
 
 - fixed #53 python2.5 incompatible enumerate calls
 - fixed #52 disable mercurial extension for web
 - fixed #51 deleting repositories don't delete it's dependent objects
 
 
 1.0.0 (**2010-11-02**)
 ----------------------
 
 - security bugfix simplehg wasn't checking for permissions on commands
   other than pull or push.
 - fixed doubled messages after push or pull in admin journal
 - templating and css corrections, fixed repo switcher on chrome, updated titles
 - admin menu accessible from options menu on repository view
 - permissions cached queries
 
 1.0.0rc4  (**2010-10-12**)
 --------------------------
 
 - fixed python2.5 missing simplejson imports (thanks to Jens Bäckman)
 - removed cache_manager settings from sqlalchemy meta
 - added sqlalchemy cache settings to ini files
 - validated password length and added second try of failure on paster setup-app
 - fixed setup database destroy prompt even when there was no db
 
 
 1.0.0rc3 (**2010-10-11**)
 -------------------------
 
 - fixed i18n during installation.
 
 1.0.0rc2 (**2010-10-11**)
 -------------------------
 
 - Disabled dirsize in file browser, it's causing nasty bug when dir renames 
   occure. After vcs is fixed it'll be put back again.
 - templating/css rewrites, optimized css.
  No newline at end of file

rhodecode/controllers/api/api.py

@@ -1,660 +1,656
 # -*- coding: utf-8 -*-
 """
     rhodecode.controllers.api
     ~~~~~~~~~~~~~~~~~~~~~~~~~
 
     API controller for RhodeCode
 
     :created_on: Aug 20, 2011
     :author: marcink
     :copyright: (C) 2011-2012 Marcin Kuzminski <[e-mail]>
     :license: GPLv3, see COPYING for more details.
 """
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
 # as published by the Free Software Foundation; version 2
 # of the License or (at your opinion) any later version of the license.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
 # MA  02110-1301, USA.
 
 import traceback
 import logging
 
 from rhodecode.controllers.api import JSONRPCController, JSONRPCError
 from rhodecode.lib.auth import HasPermissionAllDecorator, \
     HasPermissionAnyDecorator, PasswordGenerator
 
 from rhodecode.model.meta import Session
 from rhodecode.model.scm import ScmModel
 from rhodecode.model.db import User, UsersGroup, RepoGroup, Repository
 from rhodecode.model.repo import RepoModel
 from rhodecode.model.user import UserModel
 from rhodecode.model.users_group import UsersGroupModel
 from rhodecode.model.repos_group import ReposGroupModel
+from rhodecode.lib.utils import map_groups
 
 
 log = logging.getLogger(__name__)
 
 
 class ApiController(JSONRPCController):
     """
     API Controller
 
 
     Each method needs to have USER as argument this is then based on given
     API_KEY propagated as instance of user object
 
     Preferably this should be first argument also
 
 
     Each function should also **raise** JSONRPCError for any
     errors that happens
 
     """
 
     @HasPermissionAllDecorator('hg.admin')
     def pull(self, apiuser, repo_name):
         """
         Dispatch pull action on given repo
 
 
         :param user:
         :param repo_name:
         """
 
         if Repository.is_valid(repo_name) is False:
             raise JSONRPCError('Unknown repo "%s"' % repo_name)
 
         try:
             ScmModel().pull_changes(repo_name, self.rhodecode_user.username)
             return 'Pulled from %s' % repo_name
         except Exception:
             raise JSONRPCError('Unable to pull changes from "%s"' % repo_name)
 
     @HasPermissionAllDecorator('hg.admin')
     def get_user(self, apiuser, userid):
         """"
         Get a user by username
 
         :param apiuser:
         :param username:
         """
 
         user = UserModel().get_user(userid)
         if user is None:
             return user
 
         return dict(
             id=user.user_id,
             username=user.username,
             firstname=user.name,
             lastname=user.lastname,
             email=user.email,
             active=user.active,
             admin=user.admin,
             ldap_dn=user.ldap_dn
         )
 
     @HasPermissionAllDecorator('hg.admin')
     def get_users(self, apiuser):
         """"
         Get all users
 
         :param apiuser:
         """
 
         result = []
         for user in User.getAll():
             result.append(
                 dict(
                     id=user.user_id,
                     username=user.username,
                     firstname=user.name,
                     lastname=user.lastname,
                     email=user.email,
                     active=user.active,
                     admin=user.admin,
                     ldap_dn=user.ldap_dn
                 )
             )
         return result
 
     @HasPermissionAllDecorator('hg.admin')
     def create_user(self, apiuser, username, email, password, firstname=None,
                     lastname=None, active=True, admin=False, ldap_dn=None):
         """
         Create new user
 
         :param apiuser:
         :param username:
         :param password:
         :param email:
         :param name:
         :param lastname:
         :param active:
         :param admin:
         :param ldap_dn:
         """
         if User.get_by_username(username):
             raise JSONRPCError("user %s already exist" % username)
 
         if User.get_by_email(email, case_insensitive=True):
             raise JSONRPCError("email %s already exist" % email)
 
         if ldap_dn:
             # generate temporary password if ldap_dn
             password = PasswordGenerator().gen_password(length=8)
 
         try:
             usr = UserModel().create_or_update(
                 username, password, email, firstname,
                 lastname, active, admin, ldap_dn
             )
             Session.commit()
             return dict(
                 id=usr.user_id,
                 msg='created new user %s' % username
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError('failed to create user %s' % username)
 
     @HasPermissionAllDecorator('hg.admin')
     def update_user(self, apiuser, userid, username, password, email,
                     firstname, lastname, active, admin, ldap_dn):
         """
         Updates given user
 
         :param apiuser:
         :param username:
         :param password:
         :param email:
         :param name:
         :param lastname:
         :param active:
         :param admin:
         :param ldap_dn:
         """
         if not UserModel().get_user(userid):
             raise JSONRPCError("user %s does not exist" % username)
 
         try:
             usr = UserModel().create_or_update(
                 username, password, email, firstname,
                 lastname, active, admin, ldap_dn
             )
             Session.commit()
             return dict(
                 id=usr.user_id,
                 msg='updated user %s' % username
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError('failed to update user %s' % username)
 
     @HasPermissionAllDecorator('hg.admin')
     def get_users_group(self, apiuser, group_name):
         """"
         Get users group by name
 
         :param apiuser:
         :param group_name:
         """
 
         users_group = UsersGroup.get_by_group_name(group_name)
         if not users_group:
             return None
 
         members = []
         for user in users_group.members:
             user = user.user
             members.append(dict(id=user.user_id,
                             username=user.username,
                             firstname=user.name,
                             lastname=user.lastname,
                             email=user.email,
                             active=user.active,
                             admin=user.admin,
                             ldap=user.ldap_dn))
 
         return dict(id=users_group.users_group_id,
                     group_name=users_group.users_group_name,
                     active=users_group.users_group_active,
                     members=members)
 
     @HasPermissionAllDecorator('hg.admin')
     def get_users_groups(self, apiuser):
         """"
         Get all users groups
 
         :param apiuser:
         """
 
         result = []
         for users_group in UsersGroup.getAll():
             members = []
             for user in users_group.members:
                 user = user.user
                 members.append(dict(id=user.user_id,
                                 username=user.username,
                                 firstname=user.name,
                                 lastname=user.lastname,
                                 email=user.email,
                                 active=user.active,
                                 admin=user.admin,
                                 ldap=user.ldap_dn))
 
             result.append(dict(id=users_group.users_group_id,
                                 group_name=users_group.users_group_name,
                                 active=users_group.users_group_active,
                                 members=members))
         return result
 
     @HasPermissionAllDecorator('hg.admin')
     def create_users_group(self, apiuser, group_name, active=True):
         """
         Creates an new usergroup
 
         :param group_name:
         :param active:
         """
 
         if self.get_users_group(apiuser, group_name):
             raise JSONRPCError("users group %s already exist" % group_name)
 
         try:
             ug = UsersGroupModel().create(name=group_name, active=active)
             Session.commit()
             return dict(id=ug.users_group_id,
                         msg='created new users group %s' % group_name)
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError('failed to create group %s' % group_name)
 
     @HasPermissionAllDecorator('hg.admin')
     def add_user_to_users_group(self, apiuser, group_name, username):
         """"
         Add a user to a group
 
         :param apiuser:
         :param group_name:
         :param username:
         """
 
         try:
             users_group = UsersGroup.get_by_group_name(group_name)
             if not users_group:
                 raise JSONRPCError('unknown users group %s' % group_name)
 
             user = User.get_by_username(username)
             if user is None:
                 raise JSONRPCError('unknown user %s' % username)
 
             ugm = UsersGroupModel().add_user_to_group(users_group, user)
             success = True if ugm != True else False
             msg = 'added member %s to users group %s' % (username, group_name)
             msg = msg if success else 'User is already in that group'
             Session.commit()
 
             return dict(
                 id=ugm.users_group_member_id if ugm != True else None,
                 success=success,
                 msg=msg
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError('failed to add users group member')
 
     @HasPermissionAllDecorator('hg.admin')
     def remove_user_from_users_group(self, apiuser, group_name, username):
         """
         Remove user from a group
 
         :param apiuser
         :param group_name
         :param username
         """
 
         try:
             users_group = UsersGroup.get_by_group_name(group_name)
             if not users_group:
                 raise JSONRPCError('unknown users group %s' % group_name)
 
             user = User.get_by_username(username)
             if user is None:
                 raise JSONRPCError('unknown user %s' % username)
 
             success = UsersGroupModel().remove_user_from_group(users_group, user)
             msg = 'removed member %s from users group %s' % (username, group_name)
             msg = msg if success else "User wasn't in group"
             Session.commit()
             return dict(success=success, msg=msg)
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError('failed to remove user from group')
 
     @HasPermissionAnyDecorator('hg.admin')
     def get_repo(self, apiuser, repoid):
         """"
         Get repository by name
 
         :param apiuser:
         :param repo_name:
         """
 
         repo = RepoModel().get_repo(repoid)
         if repo is None:
             raise JSONRPCError('unknown repository %s' % repo)
 
         members = []
         for user in repo.repo_to_perm:
             perm = user.permission.permission_name
             user = user.user
             members.append(
                 dict(
                     type_="user",
                     id=user.user_id,
                     username=user.username,
                     firstname=user.name,
                     lastname=user.lastname,
                     email=user.email,
                     active=user.active,
                     admin=user.admin,
                     ldap=user.ldap_dn,
                     permission=perm
                 )
             )
         for users_group in repo.users_group_to_perm:
             perm = users_group.permission.permission_name
             users_group = users_group.users_group
             members.append(
                 dict(
                     type_="users_group",
                     id=users_group.users_group_id,
                     name=users_group.users_group_name,
                     active=users_group.users_group_active,
                     permission=perm
                 )
             )
 
         return dict(
             id=repo.repo_id,
             repo_name=repo.repo_name,
             type=repo.repo_type,
             description=repo.description,
             members=members
         )
 
     @HasPermissionAnyDecorator('hg.admin')
     def get_repos(self, apiuser):
         """"
         Get all repositories
 
         :param apiuser:
         """
 
         result = []
         for repository in Repository.getAll():
             result.append(
                 dict(
                     id=repository.repo_id,
                     repo_name=repository.repo_name,
                     type=repository.repo_type,
                     description=repository.description
                 )
             )
         return result
 
     @HasPermissionAnyDecorator('hg.admin')
     def get_repo_nodes(self, apiuser, repo_name, revision, root_path,
                        ret_type='all'):
         """
         returns a list of nodes and it's children
         for a given path at given revision. It's possible to specify ret_type
         to show only files or dirs
 
         :param apiuser:
         :param repo_name: name of repository
         :param revision: revision for which listing should be done
         :param root_path: path from which start displaying
         :param ret_type: return type 'all|files|dirs' nodes
         """
         try:
             _d, _f = ScmModel().get_nodes(repo_name, revision, root_path,
                                           flat=False)
             _map = {
                 'all': _d + _f,
                 'files': _f,
                 'dirs': _d,
             }
             return _map[ret_type]
         except KeyError:
             raise JSONRPCError('ret_type must be one of %s' % _map.keys())
         except Exception, e:
             raise JSONRPCError(e)
 
     @HasPermissionAnyDecorator('hg.admin', 'hg.create.repository')
     def create_repo(self, apiuser, repo_name, owner_name, description='',
                     repo_type='hg', private=False, clone_uri=None):
         """
         Create repository, if clone_url is given it makes a remote clone
 
         :param apiuser:
         :param repo_name:
         :param owner_name:
         :param description:
         :param repo_type:
         :param private:
         :param clone_uri:
         """
 
         try:
             owner = User.get_by_username(owner_name)
             if owner is None:
                 raise JSONRPCError('unknown user %s' % owner_name)
 
             if Repository.get_by_repo_name(repo_name):
                 raise JSONRPCError("repo %s already exist" % repo_name)
 
-            groups = repo_name.split('/')
+            groups = repo_name.split(Repository.url_sep())
             real_name = groups[-1]
-            groups = groups[:-1]
-            parent_id = None
-            for g in groups:
-                group = RepoGroup.get_by_group_name(g)
-                if not group:
-                    group = ReposGroupModel().create(g, '', parent_id)
-                parent_id = group.group_id
+            # create structure of groups
+            group = map_groups(repo_name)
 
             repo = RepoModel().create(
                 dict(
                     repo_name=real_name,
                     repo_name_full=repo_name,
                     description=description,
                     private=private,
                     repo_type=repo_type,
-                    repo_group=parent_id,
+                    repo_group=group.group_id if group else None,
                     clone_uri=clone_uri
                 ),
                 owner
             )
             Session.commit()
 
             return dict(
                 id=repo.repo_id,
                 msg="Created new repository %s" % repo.repo_name
             )
 
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError('failed to create repository %s' % repo_name)
 
     @HasPermissionAnyDecorator('hg.admin')
     def delete_repo(self, apiuser, repo_name):
         """
         Deletes a given repository
 
         :param repo_name:
         """
         if not Repository.get_by_repo_name(repo_name):
             raise JSONRPCError("repo %s does not exist" % repo_name)
         try:
             RepoModel().delete(repo_name)
             Session.commit()
             return dict(
                 msg='Deleted repository %s' % repo_name
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError('failed to delete repository %s' % repo_name)
 
     @HasPermissionAnyDecorator('hg.admin')
     def grant_user_permission(self, apiuser, repo_name, username, perm):
         """
         Grant permission for user on given repository, or update existing one
         if found
 
         :param repo_name:
         :param username:
         :param perm:
         """
 
         try:
             repo = Repository.get_by_repo_name(repo_name)
             if repo is None:
                 raise JSONRPCError('unknown repository %s' % repo)
 
             user = User.get_by_username(username)
             if user is None:
                 raise JSONRPCError('unknown user %s' % username)
 
             RepoModel().grant_user_permission(repo=repo, user=user, perm=perm)
 
             Session.commit()
             return dict(
                 msg='Granted perm: %s for user: %s in repo: %s' % (
                     perm, username, repo_name
                 )
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError(
                 'failed to edit permission %(repo)s for %(user)s' % dict(
                     user=username, repo=repo_name
                 )
             )
 
     @HasPermissionAnyDecorator('hg.admin')
     def revoke_user_permission(self, apiuser, repo_name, username):
         """
         Revoke permission for user on given repository
 
         :param repo_name:
         :param username:
         """
 
         try:
             repo = Repository.get_by_repo_name(repo_name)
             if repo is None:
                 raise JSONRPCError('unknown repository %s' % repo)
 
             user = User.get_by_username(username)
             if user is None:
                 raise JSONRPCError('unknown user %s' % username)
 
             RepoModel().revoke_user_permission(repo=repo_name, user=username)
 
             Session.commit()
             return dict(
                 msg='Revoked perm for user: %s in repo: %s' % (
                     username, repo_name
                 )
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError(
                 'failed to edit permission %(repo)s for %(user)s' % dict(
                     user=username, repo=repo_name
                 )
             )
 
     @HasPermissionAnyDecorator('hg.admin')
     def grant_users_group_permission(self, apiuser, repo_name, group_name, perm):
         """
         Grant permission for users group on given repository, or update
         existing one if found
 
         :param repo_name:
         :param group_name:
         :param perm:
         """
 
         try:
             repo = Repository.get_by_repo_name(repo_name)
             if repo is None:
                 raise JSONRPCError('unknown repository %s' % repo)
 
             user_group = UsersGroup.get_by_group_name(group_name)
             if user_group is None:
                 raise JSONRPCError('unknown users group %s' % user_group)
 
             RepoModel().grant_users_group_permission(repo=repo_name,
                                                      group_name=group_name,
                                                      perm=perm)
 
             Session.commit()
             return dict(
                 msg='Granted perm: %s for group: %s in repo: %s' % (
                     perm, group_name, repo_name
                 )
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError(
                 'failed to edit permission %(repo)s for %(usersgr)s' % dict(
                     usersgr=group_name, repo=repo_name
                 )
             )
 
     @HasPermissionAnyDecorator('hg.admin')
     def revoke_users_group_permission(self, apiuser, repo_name, group_name):
         """
         Revoke permission for users group on given repository
 
         :param repo_name:
         :param group_name:
         """
 
         try:
             repo = Repository.get_by_repo_name(repo_name)
             if repo is None:
                 raise JSONRPCError('unknown repository %s' % repo)
 
             user_group = UsersGroup.get_by_group_name(group_name)
             if user_group is None:
                 raise JSONRPCError('unknown users group %s' % user_group)
 
             RepoModel().revoke_users_group_permission(repo=repo_name,
                                                       group_name=group_name)
 
             Session.commit()
             return dict(
                 msg='Revoked perm for group: %s in repo: %s' % (
                     group_name, repo_name
                 )
             )
         except Exception:
             log.error(traceback.format_exc())
             raise JSONRPCError(
                 'failed to edit permission %(repo)s for %(usersgr)s' % dict(
                     usersgr=group_name, repo=repo_name
                 )
             )

rhodecode/lib/utils.py

@@ -1,662 +1,659
 # -*- coding: utf-8 -*-
 """
     rhodecode.lib.utils
     ~~~~~~~~~~~~~~~~~~~
 
     Utilities library for RhodeCode
 
     :created_on: Apr 18, 2010
     :author: marcink
     :copyright: (C) 2010-2012 Marcin Kuzminski <[e-mail]>
     :license: GPLv3, see COPYING for more details.
 """
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
 import re
 import logging
 import datetime
 import traceback
 import paste
 import beaker
 import tarfile
 import shutil
 from os.path import abspath
 from os.path import dirname as dn, join as jn
 
 from paste.script.command import Command, BadCommand
 
 from mercurial import ui, config
 
 from webhelpers.text import collapse, remove_formatting, strip_tags
 
 from rhodecode.lib.vcs import get_backend
 from rhodecode.lib.vcs.backends.base import BaseChangeset
 from rhodecode.lib.vcs.utils.lazy import LazyProperty
 from rhodecode.lib.vcs.utils.helpers import get_scm
 from rhodecode.lib.vcs.exceptions import VCSError
 
 from rhodecode.lib.caching_query import FromCache
 
 from rhodecode.model import meta
 from rhodecode.model.db import Repository, User, RhodeCodeUi, \
     UserLog, RepoGroup, RhodeCodeSetting, UserRepoGroupToPerm
 from rhodecode.model.meta import Session
 from rhodecode.model.repos_group import ReposGroupModel
 from rhodecode.lib.utils2 import safe_str, safe_unicode
 from rhodecode.lib.vcs.utils.fakemod import create_module
 
 log = logging.getLogger(__name__)
 
 REMOVED_REPO_PAT = re.compile(r'rm__\d{8}_\d{6}_\d{6}__.*')
 
 
 def recursive_replace(str_, replace=' '):
     """
     Recursive replace of given sign to just one instance
 
     :param str_: given string
     :param replace: char to find and replace multiple instances
 
     Examples::
     >>> recursive_replace("Mighty---Mighty-Bo--sstones",'-')
     'Mighty-Mighty-Bo-sstones'
     """
 
     if str_.find(replace * 2) == -1:
         return str_
     else:
         str_ = str_.replace(replace * 2, replace)
         return recursive_replace(str_, replace)
 
 
 def repo_name_slug(value):
     """
     Return slug of name of repository
     This function is called on each creation/modification
     of repository to prevent bad names in repo
     """
 
     slug = remove_formatting(value)
     slug = strip_tags(slug)
 
     for c in """=[]\;'"<>,/~!@#$%^&*()+{}|: """:
         slug = slug.replace(c, '-')
     slug = recursive_replace(slug, '-')
     slug = collapse(slug, '-')
     return slug
 
 
 def get_repo_slug(request):
     _repo = request.environ['pylons.routes_dict'].get('repo_name')
     if _repo:
         _repo = _repo.rstrip('/')
     return _repo
 
 
 def get_repos_group_slug(request):
     _group = request.environ['pylons.routes_dict'].get('group_name')
     if _group:
         _group = _group.rstrip('/')
     return _group
 
 
 def action_logger(user, action, repo, ipaddr='', sa=None, commit=False):
     """
     Action logger for various actions made by users
 
     :param user: user that made this action, can be a unique username string or
         object containing user_id attribute
     :param action: action to log, should be on of predefined unique actions for
         easy translations
     :param repo: string name of repository or object containing repo_id,
         that action was made on
     :param ipaddr: optional ip address from what the action was made
     :param sa: optional sqlalchemy session
 
     """
 
     if not sa:
         sa = meta.Session
 
     try:
         if hasattr(user, 'user_id'):
             user_obj = user
         elif isinstance(user, basestring):
             user_obj = User.get_by_username(user)
         else:
             raise Exception('You have to provide user object or username')
 
         if hasattr(repo, 'repo_id'):
             repo_obj = Repository.get(repo.repo_id)
             repo_name = repo_obj.repo_name
         elif  isinstance(repo, basestring):
             repo_name = repo.lstrip('/')
             repo_obj = Repository.get_by_repo_name(repo_name)
         else:
             raise Exception('You have to provide repository to action logger')
 
         user_log = UserLog()
         user_log.user_id = user_obj.user_id
         user_log.action = action
 
         user_log.repository_id = repo_obj.repo_id
         user_log.repository_name = repo_name
 
         user_log.action_date = datetime.datetime.now()
         user_log.user_ip = ipaddr
         sa.add(user_log)
 
         log.info(
             'Adding user %s, action %s on %s' % (user_obj, action,
                                                  safe_unicode(repo))
         )
         if commit:
             sa.commit()
     except:
         log.error(traceback.format_exc())
         raise
 
 
 def get_repos(path, recursive=False):
     """
     Scans given path for repos and return (name,(type,path)) tuple
 
     :param path: path to scan for repositories
     :param recursive: recursive search and return names with subdirs in front
     """
 
     # remove ending slash for better results
     path = path.rstrip(os.sep)
 
     def _get_repos(p):
         if not os.access(p, os.W_OK):
             return
         for dirpath in os.listdir(p):
             if os.path.isfile(os.path.join(p, dirpath)):
                 continue
             cur_path = os.path.join(p, dirpath)
             try:
                 scm_info = get_scm(cur_path)
                 yield scm_info[1].split(path, 1)[-1].lstrip(os.sep), scm_info
             except VCSError:
                 if not recursive:
                     continue
                 #check if this dir containts other repos for recursive scan
                 rec_path = os.path.join(p, dirpath)
                 if os.path.isdir(rec_path):
                     for inner_scm in _get_repos(rec_path):
                         yield inner_scm
 
     return _get_repos(path)
 
 
 def is_valid_repo(repo_name, base_path):
     """
     Returns True if given path is a valid repository False otherwise
 
     :param repo_name:
     :param base_path:
 
     :return True: if given path is a valid repository
     """
     full_path = os.path.join(safe_str(base_path), safe_str(repo_name))
 
     try:
         get_scm(full_path)
         return True
     except VCSError:
         return False
 
 
 def is_valid_repos_group(repos_group_name, base_path):
     """
     Returns True if given path is a repos group False otherwise
 
     :param repo_name:
     :param base_path:
     """
     full_path = os.path.join(safe_str(base_path), safe_str(repos_group_name))
 
     # check if it's not a repo
     if is_valid_repo(repos_group_name, base_path):
         return False
 
     # check if it's a valid path
     if os.path.isdir(full_path):
         return True
 
     return False
 
 
 def ask_ok(prompt, retries=4, complaint='Yes or no, please!'):
     while True:
         ok = raw_input(prompt)
         if ok in ('y', 'ye', 'yes'):
             return True
         if ok in ('n', 'no', 'nop', 'nope'):
             return False
         retries = retries - 1
         if retries < 0:
             raise IOError
         print complaint
 
 #propagated from mercurial documentation
 ui_sections = ['alias', 'auth',
                 'decode/encode', 'defaults',
                 'diff', 'email',
                 'extensions', 'format',
                 'merge-patterns', 'merge-tools',
                 'hooks', 'http_proxy',
                 'smtp', 'patch',
                 'paths', 'profiling',
                 'server', 'trusted',
                 'ui', 'web', ]
 
 
 def make_ui(read_from='file', path=None, checkpaths=True):
     """
     A function that will read python rc files or database
     and make an mercurial ui object from read options
 
     :param path: path to mercurial config file
     :param checkpaths: check the path
     :param read_from: read from 'file' or 'db'
     """
 
     baseui = ui.ui()
 
     # clean the baseui object
     baseui._ocfg = config.config()
     baseui._ucfg = config.config()
     baseui._tcfg = config.config()
 
     if read_from == 'file':
         if not os.path.isfile(path):
             log.debug('hgrc file is not present at %s skipping...' % path)
             return False
         log.debug('reading hgrc from %s' % path)
         cfg = config.config()
         cfg.read(path)
         for section in ui_sections:
             for k, v in cfg.items(section):
                 log.debug('settings ui from file[%s]%s:%s' % (section, k, v))
                 baseui.setconfig(section, k, v)
 
     elif read_from == 'db':
         sa = meta.Session
         ret = sa.query(RhodeCodeUi)\
             .options(FromCache("sql_cache_short", "get_hg_ui_settings"))\
             .all()
 
         hg_ui = ret
         for ui_ in hg_ui:
             if ui_.ui_active:
                 log.debug('settings ui from db[%s]%s:%s', ui_.ui_section,
                           ui_.ui_key, ui_.ui_value)
                 baseui.setconfig(ui_.ui_section, ui_.ui_key, ui_.ui_value)
 
         meta.Session.remove()
     return baseui
 
 
 def set_rhodecode_config(config):
     """
     Updates pylons config with new settings from database
 
     :param config:
     """
     hgsettings = RhodeCodeSetting.get_app_settings()
 
     for k, v in hgsettings.items():
         config[k] = v
 
 
 def invalidate_cache(cache_key, *args):
     """
     Puts cache invalidation task into db for
     further global cache invalidation
     """
 
     from rhodecode.model.scm import ScmModel
 
     if cache_key.startswith('get_repo_cached_'):
         name = cache_key.split('get_repo_cached_')[-1]
         ScmModel().mark_for_invalidation(name)
 
 
 class EmptyChangeset(BaseChangeset):
     """
     An dummy empty changeset. It's possible to pass hash when creating
     an EmptyChangeset
     """
 
     def __init__(self, cs='0' * 40, repo=None, requested_revision=None,
                  alias=None):
         self._empty_cs = cs
         self.revision = -1
         self.message = ''
         self.author = ''
         self.date = ''
         self.repository = repo
         self.requested_revision = requested_revision
         self.alias = alias
 
     @LazyProperty
     def raw_id(self):
         """
         Returns raw string identifying this changeset, useful for web
         representation.
         """
 
         return self._empty_cs
 
     @LazyProperty
     def branch(self):
         return get_backend(self.alias).DEFAULT_BRANCH_NAME
 
     @LazyProperty
     def short_id(self):
         return self.raw_id[:12]
 
     def get_file_changeset(self, path):
         return self
 
     def get_file_content(self, path):
         return u''
 
     def get_file_size(self, path):
         return 0
 
 
-def map_groups(groups):
+def map_groups(path):
     """
-    Checks for groups existence, and creates groups structures.
-    It returns last group in structure
+    Given a full path to a repository, create all nested groups that this
+    repo is inside. This function creates parent-child relationships between
+    groups and creates default perms for all new groups.
 
-    :param groups: list of groups structure
+    :param paths: full path to repository
     """
     sa = meta.Session
-
+    groups = path.split(Repository.url_sep())
     parent = None
     group = None
 
     # last element is repo in nested groups structure
     groups = groups[:-1]
     rgm = ReposGroupModel(sa)
     for lvl, group_name in enumerate(groups):
         group_name = '/'.join(groups[:lvl] + [group_name])
         group = RepoGroup.get_by_group_name(group_name)
         desc = '%s group' % group_name
 
-#        # WTF that doesn't work !?
-#        if group is None:
-#            group = rgm.create(group_name, desc, parent, just_db=True)
-#            sa.commit()
-
         # skip folders that are now removed repos
         if REMOVED_REPO_PAT.match(group_name):
             break
 
         if group is None:
-            log.debug('creating group level: %s group_name: %s' % (lvl, group_name))
+            log.debug('creating group level: %s group_name: %s' % (lvl,
+                                                                   group_name))
             group = RepoGroup(group_name, parent)
             group.group_description = desc
             sa.add(group)
             rgm._create_default_perms(group)
-            sa.commit()
+            sa.flush()
         parent = group
     return group
 
 
 def repo2db_mapper(initial_repo_list, remove_obsolete=False):
     """
     maps all repos given in initial_repo_list, non existing repositories
     are created, if remove_obsolete is True it also check for db entries
     that are not in initial_repo_list and removes them.
 
     :param initial_repo_list: list of repositories found by scanning methods
     :param remove_obsolete: check for obsolete entries in database
     """
     from rhodecode.model.repo import RepoModel
     sa = meta.Session
     rm = RepoModel()
     user = sa.query(User).filter(User.admin == True).first()
     if user is None:
         raise Exception('Missing administrative account !')
     added = []
 
     for name, repo in initial_repo_list.items():
-        group = map_groups(name.split(Repository.url_sep()))
+        group = map_groups(name)
         if not rm.get_by_repo_name(name, cache=False):
             log.info('repository %s not found creating default' % name)
             added.append(name)
             form_data = {
              'repo_name': name,
              'repo_name_full': name,
              'repo_type': repo.alias,
              'description': repo.description \
                 if repo.description != 'unknown' else '%s repository' % name,
              'private': False,
              'group_id': getattr(group, 'group_id', None)
             }
             rm.create(form_data, user, just_db=True)
     sa.commit()
     removed = []
     if remove_obsolete:
         #remove from database those repositories that are not in the filesystem
         for repo in sa.query(Repository).all():
             if repo.repo_name not in initial_repo_list.keys():
                 removed.append(repo.repo_name)
                 sa.delete(repo)
                 sa.commit()
 
     return added, removed
 
 
 # set cache regions for beaker so celery can utilise it
 def add_cache(settings):
     cache_settings = {'regions': None}
     for key in settings.keys():
         for prefix in ['beaker.cache.', 'cache.']:
             if key.startswith(prefix):
                 name = key.split(prefix)[1].strip()
                 cache_settings[name] = settings[key].strip()
     if cache_settings['regions']:
         for region in cache_settings['regions'].split(','):
             region = region.strip()
             region_settings = {}
             for key, value in cache_settings.items():
                 if key.startswith(region):
                     region_settings[key.split('.')[1]] = value
             region_settings['expire'] = int(region_settings.get('expire',
                                                                 60))
             region_settings.setdefault('lock_dir',
                                        cache_settings.get('lock_dir'))
             region_settings.setdefault('data_dir',
                                        cache_settings.get('data_dir'))
 
             if 'type' not in region_settings:
                 region_settings['type'] = cache_settings.get('type',
                                                              'memory')
             beaker.cache.cache_regions[region] = region_settings
 
 
 def load_rcextensions(root_path):
     import rhodecode
     from rhodecode.config import conf
 
     path = os.path.join(root_path, 'rcextensions', '__init__.py')
     if os.path.isfile(path):
         rcext = create_module('rc', path)
         EXT = rhodecode.EXTENSIONS = rcext
         log.debug('Found rcextensions now loading %s...' % rcext)
 
         # Additional mappings that are not present in the pygments lexers
         conf.LANGUAGES_EXTENSIONS_MAP.update(getattr(EXT, 'EXTRA_MAPPINGS', {}))
 
         #OVERRIDE OUR EXTENSIONS FROM RC-EXTENSIONS (if present)
 
         if getattr(EXT, 'INDEX_EXTENSIONS', []) != []:
             log.debug('settings custom INDEX_EXTENSIONS')
             conf.INDEX_EXTENSIONS = getattr(EXT, 'INDEX_EXTENSIONS', [])
 
         #ADDITIONAL MAPPINGS
         log.debug('adding extra into INDEX_EXTENSIONS')
         conf.INDEX_EXTENSIONS.extend(getattr(EXT, 'EXTRA_INDEX_EXTENSIONS', []))
 
 
 #==============================================================================
 # TEST FUNCTIONS AND CREATORS
 #==============================================================================
 def create_test_index(repo_location, config, full_index):
     """
     Makes default test index
 
     :param config: test config
     :param full_index:
     """
 
     from rhodecode.lib.indexers.daemon import WhooshIndexingDaemon
     from rhodecode.lib.pidlock import DaemonLock, LockHeld
 
     repo_location = repo_location
 
     index_location = os.path.join(config['app_conf']['index_dir'])
     if not os.path.exists(index_location):
         os.makedirs(index_location)
 
     try:
         l = DaemonLock(file_=jn(dn(index_location), 'make_index.lock'))
         WhooshIndexingDaemon(index_location=index_location,
                              repo_location=repo_location)\
             .run(full_index=full_index)
         l.release()
     except LockHeld:
         pass
 
 
 def create_test_env(repos_test_path, config):
     """
     Makes a fresh database and
     install test repository into tmp dir
     """
     from rhodecode.lib.db_manage import DbManage
     from rhodecode.tests import HG_REPO, TESTS_TMP_PATH
 
     # PART ONE create db
     dbconf = config['sqlalchemy.db1.url']
     log.debug('making test db %s' % dbconf)
 
     # create test dir if it doesn't exist
     if not os.path.isdir(repos_test_path):
         log.debug('Creating testdir %s' % repos_test_path)
         os.makedirs(repos_test_path)
 
     dbmanage = DbManage(log_sql=True, dbconf=dbconf, root=config['here'],
                         tests=True)
     dbmanage.create_tables(override=True)
     dbmanage.create_settings(dbmanage.config_prompt(repos_test_path))
     dbmanage.create_default_user()
     dbmanage.admin_prompt()
     dbmanage.create_permissions()
     dbmanage.populate_default_permissions()
     Session.commit()
     # PART TWO make test repo
     log.debug('making test vcs repositories')
 
     idx_path = config['app_conf']['index_dir']
     data_path = config['app_conf']['cache_dir']
 
     #clean index and data
     if idx_path and os.path.exists(idx_path):
         log.debug('remove %s' % idx_path)
         shutil.rmtree(idx_path)
 
     if data_path and os.path.exists(data_path):
         log.debug('remove %s' % data_path)
         shutil.rmtree(data_path)
 
     #CREATE DEFAULT HG REPOSITORY
     cur_dir = dn(dn(abspath(__file__)))
     tar = tarfile.open(jn(cur_dir, 'tests', "vcs_test_hg.tar.gz"))
     tar.extractall(jn(TESTS_TMP_PATH, HG_REPO))
     tar.close()
 
 
 #==============================================================================
 # PASTER COMMANDS
 #==============================================================================
 class BasePasterCommand(Command):
     """
     Abstract Base Class for paster commands.
 
     The celery commands are somewhat aggressive about loading
     celery.conf, and since our module sets the `CELERY_LOADER`
     environment variable to our loader, we have to bootstrap a bit and
     make sure we've had a chance to load the pylons config off of the
     command line, otherwise everything fails.
     """
     min_args = 1
     min_args_error = "Please provide a paster config file as an argument."
     takes_config_file = 1
     requires_config_file = True
 
     def notify_msg(self, msg, log=False):
         """Make a notification to user, additionally if logger is passed
         it logs this action using given logger
 
         :param msg: message that will be printed to user
         :param log: logging instance, to use to additionally log this message
 
         """
         if log and isinstance(log, logging):
             log(msg)
 
     def run(self, args):
         """
         Overrides Command.run
 
         Checks for a config file argument and loads it.
         """
         if len(args) < self.min_args:
             raise BadCommand(
                 self.min_args_error % {'min_args': self.min_args,
                                        'actual_args': len(args)})
 
         # Decrement because we're going to lob off the first argument.
         # @@ This is hacky
         self.min_args -= 1
         self.bootstrap_config(args[0])
         self.update_parser()
         return super(BasePasterCommand, self).run(args[1:])
 
     def update_parser(self):
         """
         Abstract method.  Allows for the class's parser to be updated
         before the superclass's `run` method is called.  Necessary to
         allow options/arguments to be passed through to the underlying
         celery command.
         """
         raise NotImplementedError("Abstract Method.")
 
     def bootstrap_config(self, conf):
         """
         Loads the pylons configuration.
         """
         from pylons import config as pylonsconfig
 
         self.path_to_ini_file = os.path.realpath(conf)
         conf = paste.deploy.appconfig('config:' + self.path_to_ini_file)
         pylonsconfig.init_app(conf.global_conf, conf.local_conf)