merge with beta

docs/installation_win.rst

@@ -0,0 +1,244
+.. _installation_win:
+
+
+Step by step Installation for Windows
+=====================================
+
+
+RhodeCode step-by-step install Guide for Windows  
+
+Target OS: Windows XP SP3 English (Clean installation) 
++ All Windows Updates until 24-may-2012 
+
+Step1 - Install Visual Studio 2008 Express
+------------------------------------------
+
+ 
+Optional: You can also install MingW, but VS2008 installation is easier 
+
+Download "Visual C++ 2008 Express Edition with SP1" from: 
+http://www.microsoft.com/visualstudio/en-us/products/2008-editions/express 
+(if not found or relocated, google for "visual studio 2008 express" for 
+updated link) 
+
+You can also download full ISO file for offline installation, just 
+choose "All - Offline Install ISO image file" in the previous page and 
+choose "Visual C++ 2008 Express" when installing. 
+
+
+.. note::
+
+  Silverlight Runtime and SQL Server 2008 Express Edition are not 
+  required, you can uncheck them 
+
+
+Step2 - Install Python
+----------------------
+
+Install Python 2.x.y (x >= 5) x86 version (32bit). DO NOT USE A 3.x version.
+Download Python 2.x.y from: 
+http://www.python.org/download/ 
+
+Choose "Windows Installer" (32bit version) not "Windows X86-64 
+Installer". While writing this guide, the latest version was v2.7.3. 
+Remember the specific major and minor version installed, because it will 
+be needed in the next step. In this case, it is "2.7". 
+
+
+Step3 - Install Win32py extensions
+----------------------------------
+ 
+Download pywin32 from: 
+http://sourceforge.net/projects/pywin32/files/ 
+
+- Click on "pywin32" folder 
+- Click on the first folder (in this case, Build 217, maybe newer when you try) 
+- Choose the file ending with ".win32-py2.x.exe" -> x being the minor 
+  version of Python you installed (in this case, 7) 
+  When writing this guide, the file was: 
+  http://sourceforge.net/projects/pywin32/files/pywin32/Build%20217/pywin32-217.win32-py2.7.exe/download 
+
+
+Step4 - Python BIN
+------------------
+
+Add Python BIN folder to the path 
+
+You have to add the Python folder to the path, you can do it manually 
+(editing "PATH" environment variable) or using Windows Support Tools 
+that came preinstalled in Vista/7 and can be installed in Windows XP. 
+
+- Using support tools on WINDOWS XP: 
+  If you use Windows XP you can install them using Windows XP CD and 
+  navigating to \SUPPORT\TOOLS. There, execute Setup.EXE (not MSI). 
+  Afterwards, open a CMD and type::
+ 
+    SETX PATH "%PATH%;[your-python-path]" -M 
+
+  Close CMD (the path variable will be updated then) 
+
+- Using support tools on WINDOWS Vista/7: 
+
+  Open a CMD and type::
+
+    SETX PATH "%PATH%;[your-python-path]" /M 
+
+  Please substitute [your-python-path] with your Python installation path. 
+  Typically: C:\\Python27 
+
+
+Step5 - RhodeCode folder structure
+----------------------------------
+
+Create a RhodeCode folder structure 
+
+This is only a example to install RhodeCode, you can of course change 
+it. However, this guide will follow the proposed structure, so please 
+later adapt the paths if you change them. My recommendation is to use 
+folders with NO SPACES. But you can try if you are brave... 
+
+Create the following folder structure::
+
+  C:\RhodeCode 
+  C:\RhodeCode\Bin 
+  C:\RhodeCode\Env 
+  C:\RhodeCode\Repos 
+
+
+Step6 - Install virtualenv
+---------------------------
+
+Install Virtual Env for Python 
+
+Navigate to: http://www.virtualenv.org/en/latest/index.html#installation 
+Right click on "virtualenv.py" file and choose "Save link as...". 
+Download to C:\\RhodeCode (or whatever you want) 
+(the file is located at 
+https://raw.github.com/pypa/virtualenv/master/virtualenv.py) 
+
+Create a virtual Python environment in C:\\RhodeCode\\Env (or similar). To 
+do so, open a CMD (Python Path should be included in Step3), navigate 
+where you downloaded "virtualenv.py", and write:: 
+
+ python virtualenv.py C:\RhodeCode\Env 
+
+(--no-site-packages is now the default behaviour of virtualenv, no need 
+to include it) 
+
+
+Step7 - Install RhodeCode
+-------------------------
+
+Finally, install RhodeCode 
+
+Close previously opened command prompt/s, and open a Visual Studio 2008 
+Command Prompt (**IMPORTANT!!**). To do so, go to Start Menu, and then open 
+"Microsoft Visual C++ 2008 Express Edition" -> "Visual Studio Tools" -> 
+"Visual Studio 2008 Command Prompt" 
+
+In that CMD (loaded with VS2008 PATHs) type::
+ 
+  cd C:\RhodeCode\Env\Scripts (or similar) 
+  activate 
+
+The prompt will change into "(Env) C:\\RhodeCode\\Env\\Scripts" or similar 
+(depending of your folder structure). Then type:: 
+
+ pip install rhodecode 
+
+(long step, please wait until fully complete) 
+
+Some warnings will appear, don't worry as they are normal.
+
+
+Step8 - Configuring RhodeCode
+-----------------------------
+
+
+steps taken from http://packages.python.org/RhodeCode/setup.html 
+
+You have to use the same Visual Studio 2008 command prompt as Step7, so 
+if you closed it reopen it following the same commands (including the 
+"activate" one). When ready, just type::
+ 
+  cd C:\RhodeCode\Bin 
+  paster make-config RhodeCode production.ini 
+
+Then, you must edit production.ini to fit your needs (ip address, ip 
+port, mail settings, database, whatever). I recommend using NotePad++ 
+(free) or similar text editor, as it handles well the EndOfLine 
+character differences between Unix and Windows 
+(http://notepad-plus-plus.org/) 
+
+For the sake of simplicity lets run it with the default settings. After 
+your edits (if any), in the previous Command Prompt, type:: 
+ 
+ paster setup-rhodecode production.ini 
+
+(this time a NEW database will be installed, you must follow a different 
+step to later UPGRADE to a newer RhodeCode version) 
+
+The script will ask you for confirmation about creating a NEW database, 
+answer yes (y) 
+The script will ask you for repository path, answer C:\\RhodeCode\\Repos 
+(or similar) 
+The script will ask you for admin username and password, answer "admin" 
++ "123456" (or whatever you want) 
+The script will ask you for admin mail, answer "[e-mail]" (or 
+whatever you want) 
+
+If you make some mistake and the script does not end, don't worry, start 
+it again. 
+
+
+Step9 - Running RhodeCode
+-------------------------
+
+
+In the previous command prompt, being in the C:\\RhodeCode\\Bin folder, 
+just type::
+ 
+ paster serve production.ini 
+
+Open yout web server, and go to http://127.0.0.1:5000 
+
+It works!! :-) 
+
+Remark: 
+If it does not work first time, just Ctrl-C the CMD process and start it 
+again. Don't forget the "http://" in Internet Explorer 
+
+
+
+What this Guide does not cover:
+
+- Installing Celery 
+- Running RhodeCode as Windows Service. You can investigate here:
+ 
+  - http://pypi.python.org/pypi/wsgisvc 
+  - http://ryrobes.com/python/running-python-scripts-as-a-windows-service/     
+  - http://wiki.pylonshq.com/display/pylonscookbook/How+to+run+Pylons+as+a+Windows+service 
+
+- Using Apache. You can investigate here:
+
+  - https://groups.google.com/group/rhodecode/msg/c433074e813ffdc4 
+
+
+Upgrading
+=========
+ 
+Stop running RhodeCode 
+Open a CommandPrompt like in Step7 (VS2008 path + activate) and type::
+ 
+ easy_install -U rhodecode 
+ cd \RhodeCode\Bin 
+
+{ backup your production.ini file now} :: 
+
+ paster make-config RhodeCode production.ini 
+
+(check changes and update your production.ini accordingly) ::
+ 
+ paster upgrade-db production.ini (update database)
+
+Full steps in http://packages.python.org/RhodeCode/upgrade.html 
  No newline at end of file

docs/installation.rst

@@ -1,124 +1,129
 .. _installation:
 
 ============
 Installation
 ============
 
 ``RhodeCode`` is written entirely in Python. Before posting any issues make 
 sure, your not missing any system libraries and using right version of 
 libraries required by RhodeCode. There's also restriction in terms of mercurial
 clients. Minimal version of hg client known working fine with RhodeCode is
 **1.6**. If you're using older client, please upgrade.
 
 
 Installing RhodeCode from Cheese Shop
 -------------------------------------
 
 Rhodecode requires python version 2.5 or higher.
 
 The easiest way to install ``rhodecode`` is to run::
 
     easy_install rhodecode
 
 Or::
 
     pip install rhodecode
 
 If you prefer to install RhodeCode manually simply grab latest release from
 http://pypi.python.org/pypi/RhodeCode, decompress the archive and run::
 
     python setup.py install
 
+Step by step installation example for Windows
+---------------------------------------------
 
-Step by step installation example
----------------------------------
+:ref:`installation_win`
+
+
+Step by step installation example for Linux
+-------------------------------------------
 
 
 For installing RhodeCode i highly recommend using separate virtualenv_. This
 way many required by RhodeCode libraries will remain sandboxed from your main
 python and making things less problematic when doing system python updates. 
 
 - Assuming you have installed virtualenv_ create a new virtual environment 
   using virtualenv command:: 
 
     virtualenv --no-site-packages /var/www/rhodecode-venv
 
 
 .. note:: Using ``--no-site-packages`` when generating your
    virtualenv is **very important**. This flag provides the necessary
    isolation for running the set of packages required by
    RhodeCode.  If you do not specify ``--no-site-packages``,
    it's possible that RhodeCode will not install properly into
    the virtualenv, or, even if it does, may not run properly,
    depending on the packages you've already got installed into your
    Python's "main" site-packages dir.
 
 
 - this will install new virtualenv_ into `/var/www/rhodecode-venv`. 
 - Activate the virtualenv_ by running::
 
     source /var/www/rhodecode-venv/bin/activate
 
 .. note:: If you're using UNIX, *do not* use ``sudo`` to run the
    ``virtualenv`` script.  It's perfectly acceptable (and desirable)
    to create a virtualenv as a normal user.
      
 - Make a folder for rhodecode data files, and configuration somewhere on the 
   filesystem. For example::
 
     mkdir /var/www/rhodecode
   
     
 - Go into the created directory run this command to install rhodecode::
 
     easy_install rhodecode
   
   or::
  
     pip install rhodecode 
 
 - This will install rhodecode together with pylons and all other required 
   python libraries into activated virtualenv
 
 Requirements for Celery (optional)
 ----------------------------------
 
 In order to gain maximum performance
 there are some third-party you must install. When RhodeCode is used 
 together with celery you have to install some kind of message broker,
 recommended one is rabbitmq_ to make the async tasks work.
 
 Of course RhodeCode works in sync mode also and then you do not have to install
 any third party applications. However, using Celery_ will give you a large 
 speed improvement when using many big repositories. If you plan to use 
 RhodeCode for say 7 to 10 repositories, RhodeCode will perform perfectly well 
 without celery running.
    
 If you make the decision to run RhodeCode with celery make sure you run 
 celeryd using paster and message broker together with the application. 
 
 .. note::
    Installing message broker and using celery is optional, RhodeCode will
    work perfectly fine without them.
 
 
 **Message Broker** 
 
 - preferred is `RabbitMq <http://www.rabbitmq.com/>`_
 - A possible alternative is `Redis <http://code.google.com/p/redis/>`_
 
 For installation instructions you can visit: 
 http://ask.github.com/celery/getting-started/index.html.
 This is a very nice tutorial on how to start using celery_ with rabbitmq_
 
 
 You can now proceed to :ref:`setup`
 -----------------------------------
 
 
 
 .. _virtualenv: http://pypi.python.org/pypi/virtualenv  
 .. _python: http://www.python.org/
 .. _mercurial: http://mercurial.selenic.com/
 .. _celery: http://celeryproject.org/
 .. _rabbitmq: http://www.rabbitmq.com/
  No newline at end of file

docs/setup.rst

@@ -1,732 +1,737
 .. _setup:
 
 =====
 Setup
 =====
 
 
 Setting up RhodeCode
 --------------------
 
 First, you will need to create a RhodeCode configuration file. Run the 
 following command to do this::
  
     paster make-config RhodeCode production.ini
 
 - This will create the file `production.ini` in the current directory. This
   configuration file contains the various settings for RhodeCode, e.g proxy 
   port, email settings, usage of static files, cache, celery settings and 
   logging.
 
 
 Next, you need to create the databases used by RhodeCode. I recommend that you
 use postgresql or sqlite (default). If you choose a database other than the
 default ensure you properly adjust the db url in your production.ini
 configuration file to use this other database. RhodeCode currently supports
 postgresql, sqlite and mysql databases. Create the database by running
 the following command::
 
     paster setup-rhodecode production.ini
 
 This will prompt you for a "root" path. This "root" path is the location where
 RhodeCode will store all of its repositories on the current machine. After
 entering this "root" path ``setup-rhodecode`` will also prompt you for a username 
 and password for the initial admin account which ``setup-rhodecode`` sets 
 up for you.
 
+setup process can be fully automated, example for lazy::
+
+    paster setup-rhodecode production.ini --user=marcink --password=secret --email=[e-mail] --repos=/home/marcink/my_repos
+    
+
 - The ``setup-rhodecode`` command will create all of the needed tables and an 
   admin account. When choosing a root path you can either use a new empty 
   location, or a location which already contains existing repositories. If you
   choose a location which contains existing repositories RhodeCode will simply 
   add all of the repositories at the chosen location to it's database. 
   (Note: make sure you specify the correct path to the root).
 - Note: the given path for mercurial_ repositories **must** be write accessible
   for the application. It's very important since the RhodeCode web interface 
   will work without write access, but when trying to do a push it will 
   eventually fail with permission denied errors unless it has write access.
 
 You are now ready to use RhodeCode, to run it simply execute::
  
     paster serve production.ini
  
 - This command runs the RhodeCode server. The web app should be available at the 
   127.0.0.1:5000. This ip and port is configurable via the production.ini 
   file created in previous step
 - Use the admin account you created above when running ``setup-rhodecode`` 
   to login to the web app.
 - The default permissions on each repository is read, and the owner is admin. 
   Remember to update these if needed.
 - In the admin panel you can toggle ldap, anonymous, permissions settings. As
   well as edit more advanced options on users and repositories
 
 Optionally users can create `rcextensions` package that extends RhodeCode
 functionality. To do this simply execute::
 
     paster make-rcext production.ini
 
 This will create `rcextensions` package in the same place that your `ini` file
 lives. With `rcextensions` it's possible to add additional mapping for whoosh, 
 stats and add additional code into the push/pull/create repo hooks. For example
 for sending signals to build-bots such as jenkins.
 Please see the `__init__.py` file inside `rcextensions` package 
 for more details.
 
 
 Using RhodeCode with SSH
 ------------------------
 
 RhodeCode currently only hosts repositories using http and https. (The addition
 of ssh hosting is a planned future feature.) However you can easily use ssh in
 parallel with RhodeCode. (Repository access via ssh is a standard "out of
 the box" feature of mercurial_ and you can use this to access any of the
 repositories that RhodeCode is hosting. See PublishingRepositories_)
 
 RhodeCode repository structures are kept in directories with the same name 
 as the project. When using repository groups, each group is a subdirectory.
 This allows you to easily use ssh for accessing repositories.
 
 In order to use ssh you need to make sure that your web-server and the users 
 login accounts have the correct permissions set on the appropriate directories.
 (Note that these permissions are independent of any permissions you have set up
 using the RhodeCode web interface.)
 
 If your main directory (the same as set in RhodeCode settings) is for example
 set to **/home/hg** and the repository you are using is named `rhodecode`, then
 to clone via ssh you should run::
 
     hg clone ssh://[e-mail]/home/hg/rhodecode
 
 Using other external tools such as mercurial-server_ or using ssh key based
 authentication is fully supported.
 
 Note: In an advanced setup, in order for your ssh access to use the same
 permissions as set up via the RhodeCode web interface, you can create an
 authentication hook to connect to the rhodecode db and runs check functions for
 permissions against that.
     
 Setting up Whoosh full text search
 ----------------------------------
 
 Starting from version 1.1 the whoosh index can be build by using the paster
 command ``make-index``. To use ``make-index`` you must specify the configuration
 file that stores the location of the index. You may specify the location of the 
 repositories (`--repo-location`).  If not specified, this value is retrieved 
 from the RhodeCode database.  This was required prior to 1.2.  Starting from 
 version 1.2 it is also possible to specify a comma separated list of 
 repositories (`--index-only`) to build index only on chooses repositories 
 skipping any other found in repos location
 
 You may optionally pass the option `-f` to enable a full index rebuild. Without
 the `-f` option, indexing will run always in "incremental" mode.
 
 For an incremental index build use::
 
 	paster make-index production.ini 
 
 For a full index rebuild use::
 
 	paster make-index production.ini -f 
 
 
 building index just for chosen repositories is possible with such command::
  
  paster make-index production.ini --index-only=vcs,rhodecode
 
 
 In order to do periodical index builds and keep your index always up to date.
 It's recommended to do a crontab entry for incremental indexing. 
 An example entry might look like this::
  
     /path/to/python/bin/paster make-index /path/to/rhodecode/production.ini 
   
 When using incremental mode (the default) whoosh will check the last
 modification date of each file and add it to be reindexed if a newer file is
 available. The indexing daemon checks for any removed files and removes them
 from index.
 
 If you want to rebuild index from scratch, you can use the `-f` flag as above,
 or in the admin panel you can check `build from scratch` flag.
 
 
 Setting up LDAP support
 -----------------------
 
 RhodeCode starting from version 1.1 supports ldap authentication. In order
 to use LDAP, you have to install the python-ldap_ package. This package is 
 available via pypi, so you can install it by running
 
 using easy_install::
 
     easy_install python-ldap
  
 using pip::
 
     pip install python-ldap
 
 .. note::
    python-ldap requires some certain libs on your system, so before installing 
    it check that you have at least `openldap`, and `sasl` libraries.
 
 LDAP settings are located in admin->ldap section,
 
 Here's a typical ldap setup::
 
  Connection settings
  Enable LDAP          = checked
  Host                 = host.example.org
  Port                 = 389
  Account              = <account>
  Password             = <password>
  Connection Security  = LDAPS connection
  Certificate Checks   = DEMAND
 
  Search settings
  Base DN              = CN=users,DC=host,DC=example,DC=org
  LDAP Filter          = (&(objectClass=user)(!(objectClass=computer)))
  LDAP Search Scope    = SUBTREE
 
  Attribute mappings
  Login Attribute      = uid
  First Name Attribute = firstName
  Last Name Attribute  = lastName
  E-mail Attribute     = mail
 
 .. _enable_ldap:
 
 Enable LDAP : required
     Whether to use LDAP for authenticating users.
 
 .. _ldap_host:
 
 Host : required
     LDAP server hostname or IP address.
 
 .. _Port:
 
 Port : required
     389 for un-encrypted LDAP, 636 for SSL-encrypted LDAP.
 
 .. _ldap_account:
 
 Account : optional
     Only required if the LDAP server does not allow anonymous browsing of
     records.  This should be a special account for record browsing.  This
     will require `LDAP Password`_ below.
 
 .. _LDAP Password:
 
 Password : optional
     Only required if the LDAP server does not allow anonymous browsing of
     records.
 
 .. _Enable LDAPS:
 
 Connection Security : required
     Defines the connection to LDAP server
 
     No encryption
         Plain non encrypted connection
         
     LDAPS connection
         Enable ldaps connection. It will likely require `Port`_ to be set to 
         a different value (standard LDAPS port is 636). When LDAPS is enabled 
         then `Certificate Checks`_ is required.
         
     START_TLS on LDAP connection
         START TLS connection
 
 .. _Certificate Checks:
 
 Certificate Checks : optional
     How SSL certificates verification is handled - this is only useful when
     `Enable LDAPS`_ is enabled.  Only DEMAND or HARD offer full SSL security 
     while the other options are susceptible to man-in-the-middle attacks.  SSL
     certificates can be installed to /etc/openldap/cacerts so that the
     DEMAND or HARD options can be used with self-signed certificates or
     certificates that do not have traceable certificates of authority.
 
     NEVER
         A serve certificate will never be requested or checked.
 
     ALLOW
         A server certificate is requested.  Failure to provide a
         certificate or providing a bad certificate will not terminate the
         session.
 
     TRY
         A server certificate is requested.  Failure to provide a
         certificate does not halt the session; providing a bad certificate
         halts the session.
 
     DEMAND
         A server certificate is requested and must be provided and
         authenticated for the session to proceed.
 
     HARD
         The same as DEMAND.
 
 .. _Base DN:
 
 Base DN : required
     The Distinguished Name (DN) where searches for users will be performed.
     Searches can be controlled by `LDAP Filter`_ and `LDAP Search Scope`_.
 
 .. _LDAP Filter:
 
 LDAP Filter : optional
     A LDAP filter defined by RFC 2254.  This is more useful when `LDAP
     Search Scope`_ is set to SUBTREE.  The filter is useful for limiting
     which LDAP objects are identified as representing Users for
     authentication.  The filter is augmented by `Login Attribute`_ below.
     This can commonly be left blank.
 
 .. _LDAP Search Scope:
 
 LDAP Search Scope : required
     This limits how far LDAP will search for a matching object.
 
     BASE
         Only allows searching of `Base DN`_ and is usually not what you
         want.
 
     ONELEVEL
         Searches all entries under `Base DN`_, but not Base DN itself.
 
     SUBTREE
         Searches all entries below `Base DN`_, but not Base DN itself.
         When using SUBTREE `LDAP Filter`_ is useful to limit object
         location.
 
 .. _Login Attribute:
 
 Login Attribute : required        
     The LDAP record attribute that will be matched as the USERNAME or
     ACCOUNT used to connect to RhodeCode.  This will be added to `LDAP
     Filter`_ for locating the User object.  If `LDAP Filter`_ is specified as
     "LDAPFILTER", `Login Attribute`_ is specified as "uid" and the user has
     connected as "jsmith" then the `LDAP Filter`_ will be augmented as below
     ::
 
         (&(LDAPFILTER)(uid=jsmith))
 
 .. _ldap_attr_firstname:
 
 First Name Attribute : required
     The LDAP record attribute which represents the user's first name.
 
 .. _ldap_attr_lastname:
 
 Last Name Attribute : required
     The LDAP record attribute which represents the user's last name.
 
 .. _ldap_attr_email:
 
 Email Attribute : required
     The LDAP record attribute which represents the user's email address.
 
 If all data are entered correctly, and python-ldap_ is properly installed
 users should be granted access to RhodeCode with ldap accounts.  At this
 time user information is copied from LDAP into the RhodeCode user database.
 This means that updates of an LDAP user object may not be reflected as a
 user update in RhodeCode.
 
 If You have problems with LDAP access and believe You entered correct
 information check out the RhodeCode logs, any error messages sent from LDAP
 will be saved there.
 
 Active Directory
 ''''''''''''''''
 
 RhodeCode can use Microsoft Active Directory for user authentication.  This
 is done through an LDAP or LDAPS connection to Active Directory.  The
 following LDAP configuration settings are typical for using Active
 Directory ::
 
  Base DN              = OU=SBSUsers,OU=Users,OU=MyBusiness,DC=v3sys,DC=local
  Login Attribute      = sAMAccountName
  First Name Attribute = givenName
  Last Name Attribute  = sn
  E-mail Attribute     = mail
 
 All other LDAP settings will likely be site-specific and should be
 appropriately configured.
 
 
 Authentication by container or reverse-proxy
 --------------------------------------------
 
 Starting with version 1.3, RhodeCode supports delegating the authentication
 of users to its WSGI container, or to a reverse-proxy server through which all
 clients access the application.
 
 When these authentication methods are enabled in RhodeCode, it uses the
 username that the container/proxy (Apache/Nginx/etc) authenticated and doesn't
 perform the authentication itself. The authorization, however, is still done by
 RhodeCode according to its settings.
 
 When a user logs in for the first time using these authentication methods,
 a matching user account is created in RhodeCode with default permissions. An
 administrator can then modify it using RhodeCode's admin interface.
 It's also possible for an administrator to create accounts and configure their
 permissions before the user logs in for the first time.
 
 Container-based authentication
 ''''''''''''''''''''''''''''''
 
 In a container-based authentication setup, RhodeCode reads the user name from
 the ``REMOTE_USER`` server variable provided by the WSGI container.
 
 After setting up your container (see `Apache's WSGI config`_), you'd need
 to configure it to require authentication on the location configured for
 RhodeCode.
 
 In order for RhodeCode to start using the provided username, you should set the
 following in the [app:main] section of your .ini file::
 
     container_auth_enabled = true
 
 
 Proxy pass-through authentication
 '''''''''''''''''''''''''''''''''
 
 In a proxy pass-through authentication setup, RhodeCode reads the user name
 from the ``X-Forwarded-User`` request header, which should be configured to be
 sent by the reverse-proxy server.
 
 After setting up your proxy solution (see `Apache virtual host reverse proxy example`_,
 `Apache as subdirectory`_ or `Nginx virtual host example`_), you'd need to
 configure the authentication and add the username in a request header named
 ``X-Forwarded-User``.
 
 For example, the following config section for Apache sets a subdirectory in a
 reverse-proxy setup with basic auth::
 
     <Location /<someprefix> >
       ProxyPass http://127.0.0.1:5000/<someprefix>
       ProxyPassReverse http://127.0.0.1:5000/<someprefix>
       SetEnvIf X-Url-Scheme https HTTPS=1
 
       AuthType Basic
       AuthName "RhodeCode authentication"
       AuthUserFile /home/web/rhodecode/.htpasswd
       require valid-user
 
       RequestHeader unset X-Forwarded-User
 
       RewriteEngine On
       RewriteCond %{LA-U:REMOTE_USER} (.+)
       RewriteRule .* - [E=RU:%1]
       RequestHeader set X-Forwarded-User %{RU}e
     </Location> 
 
 In order for RhodeCode to start using the forwarded username, you should set
 the following in the [app:main] section of your .ini file::
 
     proxypass_auth_enabled = true
 
 .. note::
    If you enable proxy pass-through authentication, make sure your server is
    only accessible through the proxy. Otherwise, any client would be able to
    forge the authentication header and could effectively become authenticated
    using any account of their liking.
 
 Integration with Issue trackers
 -------------------------------
 
 RhodeCode provides a simple integration with issue trackers. It's possible
 to define a regular expression that will fetch issue id stored in commit
 messages and replace that with an url to this issue. To enable this simply
 uncomment following variables in the ini file::
 
     url_pat = (?:^#|\s#)(\w+)
     issue_server_link = https://myissueserver.com/{repo}/issue/{id}
     issue_prefix = #
 
 `url_pat` is the regular expression that will fetch issues from commit messages.
 Default regex will match issues in format of #<number> eg. #300.
  
 Matched issues will be replace with the link specified as `issue_server_link` 
 {id} will be replaced with issue id, and {repo} with repository name.
 Since the # is striped `issue_prefix` is added as a prefix to url. 
 `issue_prefix` can be something different than # if you pass 
 ISSUE- as issue prefix this will generate an url in format::
  
   <a href="https://myissueserver.com/example_repo/issue/300">ISSUE-300</a>  
 
 Hook management
 ---------------
 
 Hooks can be managed in similar way to this used in .hgrc files.
 To access hooks setting click `advanced setup` on Hooks section of Mercurial
 Settings in Admin. 
 
 There are 4 built in hooks that cannot be changed (only enable/disable by
 checkboxes on previos section).
 To add another custom hook simply fill in first section with 
 <name>.<hook_type> and the second one with hook path. Example hooks
 can be found at *rhodecode.lib.hooks*. 
 
 
 Changing default encoding
 -------------------------
 
 By default RhodeCode uses utf8 encoding, starting from 1.3 series this
 can be changed, simply edit default_encoding in .ini file to desired one.
 This affects many parts in rhodecode including commiters names, filenames,
 encoding of commit messages. In addition RhodeCode can detect if `chardet`
 library is installed. If `chardet` is detected RhodeCode will fallback to it
 when there are encode/decode errors.
 
 
 Setting Up Celery
 -----------------
 
 Since version 1.1 celery is configured by the rhodecode ini configuration files.
 Simply set use_celery=true in the ini file then add / change the configuration 
 variables inside the ini file.
 
 Remember that the ini files use the format with '.' not with '_' like celery.
 So for example setting `BROKER_HOST` in celery means setting `broker.host` in
 the config file.
 
 In order to start using celery run::
 
  paster celeryd <configfile.ini>
 
 
 .. note::
    Make sure you run this command from the same virtualenv, and with the same 
    user that rhodecode runs.
    
 HTTPS support
 -------------
 
 There are two ways to enable https:
 
 - Set HTTP_X_URL_SCHEME in your http server headers, than rhodecode will
   recognize this headers and make proper https redirections
 - Alternatively, change the `force_https = true` flag in the ini configuration 
   to force using https, no headers are needed than to enable https
 
 
 Nginx virtual host example
 --------------------------
 
 Sample config for nginx using proxy::
 
     upstream rc {
         server 127.0.0.1:5000;
         # add more instances for load balancing
         #server 127.0.0.1:5001;
         #server 127.0.0.1:5002;
     }
     
     server {
        listen          80;
        server_name     hg.myserver.com;
        access_log      /var/log/nginx/rhodecode.access.log;
        error_log       /var/log/nginx/rhodecode.error.log;
 
        location / {
             try_files $uri @rhode;
        }
     
        location @rhode {
             proxy_pass      http://rc;
             include         /etc/nginx/proxy.conf;
        }
 
     }  
   
 Here's the proxy.conf. It's tuned so it will not timeout on long
 pushes or large pushes::
     
     proxy_redirect              off;
     proxy_set_header            Host $host;
     proxy_set_header            X-Url-Scheme $scheme;
     proxy_set_header            X-Host $http_host;
     proxy_set_header            X-Real-IP $remote_addr;
     proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header            Proxy-host $proxy_host;
     client_max_body_size        400m;
     client_body_buffer_size     128k;
     proxy_buffering             off;
     proxy_connect_timeout       7200;
     proxy_send_timeout          7200;
     proxy_read_timeout          7200;
     proxy_buffers               8 32k;
  
 Also, when using root path with nginx you might set the static files to false
 in the production.ini file::
 
     [app:main]
       use = egg:rhodecode
       full_stack = true
       static_files = false
       lang=en
       cache_dir = %(here)s/data
 
 In order to not have the statics served by the application. This improves speed.
 
 
 Apache virtual host reverse proxy example
 -----------------------------------------
 
 Here is a sample configuration file for apache using proxy::
 
     <VirtualHost *:80>
             ServerName hg.myserver.com
             ServerAlias hg.myserver.com
     
             <Proxy *>
               Order allow,deny
               Allow from all
             </Proxy>
     
             #important !
             #Directive to properly generate url (clone url) for pylons
             ProxyPreserveHost On
     
             #rhodecode instance
             ProxyPass / http://127.0.0.1:5000/
             ProxyPassReverse / http://127.0.0.1:5000/
             
             #to enable https use line below
             #SetEnvIf X-Url-Scheme https HTTPS=1
             
     </VirtualHost> 
 
 
 Additional tutorial
 http://wiki.pylonshq.com/display/pylonscookbook/Apache+as+a+reverse+proxy+for+Pylons
 
 
 Apache as subdirectory
 ----------------------
 
 Apache subdirectory part::
 
     <Location /<someprefix> >
       ProxyPass http://127.0.0.1:5000/<someprefix>
       ProxyPassReverse http://127.0.0.1:5000/<someprefix>
       SetEnvIf X-Url-Scheme https HTTPS=1
     </Location> 
 
 Besides the regular apache setup you will need to add the following line
 into [app:main] section of your .ini file::
 
     filter-with = proxy-prefix
 
 Add the following at the end of the .ini file::
 
     [filter:proxy-prefix]
     use = egg:PasteDeploy#prefix
     prefix = /<someprefix> 
 
 
 then change <someprefix> into your choosen prefix
 
 Apache's WSGI config
 --------------------
 
 Alternatively, RhodeCode can be set up with Apache under mod_wsgi. For
 that, you'll need to:
 
 - Install mod_wsgi. If using a Debian-based distro, you can install
   the package libapache2-mod-wsgi::
 
     aptitude install libapache2-mod-wsgi
 
 - Enable mod_wsgi::
 
     a2enmod wsgi
 
 - Create a wsgi dispatch script, like the one below. Make sure you
   check the paths correctly point to where you installed RhodeCode
   and its Python Virtual Environment.
 - Enable the WSGIScriptAlias directive for the wsgi dispatch script,
   as in the following example. Once again, check the paths are
   correctly specified.
 
 Here is a sample excerpt from an Apache Virtual Host configuration file::
 
     WSGIDaemonProcess pylons user=www-data group=www-data processes=1 \
         threads=4 \
         python-path=/home/web/rhodecode/pyenv/lib/python2.6/site-packages
     WSGIScriptAlias / /home/web/rhodecode/dispatch.wsgi
     WSGIPassAuthorization On
 
 Example wsgi dispatch script::
 
     import os
     os.environ["HGENCODING"] = "UTF-8"
     os.environ['PYTHON_EGG_CACHE'] = '/home/web/rhodecode/.egg-cache'
     
     # sometimes it's needed to set the curent dir
     os.chdir('/home/web/rhodecode/') 
 
     import site
     site.addsitedir("/home/web/rhodecode/pyenv/lib/python2.6/site-packages")
     
     from paste.deploy import loadapp
     from paste.script.util.logging_config import fileConfig
 
     fileConfig('/home/web/rhodecode/production.ini')
     application = loadapp('config:/home/web/rhodecode/production.ini')
 
 Note: when using mod_wsgi you'll need to install the same version of
 Mercurial that's inside RhodeCode's virtualenv also on the system's Python
 environment.
 
 
 Other configuration files
 -------------------------
 
 Some example init.d scripts can be found here, for debian and gentoo:
 
 https://rhodecode.org/rhodecode/files/tip/init.d
 
 
 Troubleshooting
 ---------------
 
 :Q: **Missing static files?**
 :A: Make sure either to set the `static_files = true` in the .ini file or
    double check the root path for your http setup. It should point to 
    for example:
    /home/my-virtual-python/lib/python2.6/site-packages/rhodecode/public
    
 | 
 
 :Q: **Can't install celery/rabbitmq**
 :A: Don't worry RhodeCode works without them too. No extra setup is required.
 
 |
  
 :Q: **Long lasting push timeouts?**
 :A: Make sure you set a longer timeouts in your proxy/fcgi settings, timeouts
     are caused by https server and not RhodeCode.
     
 | 
 
 :Q: **Large pushes timeouts?**
 :A: Make sure you set a proper max_body_size for the http server.
 
 |
 
 :Q: **Apache doesn't pass basicAuth on pull/push?**
 :A: Make sure you added `WSGIPassAuthorization true`.
 
 For further questions search the `Issues tracker`_, or post a message in the 
 `google group rhodecode`_
 
 .. _virtualenv: http://pypi.python.org/pypi/virtualenv
 .. _python: http://www.python.org/
 .. _mercurial: http://mercurial.selenic.com/
 .. _celery: http://celeryproject.org/
 .. _rabbitmq: http://www.rabbitmq.com/
 .. _python-ldap: http://www.python-ldap.org/
 .. _mercurial-server: http://www.lshift.net/mercurial-server.html
 .. _PublishingRepositories: http://mercurial.selenic.com/wiki/PublishingRepositories
 .. _Issues tracker: https://bitbucket.org/marcinkuzminski/rhodecode/issues
 .. _google group rhodecode: http://groups.google.com/group/rhodecode

rhodecode/lib/diffs.py

@@ -1,609 +1,614
 # -*- coding: utf-8 -*-
 """
     rhodecode.lib.diffs
     ~~~~~~~~~~~~~~~~~~~
 
     Set of diffing helpers, previously part of vcs
 
 
     :created_on: Dec 4, 2011
     :author: marcink
     :copyright: (C) 2010-2012 Marcin Kuzminski <[e-mail]>
     :original copyright: 2007-2008 by Armin Ronacher
     :license: GPLv3, see COPYING for more details.
 """
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import re
 import io
 import difflib
 import markupsafe
 
 from itertools import tee, imap
 
 from mercurial import patch
 from mercurial.mdiff import diffopts
 from mercurial.bundlerepo import bundlerepository
 from mercurial import localrepo
 
 from pylons.i18n.translation import _
 
 from rhodecode.lib.vcs.exceptions import VCSError
 from rhodecode.lib.vcs.nodes import FileNode, SubModuleNode
 from rhodecode.lib.helpers import escape
 from rhodecode.lib.utils import EmptyChangeset, make_ui
 
 
 def wrap_to_table(str_):
     return '''<table class="code-difftable">
                 <tr class="line no-comment">
                 <td class="lineno new"></td>
                 <td class="code no-comment"><pre>%s</pre></td>
                 </tr>
               </table>''' % str_
 
 
 def wrapped_diff(filenode_old, filenode_new, cut_off_limit=None,
                 ignore_whitespace=True, line_context=3,
                 enable_comments=False):
     """
     returns a wrapped diff into a table, checks for cut_off_limit and presents
     proper message
     """
 
     if filenode_old is None:
         filenode_old = FileNode(filenode_new.path, '', EmptyChangeset())
 
     if filenode_old.is_binary or filenode_new.is_binary:
         diff = wrap_to_table(_('binary file'))
         stats = (0, 0)
         size = 0
 
     elif cut_off_limit != -1 and (cut_off_limit is None or
     (filenode_old.size < cut_off_limit and filenode_new.size < cut_off_limit)):
 
         f_gitdiff = get_gitdiff(filenode_old, filenode_new,
                                 ignore_whitespace=ignore_whitespace,
                                 context=line_context)
         diff_processor = DiffProcessor(f_gitdiff, format='gitdiff')
 
         diff = diff_processor.as_html(enable_comments=enable_comments)
         stats = diff_processor.stat()
         size = len(diff or '')
     else:
         diff = wrap_to_table(_('Changeset was too big and was cut off, use '
                                'diff menu to display this diff'))
         stats = (0, 0)
         size = 0
     if not diff:
         submodules = filter(lambda o: isinstance(o, SubModuleNode),
                             [filenode_new, filenode_old])
         if submodules:
             diff = wrap_to_table(escape('Submodule %r' % submodules[0]))
         else:
             diff = wrap_to_table(_('No changes detected'))
 
     cs1 = filenode_old.changeset.raw_id
     cs2 = filenode_new.changeset.raw_id
 
     return size, cs1, cs2, diff, stats
 
 
 def get_gitdiff(filenode_old, filenode_new, ignore_whitespace=True, context=3):
     """
     Returns git style diff between given ``filenode_old`` and ``filenode_new``.
 
     :param ignore_whitespace: ignore whitespaces in diff
     """
     # make sure we pass in default context
     context = context or 3
     submodules = filter(lambda o: isinstance(o, SubModuleNode),
                         [filenode_new, filenode_old])
     if submodules:
         return ''
 
     for filenode in (filenode_old, filenode_new):
         if not isinstance(filenode, FileNode):
             raise VCSError("Given object should be FileNode object, not %s"
                 % filenode.__class__)
 
     repo = filenode_new.changeset.repository
     old_raw_id = getattr(filenode_old.changeset, 'raw_id', repo.EMPTY_CHANGESET)
     new_raw_id = getattr(filenode_new.changeset, 'raw_id', repo.EMPTY_CHANGESET)
 
     vcs_gitdiff = repo.get_diff(old_raw_id, new_raw_id, filenode_new.path,
                                  ignore_whitespace, context)
     return vcs_gitdiff
 
 
 class DiffProcessor(object):
     """
     Give it a unified diff and it returns a list of the files that were
     mentioned in the diff together with a dict of meta information that
     can be used to render it in a HTML template.
     """
     _chunk_re = re.compile(r'@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@(.*)')
 
     def __init__(self, diff, differ='diff', format='udiff'):
         """
         :param diff:   a text in diff format or generator
         :param format: format of diff passed, `udiff` or `gitdiff`
         """
         if isinstance(diff, basestring):
             diff = [diff]
 
         self.__udiff = diff
         self.__format = format
         self.adds = 0
         self.removes = 0
 
         if isinstance(self.__udiff, basestring):
             self.lines = iter(self.__udiff.splitlines(1))
 
         elif self.__format == 'gitdiff':
             udiff_copy = self.copy_iterator()
             self.lines = imap(self.escaper, self._parse_gitdiff(udiff_copy))
         else:
             udiff_copy = self.copy_iterator()
             self.lines = imap(self.escaper, udiff_copy)
 
         # Select a differ.
         if differ == 'difflib':
             self.differ = self._highlight_line_difflib
         else:
             self.differ = self._highlight_line_udiff
 
     def escaper(self, string):
         return markupsafe.escape(string)
 
     def copy_iterator(self):
         """
         make a fresh copy of generator, we should not iterate thru
         an original as it's needed for repeating operations on
         this instance of DiffProcessor
         """
         self.__udiff, iterator_copy = tee(self.__udiff)
         return iterator_copy
 
     def _extract_rev(self, line1, line2):
         """
         Extract the operation (A/M/D), filename and revision hint from a line.
         """
 
         try:
             if line1.startswith('--- ') and line2.startswith('+++ '):
                 l1 = line1[4:].split(None, 1)
                 old_filename = (l1[0].replace('a/', '', 1)
                                 if len(l1) >= 1 else None)
                 old_rev = l1[1] if len(l1) == 2 else 'old'
 
                 l2 = line2[4:].split(None, 1)
                 new_filename = (l2[0].replace('b/', '', 1)
                                 if len(l1) >= 1 else None)
                 new_rev = l2[1] if len(l2) == 2 else 'new'
 
                 filename = (old_filename
                             if old_filename != '/dev/null' else new_filename)
 
                 operation = 'D' if new_filename == '/dev/null' else None
                 if not operation:
                     operation = 'M' if old_filename != '/dev/null' else 'A'
 
                 return operation, filename, new_rev, old_rev
         except (ValueError, IndexError):
             pass
 
         return None, None, None, None
 
     def _parse_gitdiff(self, diffiterator):
         def line_decoder(l):
             if l.startswith('+') and not l.startswith('+++'):
                 self.adds += 1
             elif l.startswith('-') and not l.startswith('---'):
                 self.removes += 1
             return l.decode('utf8', 'replace')
 
         output = list(diffiterator)
         size = len(output)
 
         if size == 2:
             l = []
             l.extend([output[0]])
             l.extend(output[1].splitlines(1))
             return map(line_decoder, l)
         elif size == 1:
             return  map(line_decoder, output[0].splitlines(1))
         elif size == 0:
             return []
 
         raise Exception('wrong size of diff %s' % size)
 
     def _highlight_line_difflib(self, line, next_):
         """
         Highlight inline changes in both lines.
         """
 
         if line['action'] == 'del':
             old, new = line, next_
         else:
             old, new = next_, line
 
         oldwords = re.split(r'(\W)', old['line'])
         newwords = re.split(r'(\W)', new['line'])
 
         sequence = difflib.SequenceMatcher(None, oldwords, newwords)
 
         oldfragments, newfragments = [], []
         for tag, i1, i2, j1, j2 in sequence.get_opcodes():
             oldfrag = ''.join(oldwords[i1:i2])
             newfrag = ''.join(newwords[j1:j2])
             if tag != 'equal':
                 if oldfrag:
                     oldfrag = '<del>%s</del>' % oldfrag
                 if newfrag:
                     newfrag = '<ins>%s</ins>' % newfrag
             oldfragments.append(oldfrag)
             newfragments.append(newfrag)
 
         old['line'] = "".join(oldfragments)
         new['line'] = "".join(newfragments)
 
     def _highlight_line_udiff(self, line, next_):
         """
         Highlight inline changes in both lines.
         """
         start = 0
         limit = min(len(line['line']), len(next_['line']))
         while start < limit and line['line'][start] == next_['line'][start]:
             start += 1
         end = -1
         limit -= start
         while -end <= limit and line['line'][end] == next_['line'][end]:
             end -= 1
         end += 1
         if start or end:
             def do(l):
                 last = end + len(l['line'])
                 if l['action'] == 'add':
                     tag = 'ins'
                 else:
                     tag = 'del'
                 l['line'] = '%s<%s>%s</%s>%s' % (
                     l['line'][:start],
                     tag,
                     l['line'][start:last],
                     tag,
                     l['line'][last:]
                 )
-
             do(line)
             do(next_)
 
     def _parse_udiff(self):
         """
         Parse the diff an return data for the template.
         """
         lineiter = self.lines
         files = []
         try:
             line = lineiter.next()
-            # skip first context
-            skipfirst = True
-
             while 1:
                 # continue until we found the old file
                 if not line.startswith('--- '):
                     line = lineiter.next()
                     continue
 
                 chunks = []
                 stats = [0, 0]
                 operation, filename, old_rev, new_rev = \
                     self._extract_rev(line, lineiter.next())
                 files.append({
                     'filename':         filename,
                     'old_revision':     old_rev,
                     'new_revision':     new_rev,
                     'chunks':           chunks,
                     'operation':        operation,
                     'stats':            stats,
                 })
 
                 line = lineiter.next()
                 while line:
-
                     match = self._chunk_re.match(line)
                     if not match:
                         break
 
                     lines = []
                     chunks.append(lines)
 
                     old_line, old_end, new_line, new_end = \
                         [int(x or 1) for x in match.groups()[:-1]]
                     old_line -= 1
                     new_line -= 1
-                    context = len(match.groups()) == 5
+                    gr = match.groups()
+                    context = len(gr) == 5
                     old_end += old_line
                     new_end += new_line
 
                     if context:
-                        if not skipfirst:
+                        # skip context only if it's first line
+                        if int(gr[0]) > 1:
                             lines.append({
                                 'old_lineno': '...',
                                 'new_lineno': '...',
                                 'action':     'context',
                                 'line':       line,
                             })
-                        else:
-                            skipfirst = False
 
                     line = lineiter.next()
                     while old_line < old_end or new_line < new_end:
                         if line:
                             command, line = line[0], line[1:]
                         else:
                             command = ' '
                         affects_old = affects_new = False
 
                         # ignore those if we don't expect them
                         if command in '#@':
                             continue
                         elif command == '+':
                             affects_new = True
                             action = 'add'
                             stats[0] += 1
                         elif command == '-':
                             affects_old = True
                             action = 'del'
                             stats[1] += 1
                         else:
                             affects_old = affects_new = True
                             action = 'unmod'
 
-                        old_line += affects_old
-                        new_line += affects_new
-                        lines.append({
-                            'old_lineno':   affects_old and old_line or '',
-                            'new_lineno':   affects_new and new_line or '',
-                            'action':       action,
-                            'line':         line
-                        })
+                        if line.find('No newline at end of file') != -1:
+                            lines.append({
+                                'old_lineno':   '...',
+                                'new_lineno':   '...',
+                                'action':       'context',
+                                'line':         line
+                            })
+
+                        else:
+                            old_line += affects_old
+                            new_line += affects_new
+                            lines.append({
+                                'old_lineno':   affects_old and old_line or '',
+                                'new_lineno':   affects_new and new_line or '',
+                                'action':       action,
+                                'line':         line
+                            })
+
                         line = lineiter.next()
         except StopIteration:
             pass
 
         # highlight inline changes
         for diff_data in files:
             for chunk in diff_data['chunks']:
                 lineiter = iter(chunk)
                 try:
                     while 1:
                         line = lineiter.next()
                         if line['action'] != 'unmod':
                             nextline = lineiter.next()
-                            if nextline['action'] == 'unmod' or \
+                            if nextline['action'] in ['unmod', 'context'] or \
                                nextline['action'] == line['action']:
                                 continue
                             self.differ(line, nextline)
                 except StopIteration:
                     pass
         return files
 
     def prepare(self):
         """
         Prepare the passed udiff for HTML rendering. It'l return a list
         of dicts
         """
         return self._parse_udiff()
 
     def _safe_id(self, idstring):
         """Make a string safe for including in an id attribute.
 
         The HTML spec says that id attributes 'must begin with
         a letter ([A-Za-z]) and may be followed by any number
         of letters, digits ([0-9]), hyphens ("-"), underscores
         ("_"), colons (":"), and periods (".")'. These regexps
         are slightly over-zealous, in that they remove colons
         and periods unnecessarily.
 
         Whitespace is transformed into underscores, and then
         anything which is not a hyphen or a character that
         matches \w (alphanumerics and underscore) is removed.
 
         """
         # Transform all whitespace to underscore
         idstring = re.sub(r'\s', "_", '%s' % idstring)
         # Remove everything that is not a hyphen or a member of \w
         idstring = re.sub(r'(?!-)\W', "", idstring).lower()
         return idstring
 
     def raw_diff(self):
         """
         Returns raw string as udiff
         """
         udiff_copy = self.copy_iterator()
         if self.__format == 'gitdiff':
             udiff_copy = self._parse_gitdiff(udiff_copy)
         return u''.join(udiff_copy)
 
     def as_html(self, table_class='code-difftable', line_class='line',
                 new_lineno_class='lineno old', old_lineno_class='lineno new',
                 code_class='code', enable_comments=False, diff_lines=None):
         """
         Return given diff as html table with customized css classes
         """
         def _link_to_if(condition, label, url):
             """
             Generates a link if condition is meet or just the label if not.
             """
 
             if condition:
                 return '''<a href="%(url)s">%(label)s</a>''' % {
                     'url': url,
                     'label': label
                 }
             else:
                 return label
         if diff_lines is None:
             diff_lines = self.prepare()
         _html_empty = True
         _html = []
         _html.append('''<table class="%(table_class)s">\n''' % {
             'table_class': table_class
         })
         for diff in diff_lines:
             for line in diff['chunks']:
                 _html_empty = False
                 for change in line:
                     _html.append('''<tr class="%(lc)s %(action)s">\n''' % {
                         'lc': line_class,
                         'action': change['action']
                     })
                     anchor_old_id = ''
                     anchor_new_id = ''
                     anchor_old = "%(filename)s_o%(oldline_no)s" % {
                         'filename': self._safe_id(diff['filename']),
                         'oldline_no': change['old_lineno']
                     }
                     anchor_new = "%(filename)s_n%(oldline_no)s" % {
                         'filename': self._safe_id(diff['filename']),
                         'oldline_no': change['new_lineno']
                     }
                     cond_old = (change['old_lineno'] != '...' and
                                 change['old_lineno'])
                     cond_new = (change['new_lineno'] != '...' and
                                 change['new_lineno'])
                     if cond_old:
                         anchor_old_id = 'id="%s"' % anchor_old
                     if cond_new:
                         anchor_new_id = 'id="%s"' % anchor_new
                     ###########################################################
                     # OLD LINE NUMBER
                     ###########################################################
                     _html.append('''\t<td %(a_id)s class="%(olc)s">''' % {
                         'a_id': anchor_old_id,
                         'olc': old_lineno_class
                     })
 
                     _html.append('''%(link)s''' % {
                         'link': _link_to_if(True, change['old_lineno'],
                                             '#%s' % anchor_old)
                     })
                     _html.append('''</td>\n''')
                     ###########################################################
                     # NEW LINE NUMBER
                     ###########################################################
 
                     _html.append('''\t<td %(a_id)s class="%(nlc)s">''' % {
                         'a_id': anchor_new_id,
                         'nlc': new_lineno_class
                     })
 
                     _html.append('''%(link)s''' % {
                         'link': _link_to_if(True, change['new_lineno'],
                                             '#%s' % anchor_new)
                     })
                     _html.append('''</td>\n''')
                     ###########################################################
                     # CODE
                     ###########################################################
                     comments = '' if enable_comments else 'no-comment'
                     _html.append('''\t<td class="%(cc)s %(inc)s">''' % {
                         'cc': code_class,
                         'inc': comments
                     })
                     _html.append('''\n\t\t<pre>%(code)s</pre>\n''' % {
                         'code': change['line']
                     })
                     _html.append('''\t</td>''')
                     _html.append('''\n</tr>\n''')
         _html.append('''</table>''')
         if _html_empty:
             return None
         return ''.join(_html)
 
     def stat(self):
         """
         Returns tuple of added, and removed lines for this instance
         """
         return self.adds, self.removes
 
 
 def differ(org_repo, org_ref, other_repo, other_ref, discovery_data=None):
     """
     General differ between branches, bookmarks or separate but releated 
     repositories
 
     :param org_repo:
     :type org_repo:
     :param org_ref:
     :type org_ref:
     :param other_repo:
     :type other_repo:
     :param other_ref:
     :type other_ref:
     """
 
     ignore_whitespace = False
     context = 3
     org_repo = org_repo.scm_instance._repo
     other_repo = other_repo.scm_instance._repo
     opts = diffopts(git=True, ignorews=ignore_whitespace, context=context)
     org_ref = org_ref[1]
     other_ref = other_ref[1]
 
     if org_repo != other_repo:
 
         common, incoming, rheads = discovery_data
         # create a bundle (uncompressed if other repo is not local)
         if other_repo.capable('getbundle'):
             # disable repo hooks here since it's just bundle !
             # patch and reset hooks section of UI config to not run any
             # hooks on fetching archives with subrepos
             for k, _ in other_repo.ui.configitems('hooks'):
                 other_repo.ui.setconfig('hooks', k, None)
 
             unbundle = other_repo.getbundle('incoming', common=common,
                                             heads=rheads)
 
             buf = io.BytesIO()
             while True:
                 chunk = unbundle._stream.read(1024*4)
                 if not chunk:
                     break
                 buf.write(chunk)
 
             buf.seek(0)
             unbundle._stream = buf
 
         class InMemoryBundleRepo(bundlerepository):
             def __init__(self, ui, path, bundlestream):
                 self._tempparent = None
                 localrepo.localrepository.__init__(self, ui, path)
                 self.ui.setconfig('phases', 'publish', False)
 
                 self.bundle = bundlestream
 
                 # dict with the mapping 'filename' -> position in the bundle
                 self.bundlefilespos = {}
 
         ui = make_ui('db')
         bundlerepo = InMemoryBundleRepo(ui, path=other_repo.root,
                                         bundlestream=unbundle)
         return ''.join(patch.diff(bundlerepo, node1=org_ref, node2=other_ref,
                                   opts=opts))
     else:
         return ''.join(patch.diff(org_repo, node1=org_ref, node2=other_ref,
                                   opts=opts))