release: merge back stable branch into default

docs/release-notes/release-notes-4.3.1.rst

@@ -0,0 +1,15 b''
+|RCE| 4.3.1 |RNS|
+-----------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2016-08-23
+
+Fixes
+^^^^^
+
+- Core: fixed database session cleanups. This will make sure RhodeCode can
+  function correctly after database server problems. Fixes #4173, refs #4166
+- Diffs: limit the file context to ~1mln lines. Fixes #4184, also make sure
+  this doesn't trigger Integer overflow for msgpack.
  No newline at end of file

.hgtags

@@ -6,3 +6,4 b' 41c87da28a179953df86061d817bc35533c66dd2'
 baaf9f5bcea3bae0ef12ae20c8b270482e62abb6 v4.2.0
 32a70c7e56844a825f61df496ee5eaf8c3c4e189 v4.2.1
 fa695cdb411d294679ac081d595ac654e5613b03 v4.3.0
+0e4dc11b58cad833c513fe17bac39e6850edf959 v4.3.1

docs/admin/svn-http.rst

@@ -3,16 +3,21 b''
 |svn| With Write Over HTTP
 --------------------------
 
-To use |svn| with write access, the currently supported method is over HTTP.
-This requires you to configure your local machine so that it can access your
-|RCE| instance.
+To use |svn| with read/write support over the |svn| protocol, you have to
+configure HTTP |svn| backend.
 
 Prerequisites
 ^^^^^^^^^^^^^
 
-- Enable lab setting on your |RCE| instance, see :ref:`lab-settings`.
-- You need to install the following tools on your local machine: ``Apache`` and
-  ``mod_dav_svn``. Use the following Ubuntu as an example.
+- Enable HTTP support inside labs setting on your |RCE| instance,
+  see :ref:`lab-settings`.
+- You need to install the following tools on the machine that is running an
+  instance of |RCE|:
+  ``Apache HTTP Server`` and
+  ``mod_dav_svn``.
+
+
+Using Ubuntu Distribution as an example you can run:
 
 .. code-block:: bash
 
@@ -32,41 +37,80 b' Configuring Apache Setup'
    It is recommended to run Apache on a port other than 80, due to possible
    conflicts with other HTTP servers like nginx. To do this, set the
    ``Listen`` parameter in the ``/etc/apache2/ports.conf`` file, for example
-   ``Listen 8090``
+   ``Listen 8090``.
+
+
+.. warning::
 
-   It is also recommended to run apache as the same user as |RCE|, otherwise
-   permission issues could occur. To do this edit the ``/etc/apache2/envvars``
+   Make sure your Apache instance which runs the mod_dav_svn module is
+   only accessible by RhodeCode. Otherwise everyone is able to browse
+   the repositories or run subversion operations (checkout/commit/etc.).
+
+It is also recommended to run apache as the same user as |RCE|, otherwise
+permission issues could occur. To do this edit the ``/etc/apache2/envvars``
 
    .. code-block:: apache
 
-      export APACHE_RUN_USER=ubuntu
-      export APACHE_RUN_GROUP=ubuntu
+      export APACHE_RUN_USER=rhodecode
+      export APACHE_RUN_GROUP=rhodecode
 
 1. To configure Apache, create and edit a virtual hosts file, for example
-   :file:`/etc/apache2/sites-available/default.conf`, or create another
-   virtual hosts file and add a location section inside the
-   ``<VirtualHost>`` section.
+   :file:`/etc/apache2/sites-available/default.conf`. Below is an example
+   how to use one with auto-generated config ```mod_dav_svn.conf```
+   from configured |RCE| instance.
 
 .. code-block:: apache
 
-    <Location />
-        DAV svn
-        # Must be explicit path, relative not supported
-        SVNParentPath /PATH/TO/REPOSITORIES
-        SVNListParentPath On
-        Allow from all
-        Order allow,deny
-    </Location>
+    <VirtualHost *:8080>
+        ServerAdmin rhodecode-admin@localhost
+        DocumentRoot /var/www/html
+        ErrorLog ${'${APACHE_LOG_DIR}'}/error.log
+        CustomLog ${'${APACHE_LOG_DIR}'}/access.log combined
+        Include /home/user/.rccontrol/enterprise-1/mod_dav_svn.conf
+    </VirtualHost>
 
-.. note::
-
-   Once configured, check that you can see the list of repositories on your
-   |RCE| instance.
 
 2. Go to the :menuselection:`Admin --> Settings --> Labs` page, and
    enable :guilabel:`Proxy Subversion HTTP requests`, and specify the
    :guilabel:`Subversion HTTP Server URL`.
 
+3. Open the |RCE| configuration file,
+   :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
+
+4. Add the following configuration option in the ``[app:main]``
+   section if you don't have it yet.
+
+   This enable mapping of created |RCE| repo groups into special |svn| paths.
+   Each time a new repository group will be created the system will update
+   the template file, and create new mapping. Apache web server needs to be
+   reloaded to pick up the changes on this file.
+   It's recommended to add reload into a crontab so the changes can be picked
+   automatically once someone creates an repository group inside RhodeCode.
+
+
+.. code-block:: ini
+
+    ##############################################
+    ### Subversion proxy support (mod_dav_svn) ###
+    ##############################################
+    ## Enable or disable the config file generation.
+    svn.proxy.generate_config = true
+    ## Generate config file with `SVNListParentPath` set to `On`.
+    svn.proxy.list_parent_path = true
+    ## Set location and file name of generated config file.
+    svn.proxy.config_file_path = %(here)s/mod_dav_svn.conf
+    ## File system path to the directory containing the repositories served by
+    ## RhodeCode.
+    svn.proxy.parent_path_root = /path/to/repo_store
+    ## Used as a prefix to the <Location> block in the generated config file. In
+    ## most cases it should be set to `/`.
+    svn.proxy.location_root = /
+
+
+This would create a special template file called ```mod_dav_svn.conf```. We
+used that file path in the apache config above inside the Include statement.
+
+
 Using |svn|
 ^^^^^^^^^^^
 

docs/release-notes/release-notes.rst

@@ -9,6 +9,7 b' Release Notes'
 .. toctree::
    :maxdepth: 1
 
+   release-notes-4.3.1.rst
    release-notes-4.3.0.rst
    release-notes-4.2.1.rst
    release-notes-4.2.0.rst

rhodecode/config/middleware.py

@@ -39,6 +39,7 b' from routes.middleware import RoutesMidd'
 import routes.util
 
 import rhodecode
+from rhodecode.model import meta
 from rhodecode.config import patches
 from rhodecode.config.routing import STATIC_FILE_PREFIX
 from rhodecode.config.environment import (
@@ -159,6 +160,10 b' def make_pyramid_app(global_config, **se'
     pyramid_app = config.make_wsgi_app()
     pyramid_app = wrap_app_in_wsgi_middlewares(pyramid_app, config)
     pyramid_app.config = config
+
+    # creating the app uses a connection - return it after we are done
+    meta.Session.remove()
+
     return pyramid_app
 
 
@@ -381,7 +386,25 b' def wrap_app_in_wsgi_middlewares(pyramid'
         pyramid_app = make_gzip_middleware(
             pyramid_app, settings, compress_level=1)
 
-    return pyramid_app
+
+    # this should be the outer most middleware in the wsgi stack since
+    # middleware like Routes make database calls
+    def pyramid_app_with_cleanup(environ, start_response):
+        try:
+            return pyramid_app(environ, start_response)
+        finally:
+            # Dispose current database session and rollback uncommitted
+            # transactions.
+            meta.Session.remove()
+
+            # In a single threaded mode server, on non sqlite db we should have
+            # '0 Current Checked out connections' at the end of a request,
+            # if not, then something, somewhere is leaving a connection open
+            pool = meta.Base.metadata.bind.engine.pool
+            log.debug('sa pool status: %s', pool.status())
+
+
+    return pyramid_app_with_cleanup
 
 
 def sanitize_settings_and_apply_defaults(settings):

rhodecode/lib/diffs.py

@@ -40,12 +40,17 b' from rhodecode.lib.utils2 import safe_un'
 
 log = logging.getLogger(__name__)
 
+# define max context, a file with more than this numbers of lines is unusable
+# in browser anyway
+MAX_CONTEXT = 1024 * 1014
+
 
 class OPS(object):
     ADD = 'A'
     MOD = 'M'
     DEL = 'D'
 
+
 def wrap_to_table(str_):
     return '''<table class="code-difftable">
                 <tr class="line no-comment">
@@ -57,8 +62,8 b' def wrap_to_table(str_):'
 
 
 def wrapped_diff(filenode_old, filenode_new, diff_limit=None, file_limit=None,
-                show_full_diff=False, ignore_whitespace=True, line_context=3,
-                enable_comments=False):
+                 show_full_diff=False, ignore_whitespace=True, line_context=3,
+                 enable_comments=False):
     """
     returns a wrapped diff into a table, checks for cut_off_limit for file and
     whole diff and presents proper message
@@ -79,8 +84,9 b' def wrapped_diff(filenode_old, filenode_'
         f_gitdiff = get_gitdiff(filenode_old, filenode_new,
                                 ignore_whitespace=ignore_whitespace,
                                 context=line_context)
-        diff_processor = DiffProcessor(f_gitdiff, format='gitdiff', diff_limit=diff_limit,
-                            file_limit=file_limit, show_full_diff=show_full_diff)
+        diff_processor = DiffProcessor(
+            f_gitdiff, format='gitdiff', diff_limit=diff_limit,
+            file_limit=file_limit, show_full_diff=show_full_diff)
         _parsed = diff_processor.prepare()
 
         diff = diff_processor.as_html(enable_comments=enable_comments)
@@ -115,6 +121,10 b' def get_gitdiff(filenode_old, filenode_n'
     """
     # make sure we pass in default context
     context = context or 3
+    # protect against IntOverflow when passing HUGE context
+    if context > MAX_CONTEXT:
+        context = MAX_CONTEXT
+
     submodules = filter(lambda o: isinstance(o, SubModuleNode),
                         [filenode_new, filenode_old])
     if submodules:
@@ -190,7 +200,8 b' class DiffProcessor(object):'
     # used for inline highlighter word split
     _token_re = re.compile(r'()(&gt;|&lt;|&amp;|\W+?)')
 
-    def __init__(self, diff, format='gitdiff', diff_limit=None, file_limit=None, show_full_diff=True):
+    def __init__(self, diff, format='gitdiff', diff_limit=None,
+                 file_limit=None, show_full_diff=True):
         """
         :param diff: A `Diff` object representing a diff from a vcs backend
         :param format: format of diff passed, `udiff` or `gitdiff`

rhodecode/lib/hooks_daemon.py

@@ -30,6 +30,7 b' import Pyro4'
 import pylons
 import rhodecode
 
+from rhodecode.model import meta
 from rhodecode.lib import hooks_base
 from rhodecode.lib.utils2 import (
     AttributeDict, safe_str, get_routes_generator_for_server_url)
@@ -64,7 +65,10 b' class HooksHttpHandler(BaseHTTPRequestHa'
 
     def _call_hook(self, method, extras):
         hooks = Hooks()
-        result = getattr(hooks, method)(extras)
+        try:
+            result = getattr(hooks, method)(extras)
+        finally:
+            meta.Session.remove()
         return result
 
     def log_message(self, format, *args):
@@ -261,6 +265,7 b' class Hooks(object):'
             }
         finally:
             pylons.url._pop_object()
+            meta.Session.remove()
 
         return {
             'status': result.status,

rhodecode/lib/middleware/simplevcs.py

@@ -406,8 +406,11 b' class SimpleVCS(object):'
                 yield chunk
         finally:
             # invalidate cache on push
-            if action == 'push':
-                self._invalidate_cache(repo_name)
+            try:
+                if action == 'push':
+                    self._invalidate_cache(repo_name)
+            finally:
+                meta.Session.remove()
 
     def _get_repository_name(self, environ):
         """Get repository name out of the environmnent

rhodecode/tweens.py

@@ -40,40 +40,35 b' def pylons_compatibility_tween_factory(h'
         from pyramid. For example while rendering an old template that uses the
         'c' or 'h' objects. This tween sets up the needed pylons globals.
         """
-        try:
-            config = rhodecode.CONFIG
-            environ = request.environ
-            session = request.session
-            session_key = (config['pylons.environ_config']
-                           .get('session', 'beaker.session'))
+        config = rhodecode.CONFIG
+        environ = request.environ
+        session = request.session
+        session_key = (config['pylons.environ_config']
+                       .get('session', 'beaker.session'))
 
-            # Setup pylons globals.
-            pylons.config._push_object(config)
-            pylons.request._push_object(request)
-            pylons.session._push_object(session)
-            environ[session_key] = session
-            pylons.url._push_object(URLGenerator(config['routes.map'],
-                                                 environ))
+        # Setup pylons globals.
+        pylons.config._push_object(config)
+        pylons.request._push_object(request)
+        pylons.session._push_object(session)
+        environ[session_key] = session
+        pylons.url._push_object(URLGenerator(config['routes.map'],
+                                             environ))
 
-            # TODO: Maybe we should use the language from pyramid.
-            translator = _get_translator(config.get('lang'))
-            pylons.translator._push_object(translator)
-
-            # Get the rhodecode auth user object and make it available.
-            auth_user = get_auth_user(environ)
-            request.user = auth_user
-            environ['rc_auth_user'] = auth_user
+        # TODO: Maybe we should use the language from pyramid.
+        translator = _get_translator(config.get('lang'))
+        pylons.translator._push_object(translator)
 
-            # Setup the pylons context object ('c')
-            context = ContextObj()
-            context.rhodecode_user = auth_user
-            attach_context_attributes(context, request)
-            pylons.tmpl_context._push_object(context)
-            return handler(request)
-        finally:
-            # Dispose current database session and rollback uncommitted
-            # transactions.
-            meta.Session.remove()
+        # Get the rhodecode auth user object and make it available.
+        auth_user = get_auth_user(environ)
+        request.user = auth_user
+        environ['rc_auth_user'] = auth_user
+
+        # Setup the pylons context object ('c')
+        context = ContextObj()
+        context.rhodecode_user = auth_user
+        attach_context_attributes(context, request)
+        pylons.tmpl_context._push_object(context)
+        return handler(request)
 
     return pylons_compatibility_tween