middleware: drop gzipper middleware

It added unfortunate buffering, was only enabled for static_files=true, and
isn't available for TurboGears2.

If necessary, it is better to configure the web server to provide it.

docs/usage/performance.rst

@@ -1,112 +1,118 b''
 .. _performance:
 
 ================================
 Optimizing Kallithea performance
 ================================
 
 When serving a large amount of big repositories, Kallithea can start performing
 slower than expected. Because of the demanding nature of handling large amounts
 of data from version control systems, here are some tips on how to get the best
 performance.
 
 
 Fast storage
 ------------
 
 Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM
 is usually more important than a fast CPU.
 
 
 Caching
 -------
 
 Tweak beaker cache settings in the ini file. The actual effect of that is
 questionable.
 
 
 Database
 --------
 
 SQLite is a good option when having a small load on the system. But due to
 locking issues with SQLite, it is not recommended to use it for larger
 deployments.
 
 Switching to MySQL or PostgreSQL will result in an immediate performance
 increase. A tool like SQLAlchemyGrate_ can be used for migrating to another
 database platform.
 
 
 Horizontal scaling
 ------------------
 
 Scaling horizontally means running several Kallithea instances and let them
 share the load. That can give huge performance benefits when dealing with large
 amounts of traffic (many users, CI servers, etc.). Kallithea can be scaled
 horizontally on one (recommended) or multiple machines.
 
 It is generally possible to run WSGI applications multithreaded, so that
 several HTTP requests are served from the same Python process at once. That can
 in principle give better utilization of internal caches and less process
 overhead.
 
 One danger of running multithreaded is that program execution becomes much more
 complex; programs must be written to consider all combinations of events and
 problems might depend on timing and be impossible to reproduce.
 
 Kallithea can't promise to be thread-safe, just like the embedded Mercurial
 backend doesn't make any strong promises when used as Kallithea uses it.
 Instead, we recommend scaling by using multiple server processes.
 
 Web servers with multiple worker processes (such as ``mod_wsgi`` with the
 ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.
 
 In order to scale horizontally on multiple machines, you need to do the
 following:
 
     - Each instance's ``data`` storage needs to be configured to be stored on a
       shared disk storage, preferably together with repositories. This ``data``
       dir contains template caches, sessions, whoosh index and is used for
       task locking (so it is safe across multiple instances). Set the
       ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
       variables in each .ini file to a shared location across Kallithea instances
     - If using several Celery instances,
       the message broker should be common to all of them (e.g.,  one
       shared RabbitMQ server)
     - Load balance using round robin or IP hash, recommended is writing LB rules
       that will separate regular user traffic from automated processes like CI
       servers or build bots.
 
 
 Serve static files directly from the web server
 -----------------------------------------------
 
 With the default ``static_files`` ini setting, the Kallithea WSGI application
-will take care of serving the static files found in ``kallithea/public`` from
-the root of the application URL. While doing that, it will currently also
-apply buffering and compression of all the responses it is serving.
+will take care of serving the static files from ``kallithea/public/`` at the
+root of the application URL.
 
-The actual serving of the static files is unlikely to be a problem in a
-Kallithea setup. The buffering of responses is more likely to be a problem;
-large responses (clones or pulls) will have to be fully processed and spooled
-to disk or memory before the client will see any response.
+The actual serving of the static files is very fast and unlikely to be a
+problem in a Kallithea setup - the responses generated by Kallithea from
+database and repository content will take significantly more time and
+resources.
 
 To serve static files from the web server, use something like this Apache config
 snippet::
 
         Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/
         Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/
         Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/
         Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/
         Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/
 
 Then disable serving of static files in the ``.ini`` ``app:main`` section::
 
         static_files = false
 
 If using Kallithea installed as a package, you should be able to find the files
-under site-packages/kallithea, either in your Python installation or in your
+under ``site-packages/kallithea``, either in your Python installation or in your
 virtualenv. When upgrading, make sure to update the web server configuration
 too if necessary.
 
+It might also be possible to improve performance by configuring the web server
+to compress responses (served from static files or generated by Kallithea) when
+serving them. That might also imply buffering of responses - that is more
+likely to be a problem; large responses (clones or pulls) will have to be fully
+processed and spooled to disk or memory before the client will see any
+response. See the documentation for your web server.
+
 
 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate

kallithea/config/middleware.py

@@ -1,112 +1,110 b''
 # -*- coding: utf-8 -*-
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 """
     Pylons middleware initialization
 """
 
 from routes.middleware import RoutesMiddleware
 from paste.cascade import Cascade
 from paste.registry import RegistryManager
 from paste.urlparser import StaticURLParser
 from paste.deploy.converters import asbool
-from paste.gzipper import make_gzip_middleware
 
 from pylons.middleware import ErrorHandler, StatusCodeRedirect
 from pylons.wsgiapp import PylonsApp
 
 from kallithea.lib.middleware.simplehg import SimpleHg
 from kallithea.lib.middleware.simplegit import SimpleGit
 from kallithea.lib.middleware.https_fixup import HttpsFixup
 from kallithea.lib.middleware.sessionmiddleware import SecureSessionMiddleware
 from kallithea.config.environment import load_environment
 from kallithea.lib.middleware.wrapper import RequestWrapper
 
 
 def make_app(global_conf, full_stack=True, static_files=True, **app_conf):
     """Create a Pylons WSGI application and return it
 
     ``global_conf``
         The inherited configuration for this application. Normally from
         the [DEFAULT] section of the Paste ini file.
 
     ``full_stack``
         Whether or not this application provides a full WSGI stack (by
         default, meaning it handles its own exceptions and errors).
         Disable full_stack when this application is "managed" by
         another WSGI middleware.
 
     ``app_conf``
         The application's local configuration. Normally specified in
         the [app:<name>] section of the Paste ini file (where <name>
         defaults to main).
 
     """
     # Configure the Pylons environment
     config = load_environment(global_conf, app_conf)
 
     # The Pylons WSGI app
     app = PylonsApp(config=config)
 
     # Routing/Session/Cache Middleware
     app = RoutesMiddleware(app, config['routes.map'], use_method_override=False)
     app = SecureSessionMiddleware(app, config)
 
     # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)
     if asbool(config['pdebug']):
         from kallithea.lib.profiler import ProfilingMiddleware
         app = ProfilingMiddleware(app)
 
     if asbool(full_stack):
 
         from kallithea.lib.middleware.sentry import Sentry
         from kallithea.lib.middleware.errormator import Errormator
         if Errormator and asbool(config['app_conf'].get('errormator')):
             app = Errormator(app, config)
         elif Sentry:
             app = Sentry(app, config)
 
         # Handle Python exceptions
         app = ErrorHandler(app, global_conf, **config['pylons.errorware'])
 
         # Display error documents for 401, 403, 404 status codes (and
         # 500 when debug is disabled)
         # Note: will buffer the output in memory!
         if asbool(config['debug']):
             app = StatusCodeRedirect(app)
         else:
             app = StatusCodeRedirect(app, [400, 401, 403, 404, 500])
 
         # we want our low level middleware to get to the request ASAP. We don't
         # need any pylons stack middleware in them - especially no StatusCodeRedirect buffering
         app = SimpleHg(app, config)
         app = SimpleGit(app, config)
 
         # Enable https redirects based on HTTP_X_URL_SCHEME set by proxy
         if any(asbool(config.get(x)) for x in ['https_fixup', 'force_https', 'use_htsts']):
             app = HttpsFixup(app, config)
 
         app = RequestWrapper(app, config) # logging
 
     # Establish the Registry for this application
     app = RegistryManager(app) # thread / request-local module globals / variables
 
     if asbool(static_files):
         # Serve static files
         static_app = StaticURLParser(config['pylons.paths']['static_files'])
         app = Cascade([static_app, app])
-        app = make_gzip_middleware(app, global_conf, compress_level=1)
 
     app.config = config
 
     return app