docs: updated section on performance, scaling, ssl support

docs/admin/gunicorn-ssl-support.rst

@@ -0,0 +1,50 b''
+.. _gunicorn-ssl-support:
+
+
+Gunicorn SSL support
+--------------------
+
+
+:term:`Gunicorn` wsgi server allows users to use HTTPS connection directly
+without a need to use HTTP server like Nginx or Apache. To Configure
+SSL support directly with :term:`Gunicorn` you need to simply add the key
+and certificate paths to your configuration file.
+
+1. Open the :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
+2. In the ``[server:main]`` section, add two new variables
+   called `certfile` and `keyfile`.
+
+.. code-block:: ini
+
+    [server:main]
+    host = 127.0.0.1
+    port = 10002
+    use = egg:gunicorn#main
+    workers = 1
+    threads = 1
+    proc_name = RhodeCodeEnterprise
+    worker_class = sync
+    max_requests = 1000
+    timeout = 3600
+    # adding ssl support
+    certfile = /home/ssl/my_server_com.pem
+    keyfile = /home/ssl/my_server_com.key
+
+4. Save your changes.
+5. Restart your |RCE| instance, using the following command:
+
+.. code-block:: bash
+
+    $ rccontrol restart enterprise-1
+
+After this is enabled you can *only* access your instances via https://
+protocol. Check out more docs here `Gunicorn SSL Docs`_
+
+.. note::
+
+   This change only can be applied to |RCE|. VCSServer doesn't support SSL
+   and should be only used with http protocol. Because only |RCE| is available
+   externally all communication will still be over SSL even without VCSServer
+   SSL enabled.
+
+.. _Gunicorn SSL Docs: http://docs.gunicorn.org/en/stable/settings.html#ssl

docs/admin/system-admin.rst

@@ -1,30 +1,31 b''
 .. _rhodecode-admin-ref:
 
 System Administration
 =====================
 
 The following are the most common system administration tasks.
 
 .. only:: latex
 
     * :ref:`vcs-server`
     * :ref:`apache-ws-ref`
     * :ref:`nginx-ws-ref`
     * :ref:`rhodecode-tuning-ref`
     * :ref:`indexing-ref`
     * :ref:`rhodecode-reset-ref`
 
 .. toctree::
 
     config-files-overview
     vcs-server
     svn-http
+    gunicorn-ssl-support
     apache-config
     nginx-config
     backup-restore
     tuning-rhodecode
     indexing
     reset-information
     enable-debug
     admin-tricks
     cleanup-cmds

docs/admin/tuning-gunicorn.rst

@@ -1,111 +1,124 b''
 .. _increase-gunicorn:
 
 Increase Gunicorn Workers
 -------------------------
 
-.. important::
+
+|RCE| comes with `Gunicorn`_ packaged in its Nix environment.
+Gunicorn is a Python WSGI HTTP Server for UNIX.
 
-   If you increase the number of :term:`Gunicorn` workers, you also need to
-   increase the threadpool size of the VCS Server. The recommended size is
-   6 times the number of Gunicorn workers. To set this, see
-   :ref:`vcs-server-config-file`.
+To improve |RCE| performance you can increase the number of `Gunicorn`_  workers.
+This allows to handle more connections concurently, and provide better
+responsiveness and performance.
 
-|RCE| comes with `Gunicorn`_ packaged in its Nix environment. To improve
-performance you can increase the number of workers. To do this, use the
-following steps:
+By default during installation |RCC|  tries to detect how many CPUs are
+available in the system, and set the number workers based on that information.
+However sometimes it's better to manually set the number of workers.
+
+To do this, use the following steps:
 
 1. Open the :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
 2. In the ``[server:main]`` section, increase the number of Gunicorn
    ``workers`` using the following formula :math:`(2 * Cores) + 1`.
 
 .. code-block:: ini
 
-    [server:main]
-    host = 127.0.0.1
-    port = 10002
     use = egg:gunicorn#main
-    workers = 1
-    threads = 1
-    proc_name = RhodeCodeEnterprise
+    ## Sets the number of process workers. You must set `instance_id = *`
+    ## when this option is set to more than one worker, recommended
+    ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
+    ## The `instance_id = *` must be set in the [app:main] section below
+    workers = 4
+    ## process name
+    proc_name = rhodecode
+    ## type of worker class, one of sync, gevent
+    ## recommended for bigger setup is using of of other than sync one
     worker_class = sync
+    ## The maximum number of simultaneous clients. Valid only for Gevent
+    #worker_connections = 10
+    ## max number of requests that worker will handle before being gracefully
+    ## restarted, could prevent memory leaks
     max_requests = 1000
-    timeout = 3600
+    max_requests_jitter = 30
+    ## amount of time a worker can spend with handling a request before it
+    ## gets killed and restarted. Set to 6hrs
+    timeout = 21600
 
 3. In the ``[app:main]`` section, set the ``instance_id`` property to ``*``.
 
 .. code-block:: ini
 
     # In the [app:main] section
     [app:main]
     # You must set `instance_id = *`
     instance_id = *
 
-4. Save your changes.
-5. Restart your |RCE| instance, using the following command:
+4. Change the VCSServer workers too. Open the
+   :file:`home/{user}/.rccontrol/{instance-id}/vcsserver.ini` file.
+
+5. In the ``[server:main]`` section, increase the number of Gunicorn
+   ``workers`` using the following formula :math:`(2 * Cores) + 1`.
+
+.. code-block:: ini
+
+    ## run with gunicorn --log-config vcsserver.ini --paste vcsserver.ini
+    use = egg:gunicorn#main
+    ## Sets the number of process workers. Recommended
+    ## value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers
+    workers = 4
+    ## process name
+    proc_name = rhodecode_vcsserver
+    ## type of worker class, currently `sync` is the only option allowed.
+    worker_class = sync
+    ## The maximum number of simultaneous clients. Valid only for Gevent
+    #worker_connections = 10
+    ## max number of requests that worker will handle before being gracefully
+    ## restarted, could prevent memory leaks
+    max_requests = 1000
+    max_requests_jitter = 30
+    ## amount of time a worker can spend with handling a request before it
+    ## gets killed and restarted. Set to 6hrs
+    timeout = 21600
+
+6. Save your changes.
+7. Restart your |RCE| instances, using the following command:
 
 .. code-block:: bash
 
-    $ rccontrol restart enterprise-1
+    $ rccontrol restart '*'
+
+
+Gunicorn Gevent Backend
+-----------------------
 
-If you scale across different machines, each |RCM| instance
-needs to store its data on a shared disk, preferably together with your
-|repos|. This data directory contains template caches, a whoosh index,
-and is used for task locking to ensure safety across multiple instances.
-To do this, set the following properties in the :file:`rhodecode.ini` file to
-set the shared location across all |RCM| instances.
+Gevent is an asynchronous worker type for Gunicorn. It allows accepting multiple
+connections on a single `Gunicorn`_  worker. This means you can handle 100s
+of concurrent clones, or API calls using just few workers. A setting called
+`worker_connections` defines on how many connections each worker can
+handle using `Gevent`.
+
+
+To enable `Gevent` on |RCE| do the following:
+
+
+1. Open the :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
+2. In the ``[server:main]`` section, change `worker_class` for Gunicorn.
+
 
 .. code-block:: ini
 
-    cache_dir = /file/path           # set to shared location
-    search.location = /file/path           # set to shared location
+    ## type of worker class, one of sync, gevent
+    ## recommended for bigger setup is using of of other than sync one
+    worker_class = gevent
+    ## The maximum number of simultaneous clients. Valid only for Gevent
+    worker_connections = 30
 
-    ####################################
-    ###         BEAKER CACHE        ####
-    ####################################
-    beaker.cache.data_dir = /file/path       # set to shared location
-    beaker.cache.lock_dir = /file/path       # set to shared location
+
+.. note::
+
+    `Gevent` is currently only supported for Enterprise/Community instances.
+    VCSServer doesn't yet support gevent.
 
 
 
-Gunicorn SSL support
---------------------
-
-
-:term:`Gunicorn` wsgi server allows users to use HTTPS connection directly
-without a need to use HTTP server like Nginx or Apache. To Configure
-SSL support directly with :term:`Gunicorn` you need to simply add the key
-and certificate paths to your configuration file.
-
-1. Open the :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
-2. In the ``[server:main]`` section, add two new variables
-   called `certfile` and `keyfile`.
-
-.. code-block:: ini
-
-    [server:main]
-    host = 127.0.0.1
-    port = 10002
-    use = egg:gunicorn#main
-    workers = 1
-    threads = 1
-    proc_name = RhodeCodeEnterprise
-    worker_class = sync
-    max_requests = 1000
-    timeout = 3600
-    # adding ssl support
-    certfile = /home/ssl/my_server_com.pem
-    keyfile = /home/ssl/my_server_com.key
-
-4. Save your changes.
-5. Restart your |RCE| instance, using the following command:
-
-.. code-block:: bash
-
-    $ rccontrol restart enterprise-1
-
-After this is enabled you can *only* access your instances via https://
-protocol. Check out more docs here `Gunicorn SSL Docs`_
-
-
 .. _Gunicorn: http://gunicorn.org/
-.. _Gunicorn SSL Docs: http://docs.gunicorn.org/en/stable/settings.html#ssl

docs/admin/tuning-scale-horizontally.rst

@@ -1,51 +1,58 b''
 .. _scale-horizontal:
 
 Scale Horizontally
 ------------------
 
+|RCE| is built in a way it support horizontal scaling across multiple machines.
+There are two main pre-requisites for that:
+
+- Shared storage that each machine can access.
+- Shared DB connection across machines.
+
+
 Horizontal scaling means adding more machines or workers into your pool of
 resources. Horizontally scaling |RCE| gives a huge performance increase,
 especially under large traffic scenarios with a high number of requests. This
 is very beneficial when |RCE| is serving many users simultaneously,
 or if continuous integration servers are automatically pulling and pushing code.
 
-To horizontally scale |RCE| you should use the following steps:
+
+If you scale across different machines, each |RCM| instance
+needs to store its data on a shared disk, preferably together with your
+|repos|. This data directory contains template caches, a full text search index,
+and is used for task locking to ensure safety across multiple instances.
+To do this, set the following properties in the :file:`rhodecode.ini` file to
+set the shared location across all |RCM| instances.
+
+.. code-block:: ini
+
+    cache_dir = /shared/path/caches                 # set to shared location
+    search.location = /shared/path/search_index     # set to shared location
 
-1. In the :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file,
-   set ``instance_id = *``. This enables |RCE| to use multiple nodes.
-2. Define the number of worker threads using the formula
-   :math:`(2 * Cores) + 1`. For example 4 CPU cores would lead to
-   :math:`(2 * 4) + 1 = 9` workers. In some cases it's ok to increase number of
-   workers even beyond this formula. Generally the more workers, the more
-   simultaneous connections the system can handle.
+    ####################################
+    ###         BEAKER CACHE        ####
+    ####################################
+    beaker.cache.data_dir = /shared/path/data       # set to shared location
+    beaker.cache.lock_dir = /shared/path/lock       # set to shared location
+
+
+.. note::
+
+    If you use custom caches such as `beaker.cache.auth_plugins.` it's recommended
+    to set it to the memcached/redis or database backend so it can be shared
+    across machines.
+
 
 It is recommended to create another dedicated |RCE| instance to handle
 traffic from build farms or continuous integration servers.
 
 .. note::
 
    You should configure your load balancing accordingly. We recommend writing
    load balancing rules that will separate regular user traffic from
    automated process traffic like continuous servers or build bots.
 
-If you scale across different machines, each |RCE| instance needs to store
-its data on a shared disk, preferably together with your repositories. This
-data directory contains template caches, a whoosh index,
-and is used for task locking to ensure safety across multiple instances. To
-do this, set the following properties in the
-:file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file to set
-the shared location across all |RCE| instances.
-
-.. code-block:: ini
-
-         cache_dir = /file/path               # set to shared directory location
-         search.location = /file/path               # set to shared directory location
-         beaker.cache.data_dir = /file/path   # set to shared directory location
-         beaker.cache.lock_dir = /file/path   # set to shared directory location
-
 .. note::
 
      If Celery is used on each instance then you should run separate Celery
      instances, but the message broker should be the same for all of them.
-     This excludes one RabbitMQ shared server.
-

docs/contributing/api.rst

@@ -1,60 +1,58 b''
 
 ===================
 CONTRIBUTING TO API
 ===================
 
 
 
 Naming conventions
 ==================
 
 We keep the calls in the form ``{verb}_{noun}``.
 
 
 
 Change and Deprecation
 ======================
 
-API deprecation is documented in the section :ref:`deprecated` together with
+API deprecation is documented in the section `deprecated` together with
 other notes about deprecated parts of the application.
 
 
 Deprecated API calls
 --------------------
 
-- Make sure to add them into the section :ref:`deprecated`.
-
 - Use `deprecated` inside of the call docstring to make our users aware of the
   deprecation::
 
     .. deprecated:: 1.2.3
 
        Use `new_call_name` instead to fetch this information.
 
 - Make sure to log on level `logging.WARNING` a message that the API call or
   specific parameters are deprecated.
 
 - If possible return deprecation information inside of the result from the API
   call. Use the attribute `_warning_` to contain a message.
 
 
 Changed API calls
 -----------------
 
 - If the change is significant, consider to use `versionchanged` in the
   docstring::
 
     .. versionchanged:: 1.2.3
 
        Optional explanation if reasonable.
 
 
 Added API calls
 ---------------
 
 - Use `versionadded` to document since which version this API call is
   available::
 
     .. versionadded:: 1.2.3
 
        Optional explanation if reasonable.