docs: update full text search indexing documentation

docs/admin/backup-restore.rst

@@ -1,151 +1,151 b''
 .. _backup-ref:
 
 Backup and Restore
 ==================
 
 *“The condition of any backup is unknown until a restore is attempted.”*
 `Schrödinger's Backup`_
 
 To snapshot an instance of |RCE|, and save its settings, you need to backup the
 following parts of the system at the same time.
 
 * The |repos| managed by the instance together with the stored Gists.
 * The |RCE| database.
 * Any configuration files or extensions that you've configured. In most
   cases it's only the :file:`rhodecode.ini` file.
 * Installer files such as those in `/opt/rhodecode` can be backed-up, however
   it's not required since in case of a recovery installer simply
   re-creates those.
 
 
 .. important::
 
    Ideally you should script all of these functions so that it creates a
    backup snapshot of your system at a particular timestamp and then run that
    script regularly.
 
 Backup Details
 --------------
 
 To backup the relevant parts of |RCE| required to restore your system, use
 the information in this section to identify what is important to you.
 
 Repository Backup
 ^^^^^^^^^^^^^^^^^
 
 To back up your |repos|, use the API to get a list of all |repos| managed,
 and then clone them to your backup location. This is the most safe backup option.
 Backing up the storage directory could potentially result in a backup of
 partially committed files or commits. (Backup taking place during a big push)
 As an alternative you could use a rsync or simple `cp` commands if you can
 ensure your instance is only in read-only mode or stopped at the moment.
 
 
 Use the ``get_repos`` method to list all your managed |repos|,
 and use the ``clone_uri`` information that is returned. See the :ref:`api`
 for more information. Be sure to keep the structure or repositories with their
 repository groups.
 
 .. important::
 
    This will not work for |svn| |repos|. Currently the only way to back up
    your |svn| |repos| is to make a copy of them.
 
    It is also important to note, that you can only restore the |svn| |repos|
    using the same version as they were saved with.
 
 Database Backup
 ^^^^^^^^^^^^^^^
 
 The instance database contains all the |RCE| permissions settings,
 and user management information. To backup your database,
 export it using the following appropriate example, and then move it to your
 backup location:
 
 .. code-block:: bash
 
    # For MySQL DBs
    $ mysqldump -u <uname> -p <pass> rhodecode_db_name > mysql-db-backup
    # MySQL restore command
    $ mysql -u <uname> -p <pass> rhodecode_db_name < mysql-db-backup
 
    # For PostgreSQL DBs
    $ PGPASSWORD=<pass> pg_dump --inserts -U <uname> -h localhost rhodecode_db_name > postgresql-db-backup
    # PosgreSQL restore
    $ PGPASSWORD=<pass> psql -U <uname> -h localhost -d rhodecode_db_name -1 -f postgresql-db-backup
 
    # For SQLite
    $ sqlite3 rhodecode.db ‘.dump’ > sqlite-db-backup
    # SQLite restore
    $ copy sqlite-db-backup rhodecode.db
 
 
 The default |RCE| SQLite database location is
 :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.db`
 
 If running MySQL or PostgreSQL databases, you will have configured these
 separately, for more information see :ref:`rhodecode-database-ref`
 
 Configuration File Backup
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Depending on your setup, you could have a number of configuration files that
 should be backed up. You may have some, or all of the configuration files
 listed in the :ref:`config-rce-files` section. Ideally you should back these
 up at the same time as the database and |repos|. It really depends on if you need
 the configuration file like logs, custom modules. We always recommend backing
 those up.
 
 Gist Backup
 ^^^^^^^^^^^
 
 To backup the gists on your |RCE| instance you usually have to backup the
 gist storage path. If this haven't been changed it's located inside
 :file:`.rc_gist_store` and the metadata in :file:`.rc_gist_metadata`.
 You can use the ``get_users`` and ``get_gists`` API methods to fetch the
 gists for each user on the instance.
 
 Extension Backups
 ^^^^^^^^^^^^^^^^^
 
 You should also backup any extensions added in the
 :file:`home/{user}/.rccontrol/{instance-id}/rcextensions` directory.
 
 Full-text Search Backup
 ^^^^^^^^^^^^^^^^^^^^^^^
 
 You may also have full text search set up, but the index can be rebuild from
 re-imported |repos| if necessary. You will most likely want to backup your
-:file:`mapping.ini` file if you've configured that. For more information, see
+:file:`search_mapping.ini` file if you've configured that. For more information, see
 the :ref:`indexing-ref` section.
 
 Restoration Steps
 -----------------
 
 To restore an instance of |RCE| from its backed up components, to a fresh
 system use the following steps.
 
 1. Install a new instance of |RCE| using sqlite option as database.
 2. Restore your database.
 3. Once installed, replace you backed up the :file:`rhodecode.ini` with your
    backup version. Ensure this file points to the restored
    database, see the :ref:`config-database` section.
 4. Restart |RCE| and remap and rescan your |repos| to verify filesystem access,
    see the :ref:`remap-rescan` section.
 
 
 Post Restoration Steps
 ^^^^^^^^^^^^^^^^^^^^^^
 
 Once you have restored your |RCE| instance to basic functionality, you can
 then work on restoring any specific setup changes you had made.
 
-* To recreate the |RCE| index, use the backed up :file:`mapping.ini` file if
+* To recreate the |RCE| index, use the backed up :file:`search_mapping.ini` file if
   you had made changes and rerun the indexer. See the
   :ref:`indexing-ref` section for details.
 * To reconfigure any extensions, copy the backed up extensions into the
   :file:`/home/{user}/.rccontrol/{instance-id}/rcextensions` and also specify
   any custom hooks if necessary. See the :ref:`extensions-hooks-ref` section for
   details.
 
 .. _Schrödinger's Backup: http://novabackup.novastor.com/blog/schrodingers-backup-good-bad-backup/

docs/admin/config-files-overview.rst

@@ -1,74 +1,74 b''
 .. _config-files:
 
 Configuration Files Overview
 ============================
 
 |RCE| and |RCC| have a number of different configuration files. The following
 is a brief explanation of each, and links to their associated configuration
 sections.
 
 .. rst-class:: dl-horizontal
 
     \- **rhodecode.ini**
         Default location:
         :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
 
         This is the main |RCE| configuration file and controls much of its
         default behaviour. It is also used to configure certain customer
         settings. Here are some of the most common reasons to make changes to
         this file.
 
         * :ref:`config-database`
         * :ref:`set-up-mail`
         * :ref:`increase-gunicorn`
         * :ref:`x-frame`
 
-    \- **mapping.ini**
+    \- **search_mapping.ini**
         Default location:
-        :file:`/home/{user}/.rccontrol/{instance-id}/mapping.ini`
+        :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`
 
         This file is used to control the |RCE| indexer. It comes configured
         to index your instance. To change the default configuration, see
         :ref:`advanced-indexing`.
 
     \- **vcsserver.ini**
         Default location:
         :file:`/home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini`
 
         The VCS Server handles the connection between your |repos| and |RCE|.
         See the :ref:`vcs-server` section for configuration options and more
         detailed information.
 
     \- **supervisord.ini**
         Default location:
         :file:`/home/{user}/.rccontrol/supervisor/supervisord.ini`
 
         |RCC| uses Supervisor to monitor and manage installed instances of
         |RCE| and the VCS Server. |RCC| will manage this file completely,
         unless you install |RCE| in self-managed mode. For more information,
         see the :ref:`Supervisor Setup<control:supervisor-setup>` section.
 
     \- **.rccontrol.ini**
         Default location: :file:`/home/{user}/.rccontrol.ini`
 
         This file contains the instances that |RCC| starts at boot, which is all
         by default, but for more information, see
         the :ref:`Manually Start At Boot <control:set-start-boot>` section.
 
     \- **.rhoderc**
         Default location: :file:`/home/{user}/.rhoderc`
 
         This file is used by the |RCE| API when accessing an instance from a
         remote machine. The API checks this file for connection and
         authentication details. For more details, see the :ref:`config-rhoderc`
         section.
 
     \- **MANIFEST**
         Default location: :file:`/home/{user}/.rccontrol/cache/MANIFEST`
 
         |RCC| uses this file to source the latest available builds from the
         secure RhodeCode download channels. The only reason to mess with this file
         is if you need to do an offline installation,
         see the :ref:`Offline Installation<control:offline-installer-ref>`
         instructions, otherwise |RCC| will completely manage this file.
 

docs/admin/indexing.rst

@@ -1,276 +1,363 b''
 .. _indexing-ref:
 
 Full-text Search
 ----------------
 
 By default RhodeCode is configured to use `Whoosh`_ to index |repos| and
 provide full-text search.
 
-|RCE| also provides support for `Elasticsearch`_ as a backend for scalable
-search. See :ref:`enable-elasticsearch` for details.
+|RCE| also provides support for `Elasticsearch 6`_ as a backend more for advanced
+and scalable search. See :ref:`enable-elasticsearch` for details.
 
 Indexing
 ^^^^^^^^
 
-To run the indexer you need to use an |authtoken| with admin rights to all
-|repos|.
+To run the indexer you need to have an |authtoken| with admin rights to all |repos|.
 
 To index new content added, you have the option to set the indexer up in a
 number of ways, for example:
 
-* Call the indexer via a cron job. We recommend running this nightly,
-  unless you need everything indexed immediately.
-* Set the indexer to infinitely loop and reindex as soon as it has run its
-  cycle.
+* Call the indexer via a cron job. We recommend running this once at night.
+  In case you need everything indexed immediately it's possible to index few
+  times during the day.
+* Set the indexer to infinitely loop and reindex as soon as it has run its previous cycle.
 * Hook the indexer up with your CI server to reindex after each push.
 
 The indexer works by indexing new commits added since the last run. If you
 wish to build a brand new index from scratch each time,
 use the ``force`` option in the configuration file.
 
 .. important::
 
    You need to have |RCT| installed, see :ref:`install-tools`. Since |RCE|
-   3.5.0 they are installed by default.
+   3.5.0 they are installed by default and available with community/enterprise installations.
 
 To set up indexing, use the following steps:
 
 1. :ref:`config-rhoderc`, if running tools remotely.
 2. :ref:`run-index`
 3. :ref:`set-index`
 4. :ref:`advanced-indexing`
 
 .. _config-rhoderc:
 
 Configure the ``.rhoderc`` File
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+.. note::
+
+    Optionally it's possible to use indexer without the ``.rhoderc``. Simply instead of
+    executing with `--instance-name=enterprise-1` execute providing the host and token
+    directly: `--api-host=http://127.0.0.1:10000 --api-key=<auth token goes here>
+
+
 |RCT| uses the :file:`/home/{user}/.rhoderc` file for connection details
 to |RCE| instances. If this file is not automatically created,
 you can configure it using the following example. You need to configure the
 details for each instance you want to index.
 
 .. code-block:: bash
 
     # Check the instance details
     # of the instance you want to index
     $ rccontrol status
 
-     - NAME: enterprise-1
-     - STATUS: RUNNING
-     - TYPE: Momentum
-     - VERSION: 1.5.0
-     - URL: http://127.0.0.1:10000
+    - NAME: enterprise-1
+    - STATUS: RUNNING
+    - TYPE: Enterprise
+    - VERSION: 4.1.0
+    - URL: http://127.0.0.1:10003
 
 To get your API Token, on the |RCE| interface go to
 :menuselection:`username --> My Account --> Auth tokens`
 
 .. code-block:: ini
 
     # Configure .rhoderc with matching details
     # This allows the indexer to connect to the instance
     [instance:enterprise-1]
     api_host = http://127.0.0.1:10000
     api_key = <auth token goes here>
-    repo_dir = /home/<username>/repos
+
 
 .. _run-index:
 
 Run the Indexer
 ^^^^^^^^^^^^^^^
 
-Run the indexer using the following command, and specify the instance you
-want to index:
+Run the indexer using the following command, and specify the instance you want to index:
 
 .. code-block:: bash
 
-   # From inside a virtualevv
-   (venv)$ rhodecode-index --instance-name=enterprise-1
-
    # Using default installation
    $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
        --instance-name=enterprise-1
 
    # Using a custom mapping file
    $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
        --instance-name=enterprise-1 \
-       --mapping=/home/user/.rccontrol/enterprise-1/mapping.ini
+       --mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini
+
+   # Using a custom mapping file and invocation without ``.rhoderc``
+   $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
+       --api-host=http://rhodecodecode.myserver.com --api-key=xxxxx \
+       --mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini
+
+   # From inside a virtualev on your local machine or CI server.
+   (venv)$ rhodecode-index --instance-name=enterprise-1
+
 
 .. note::
 
    In case of often indexing the index may become fragmented. Most often a result of that
    is error about `too many open files`. To fix this indexer needs to be executed with
    --optimize flag. E.g `rhodecode-index --instance-name=enterprise-1 --optimize`
    This should be executed regularly, once a week is recommended.
 
 
 .. _set-index:
 
 Schedule the Indexer
 ^^^^^^^^^^^^^^^^^^^^
 
 To schedule the indexer, configure the crontab file to run the indexer inside
 your |RCT| virtualenv using the following steps.
 
 1. Open the crontab file, using ``crontab -e``.
 2. Add the indexer to the crontab, and schedule it to run as regularly as you
    wish.
 3. Save the file.
 
 .. code-block:: bash
 
     $ crontab -e
 
     # The virtualenv can be called using its full path, so for example you can
     # put this example into the crontab
 
     # Run the indexer daily at 4am using the default mapping settings
     * 4 * * * /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \
     --instance-name=enterprise-1
 
     # Run the indexer every Sunday at 3am using default mapping
     * 3 * * 0 /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \
     --instance-name=enterprise-1
 
     # Run the indexer every 15 minutes
     # using a specially configured mapping file
     */15 * * * * ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
        --instance-name=enterprise-4 \
-       --mapping=/home/user/.rccontrol/enterprise-4/mapping.ini
+       --mapping=/home/user/.rccontrol/enterprise-4/search_mapping.ini
 
 .. _advanced-indexing:
 
 Advanced Indexing
 ^^^^^^^^^^^^^^^^^
 
-|RCT| indexes based on the :file:`mapping.ini` file. To configure your index,
-you can specify different options in this file. The default location is:
+
+Force Re-Indexing single repository
++++++++++++++++++++++++++++++++++++
+
+Often it's required to re-index whole repository because of some repository changes,
+or to remove some indexed secrets, or files. There's a special `--repo-name=` flag
+for the indexer that limits execution to a single repository. For example to force-reindex
+single repository such call can be made::
+
+    rhodecode-index --instance-name=enterprise-1 --force --repo-name=rhodecode-vcsserver
+
+
+Removing repositories from index
+++++++++++++++++++++++++++++++++
+
+The indexer automatically removes renamed repositories and builds index for new names.
+In case that you wish to remove indexed repository manually such call would allow that::
 
-* :file:`/home/{user}/.rccontrol/{instance-id}/mapping.ini`, using default
-  |RCT|.
+    rhodecode-index --instance-name=enterprise-1 --remove-only --repo-name=rhodecode-vcsserver
+
+
+Using search_mapping.ini file for advanced index rules
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+By default rhodecode-index runs for all repositories, all files with parsing limits
+defined by the CLI default arguments. You can change those limits by calling with
+different flags such as `--max-filesize 2048kb` or `--repo-limit 10`
+
+For more advanced execution logic it's possible to use a configuration file that
+would define detailed rules which repositories and how should be indexed.
+
+|RCT| provides an example index configuration file called :file:`search_mapping.ini`.
+This file is created by default during installation and is located at:
+
+* :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`, using default |RCT|.
 * :file:`~/venv/lib/python2.7/site-packages/rhodecode_tools/templates/mapping.ini`,
   when using ``virtualenv``.
 
 .. note::
 
-    If you need to create the :file:`mapping.ini` file, use the |RCT|
-    ``rhodecode-index --create-mapping path/to/file`` API call. For details,
-    see the :ref:`tools-cli` section.
-
-The indexer runs in a random order to prevent a failing |repo| from stopping
-a build. To configure different indexing scenarios, set the following options
-inside the :file:`mapping.ini` and specify the altered file using the
-``--mapping`` option.
+    If you need to create the :file:`search_mapping.ini` file manually, use the |RCT|
+    ``rhodecode-index --create-mapping path/to/search_mapping.ini`` API call.
+    For details, see the :ref:`tools-cli` section.
 
-* ``index_files`` : Index the specified file types.
-* ``skip_files`` : Do not index the specified file types.
-* ``index_files_content`` : Index the content of the specified file types.
-* ``skip_files_content`` : Do not index the content of the specified files.
-* ``force`` : Create a fresh index on each run.
-* ``max_filesize`` : Files larger than the set size will not be indexed.
-* ``commit_parse_limit`` : Set the batch size when indexing commit messages.
-  Set to a lower number to lessen memory load.
-* ``repo_limit`` : Set the maximum number or |repos| indexed per run.
-* ``[INCLUDE]`` : Set |repos| you want indexed. This takes precedent over
-  ``[EXCLUDE]``.
-* ``[EXCLUDE]`` : Set |repos| you do not want indexed. Exclude can be used to
-  not index branches, forks, or log |repos|.
+To Run the indexer with mapping file provide it using `--mapping` flag::
 
-At the end of the file you can specify conditions for specific |repos| that
-will override the default values. To configure your indexer,
-use the following example :file:`mapping.ini` file.
+    rhodecode-index --instance-name=enterprise-1 --mapping=/my/path/search_mapping.ini
+
+
+Here's a detailed example of using :file:`search_mapping.ini` file.
 
 .. code-block:: ini
 
     [__DEFAULT__]
-    # default patterns for indexing files and content of files.
-    # Binary files are skipped by default.
+    ; Create index on commits data, and files data in this order. Available options
+    ; are `commits`, `files`
+    index_types = commits,files
+
+    ; Commit fetch limit. In what amount of chunks commits should be fetched
+    ; via api and parsed. This allows server to transfer smaller chunks and be less loaded
+    commit_fetch_limit = 1000
 
-    # Index python and markdown files
-    index_files = *.py, *.md
+    ; Commit process limit. Limit the number of commits indexer should fetch, and
+    ; store inside the full text search index. eg. if repo has 2000 commits, and
+    ; limit is 1000, on the first run it will process commits 0-1000 and on the
+    ; second 1000-2000 commits. Help reduce memory usage, default is 50000
+    ; (set -1 for unlimited)
+    commit_process_limit = 50000
 
-    # Do not index these file types
-    skip_files = *.svg, *.log, *.dump, *.txt
+    ; Limit of how many repositories each run can process, default is -1 (unlimited)
+    ; in case of 1000s of repositories it's better to execute in chunks to not overload
+    ; the server.
+    repo_limit = -1
 
-    # Index both file types and their content
-    index_files_content = *.cpp, *.ini, *.py
+    ; Default patterns for indexing files and content of files. Binary files
+    ; are skipped by default.
+
+    ; Add to index those comma separated files; globs syntax
+    ; e.g index_files = *.py, *.c, *.h, *.js
+    index_files = *,
 
-    # Index file names, but not file content
-    skip_files_content = *.svg,
+    ; Do not add to index those comma separated files, this excludes
+    ; both search by name and content; globs syntax
+    ; e.g index_files = *.key, *.sql, *.xml
+    skip_files = ,
 
-    # Force rebuilding an index from scratch. Each repository will be rebuild
-    # from scratch with a global flag. Use local flag to rebuild single repos
+    ; Add to index content of those comma separated files; globs syntax
+    ; e.g index_files = *.h, *.obj
+    index_files_content = *,
+
+    ; Do not add to index content of those comma separated files; globs syntax
+    ; e.g index_files = *.exe, *.bin, *.log, *.dump
+    skip_files_content = ,
+
+    ; Force rebuilding an index from scratch. Each repository will be rebuild from
+    ; scratch with a global flag. Use --repo-name=NAME --force to rebuild single repo
     force = false
 
-    # Do not index files larger than 385KB
-    max_filesize = 385KB
+    ; maximum file size that indexer will use, files above that limit are not going
+    ; to have they content indexed.
+    ; Possible options are KB (kilobytes), MB (megabytes), eg 1MB or 1024KB
+    max_filesize = 2MB
 
-    # Limit commit indexing to 500 per batch
-    commit_parse_limit = 500
-
-    # Limit each index run to 25 repos
-    repo_limit = 25
 
-    # __INCLUDE__ is more important that __EXCLUDE__.
-
-    [__INCLUDE__]
-    # Include all repos with these names
+    [__INDEX_RULES__]
+    ; Ordered match rules for repositories. A list of all repositories will be fetched
+    ; using API and this list will be filtered using those rules.
+    ; Syntax for entry: `glob_pattern_OR_full_repo_name = 0 OR 1` where 0=exclude, 1=include
+    ; When this ordered list is traversed first match will return the include/exclude marker
+    ; For example:
+    ;    upstream/binary_repo = 0
+    ;    upstream/subrepo/xml_files = 0
+    ;    upstream/* = 1
+    ;    special-repo = 1
+    ;    * = 0
+    ; This will index all repositories under upstream/*, but skip upstream/binary_repo
+    ; and upstream/sub_repo/xml_files, last * = 0 means skip all other matches
 
-    docs/* = 1
-    lib/* = 1
-
-    [__EXCLUDE__]
-    # Do not include the following repo in index
+    ; Another example:
+    ;    *-fork = 0
+    ;    * = 1
+    ; This will index all repositories, except those that have -fork as suffix.
 
-    dev-docs/* = 1
-    legacy-repos/* = 1
-    *-dev/* = 1
+    rhodecode-vcsserver = 1
+    rhodecode-enterprise-ce = 1
+    upstream/mozilla/firefox-repo = 0
+    upstream/git-binaries = 0
+    upstream/* = 1
+    * = 0
 
-    # Each repo that needs special indexing is a separate section below.
-    # In each section set the options to override the global configuration
-    # parameters above.
-    # If special settings are not configured, the global configuration values
-    # above are inherited. If no special repositories are
-    # defined here RhodeCode will use the API to ask for all repositories
+    ; == EXPLICIT REPOSITORY INDEXING ==
+    ; If defined this will skip using __INDEX_RULES__, and will not use API to fetch
+    ; list of repositories, it will explicitly take names defined with [NAME] format and
+    ; try to build the index, to build index just for repo_name_1 and special-repo use:
+    ;    [repo_name_1]
+    ;    [special-repo]
 
-    # For this repo use different settings
-    [special-repo]
-    commit_parse_limit = 20,
-    skip_files = *.idea, *.xml,
+    ; == PER REPOSITORY CONFIGURATION ==
+    ; This allows overriding the global configuration per repository.
+    ; example to set specific file limit, and skip certain files for repository special-repo
+    ;    [conf:special-repo]
+    ;    max_filesize = 5mb
+    ;    skip_files = *.xml, *.sql
+    ;    index_types = files,
 
-    # For another repo use different settings
-    [another-special-repo]
-    index_files = *,
-    max_filesize = 800MB
-    commit_parse_limit = 20000
+    [conf:rhodecode-vcsserver]
+    index_types = files,
+    max_filesize = 5mb
+    skip_files = *.xml, *.sql
+    index_files = *.py, *.c, *.h, *.js
+
+
+In case of 1000s of repositories it can be tricky to write the include/exclude rules at first.
+There's a special flag to test the mapping file rules and list repositories that would
+be indexed. Run the indexer with `--show-matched-repos` to list only the match rules::
+
+    rhodecode-index --instance-name=enterprise-1 --show-matched-repos --mapping=/my/path/search_mapping.ini
+
 
 .. _enable-elasticsearch:
 
 Enabling Elasticsearch
 ^^^^^^^^^^^^^^^^^^^^^^
 
+Elasticsearch is available in EE edition only. It provides much scalable and more advanced
+search capabilities. While Whoosh is fine for upto 1-2GB of data beyond that amount of
+data it starts slowing down, and can cause other problems. Elasticsearch 6 also provides
+much more advanced query language allowing advanced filtering by file paths, extensions
+OR statements, ranges etc. Please check query language examples in the search field for
+some advanced query language usage.
+
+
 1. Open the :file:`rhodecode.ini` file for the instance you wish to edit. The
    default location is
    :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
 2. Find the search configuration section:
 
 .. code-block:: ini
 
     ###################################
     ## SEARCH INDEXING CONFIGURATION ##
     ###################################
 
     search.module = rhodecode.lib.index.whoosh
     search.location = %(here)s/data/index
 
 and change it to:
 
 .. code-block:: ini
 
     search.module = rc_elasticsearch
-    search.location = http://localhost:9200/
+    search.location = http://localhost:9200
+    ## specify Elastic Search version, 6 for latest or 2 for legacy
+    search.es_version = 6
+
+where ``search.location`` points to the elasticsearch server
+by default running on port 9200.
 
-where ``search.location`` points to the elasticsearch server.
+Index invocation also needs change. Please provide --es-version= and
+--engine-location= parameters to define elasticsearch server location and it's version.
+For example::
+
+    rhodecode-index --instace-name=enterprise-1 --es-version=6 --engine-location=http://localhost:9200
+
 
 .. _Whoosh: https://pypi.python.org/pypi/Whoosh/
-.. _Elasticsearch: https://www.elastic.co/
  No newline at end of file
+.. _Elasticsearch 6: https://www.elastic.co/
  No newline at end of file

docs/admin/system-overview.rst

@@ -1,171 +1,171 b''
 .. _system-overview-ref:
 
 System Overview
 ===============
 
 Latest Version
 --------------
 
 * |release| on Unix and Windows systems.
 
 System Architecture
 -------------------
 
 The following diagram shows a typical production architecture.
 
 .. image:: ../images/architecture-diagram.png
   :align: center
 
 Supported Operating Systems
 ---------------------------
 
 Linux
 ^^^^^
 
 * Ubuntu 14.04
 * CentOS 6.2 and 7
 * Debian 7.8
 * RedHat Fedora
 * Arch Linux
 * SUSE Linux
 
 Windows
 ^^^^^^^
 
 * Windows Vista Ultimate 64bit
 * Windows 7 Ultimate 64bit
 * Windows 8 Professional 64bit
 * Windows 8.1 Enterprise 64bit
 * Windows Server 2008 64bit
 * Windows Server 2008-R2 64bit
 * Windows Server 2012 64bit
 
 Supported Databases
 -------------------
 
 * SQLite
 * MySQL
 * MariaDB
 * PostgreSQL
 
 Supported Browsers
 ------------------
 
 * Chrome
 * Safari
 * Firefox
 * Internet Explorer 10 & 11
 
 System Requirements
 -------------------
 
 |RCE| performs best on machines with ultra-fast hard disks. Generally disk
 performance is more important than CPU performance. In a corporate production
 environment handling 1000s of users and |repos| you should deploy on a 12+
 core 64GB RAM server. In short, the more RAM the better.
 
 
 For example:
 
  - for team of 1 - 5 active users you can run on 1GB RAM machine with 1CPU
  - above 250 active users, |RCE| needs at least 8GB of memory.
    Number of CPUs is less important, but recommended to have at least 2-3 CPUs
 
 
 .. _config-rce-files:
 
 Configuration Files
 -------------------
 
 * :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
-* :file:`/home/{user}/.rccontrol/{instance-id}/mapping.ini`
+* :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`
 * :file:`/home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini`
 * :file:`/home/{user}/.rccontrol/supervisor/supervisord.ini`
 * :file:`/home/{user}/.rccontrol.ini`
 * :file:`/home/{user}/.rhoderc`
 * :file:`/home/{user}/.rccontrol/cache/MANIFEST`
 
 For more information, see the :ref:`config-files` section.
 
 Log Files
 ---------
 
 * :file:`/home/{user}/.rccontrol/{instance-id}/enterprise.log`
 * :file:`/home/{user}/.rccontrol/{vcsserver-id}/vcsserver.log`
 * :file:`/home/{user}/.rccontrol/supervisor/supervisord.log`
 * :file:`/tmp/rccontrol.log`
 * :file:`/tmp/rhodecode_tools.log`
 
 Storage Files
 -------------
 
 * :file:`/home/{user}/.rccontrol/{instance-id}/data/index/{index-file.toc}`
 * :file:`/home/{user}/repos/.rc_gist_store`
 * :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.db`
 * :file:`/opt/rhodecode/store/{unique-hash}`
 
 Default Repositories Location
 -----------------------------
 
 * :file:`/home/{user}/repos`
 
 Connection Methods
 ------------------
 
 * HTTPS
 * SSH
 * |RCE| API
 
 Internationalization Support
 ----------------------------
 
 Currently available in the following languages, see `Transifex`_ for the
 latest details. If you want a new language added, please contact us. To
 configure your language settings, see the :ref:`set-lang` section.
 
 .. hlist::
 
   * Belorussian
   * Chinese
   * French
   * German
   * Italian
   * Japanese
   * Portuguese
   * Polish
   * Russian
   * Spanish
 
 Licencing Information
 ---------------------
 
 * See licencing information `here`_
 
 Peer-to-peer Failover Support
 -----------------------------
 
 * Yes
 
 Additional Binaries
 -------------------
 
 * Yes, see :ref:`rhodecode-nix-ref` for full details.
 
 Remote Connectivity
 -------------------
 
 * Available
 
 Executable Files
 ----------------
 
 Windows: :file:`RhodeCode-installer-{version}.exe`
 
 Deprecated Support
 ------------------
 
 - Internet Explorer 8 support deprecated since version 3.7.0.
 - Internet Explorer 9 support deprecated since version 3.8.0.
 
 .. _here: https://rhodecode.com/licenses/
 .. _Transifex: https://www.transifex.com/projects/p/RhodeCode/

docs/tools/tools-cli.rst

@@ -1,578 +1,578 b''
 .. _tools-cli:
 
 |RCT| CLI
 ---------
 
 The commands available with |RCT| can be split into three categories:
 
 - Remotely executable commands that can be run from your local machine once you
   have your connection details to |RCE| configured.
 - Locally executable commands the can be run on the server to carry out
   general maintenance.
 - Local configuration commands used to help set up your |RCT| configuration.
 
 
 rhodecode-tools
 ---------------
 
 Use |RCT| to setup automation, run the indexer, and install extensions for
 your |RCE| instances. Options:
 
 .. rst-class:: dl-horizontal
 
     \ - -apihost <api_host>
         Set the API host value.
 
     \ - -apikey <apikey_value>
         Set the API key value.
 
     \-c, - -config <config_file>
         Create a configuration file. The default file is created
         in ``~/.rhoderc``
 
     \ - -save-config
         Save the configuration file.
 
     \ - -show-config
         Show the current configuration values.
 
     \ - -format {json,pretty}
         Set the formatted representation.
 
 Example usage:
 
 .. code-block:: bash
 
    $ rhodecode-tools --apikey=key --apihost=http://rhodecode.server \
        --save-config
 
 rhodecode-api
 -------------
 
 The RhodeCode API lets you connect to |RCE| and carry out management tasks from a
 remote machine, for more information about the API, see the :ref:`api`. To
 pass arguments on the command-line use the ``method:option`` syntax.
 
 Example usage:
 
 .. code-block:: bash
 
     # Run the get_repos API call and sample output
     $ rhodecode-api --instance-name=enterprise-1 create_repo \
         repo_name:brand-new repo_type:hg description:repo-desc
 
     {
       "error": null,
       "id": 1110,
       "result": {
         "msg": "Created new repository `brand-new`",
         "success": true,
         "task": null
       }
     }
 
 Options:
 
 .. rst-class:: dl-horizontal
 
     \ - -api-cache-only
         Requires a cache to be present when running this call
 
     \ - -api-cache-rebuild
         Replaces existing cached values with new ones from server
 
     \ - -api-cache <PATH>
         Use a special cache dir to read responses from instead of the server
 
     \ - -api-cert-verify
          Verify the endpoint ssl certificate
 
     \ - -api-cert <PATH>
          Path to alternate CA bundle.
 
     \ - -apihost <api_host>
         Set the API host value.
 
     \ - -apikey <apikey_value>
         Set the API key value.
 
     \ - -instance-name <instance-id>
         Set the instance name
 
     \-I, - -install-dir <DIR>
         Location of application instances
 
     \-c, - -config <.rhoderc-file>
         Location of the :file:`.rhoderc`
 
     \-F, - -format {json,pretty}
         Set the formatted representation.
 
     \-h, - -help
         Show help messages.
 
     \-v, - -verbose
         Enable verbose messaging
 
 rhodecode-cleanup-gists
 -----------------------
 
 Use this to delete gists within |RCE|. Options:
 
 .. rst-class:: dl-horizontal
 
     \-c, - -config <config_file>
         Set the file path to the configuration file. The default file is
         :file:`/home/{user}/.rhoderc`
 
     \ - -corrupted
         Remove gists with corrupted metadata.
 
     \ - -dont-ask
         Remove gists without asking for confirmation.
 
     \-h, - -help
         Show help messages. current configuration values.
 
     \ - -instance-name <instance-id>
         Set the instance name.
 
     \-R, - -repo-dir
         Set the repository file path.
 
     \ - -version
         Display your |RCT| version.
 
 Example usage:
 
 .. code-block:: bash
 
     # Clean up gists related to an instance
     $ rhodecode-cleanup-gists --instance-name=enterprise-1
     Scanning for gists in /home/brian/repos/.rc_gist_store...
     preparing to remove [3] found gists
 
     # Clean up corrupted gists in an instance
     $ rhodecode-cleanup-gists --instance-name=enterprise-1 --corrupted
     Scanning for gists in /home/brian/repos/.rc_gist_store...
     preparing to remove [2] found gists
     the following gists will be archived:
       * EXPIRED: BAD METADATA        | /home/brian/repos/.rc_gist_store/5
       * EXPIRED: BAD METADATA        | /home/brian/repos/.rc_gist_store/8FtC
     are you sure you want to archive them? [y/N]: y
     removing gist /home/brian/repos/.rc_gist_store/5
     removing gist /home/brian/repos/.rc_gist_store/8FtCKdcbRKmEvRzTVsEt
 
 rhodecode-cleanup-repos
 -----------------------
 
 Use this to manage |repos| and |repo| groups within |RCE|. Options:
 
 .. rst-class:: dl-horizontal
 
     \-c, - -config <config_file>
         Set the file path to the configuration file. The default file is
         :file:`/home/{user}/.rhoderc`.
 
     \-h, - -help
         Show help messages. current configuration values.
 
     \ - -interactive
         Enable an interactive prompt for each repository when deleting.
 
     \ - -include-groups
         Remove repository groups.
 
     \ - -instance-name <instance-id>
         Set the instance name.
 
     \ - -list-only
         Display repositories selected for deletion.
 
     \ - -older-than <str>
         Delete repositories older that a specified time.
         You can use the following suffixes; d for days, h for hours,
         m for minutes, s for seconds.
 
     \-R, - -repo-dir
         Set the repository file path
 
 Example usage:
 
 .. code-block:: bash
 
     # Cleaning up repos using tools installed with RCE 350 and above
     $ ~/.rccontrol/enterprise-4/profile/bin/rhodecode-cleanup-repos \
         --instance-name=enterprise-4 --older-than=1d
     Scanning for repositories in /home/brian/repos...
     preparing to remove [2] found repositories older than 1 day, 0:00:00 (1d)
 
     the following repositories will be deleted completely:
     * REMOVED: 2015-08-05 00:23:18 | /home/brian/repos/rm__20150805_002318_831
     * REMOVED: 2015-08-04 01:22:10 | /home/brian/repos/rm__20150804_012210_336
     are you sure you want to remove them? [y/N]:
 
     # Clean up repos older than 1 year
     # If using virtualenv and pre RCE 350 tools installation
     (venv)$ rhodecode-cleanup-repos --instance-name=enterprise-1 \
         --older-than=365d
 
     Scanning for repositories in /home/brian/repos...
     preparing to remove [343] found repositories older than 365 days
 
     # clean up repos older than 3 days
     # If using virtualenv and pre RCE 350 tools installation
     (venv)$ rhodecode-cleanup-repos --instance-name=enterprise-1 \
         --older-than=3d
     Scanning for repositories in /home/brian/repos...
     preparing to remove [3] found repositories older than 3 days
 
 .. _tools-config:
 
 rhodecode-config
 ----------------
 
 Use this to create or update a |RCE| configuration file on the local machine.
 
 .. rst-class:: dl-horizontal
 
     \- -filename </path/to/config_file>
         Set the file path to the |RCE| configuration file.
 
     \- -show-defaults
         Display the defaults set in the |RCE| configuration file.
 
     \- -update
         Update the configuration with the new settings passed on the command
         line.
 
 .. code-block:: bash
 
     # Create a new config file
     $ rhodecode-config --filename=dev.ini
     Wrote new config file in /Users/user/dev.ini
 
     # Update config value for given section:
     $ rhodecode-config --update --filename=prod.ini [handler_console]level=INFO
 
     $ rhodecode-config --filename=dev.ini --show-defaults
     lang=en
     cpu_number=4
     uuid=<function <lambda> at 0x10d86ac08>
     license_token=ff1e-aa9c-bb66-11e5
     host=127.0.0.1
     here=/Users/brian
     error_aggregation_service=None
     database_url=sqlite:///%(here)s/rhodecode.db?timeout=30
     git_path=git
     http_server=waitress
     port=5000
 
 .. _tools-rhodecode-extensions:
 
 rhodecode-extensions
 --------------------
 
 The `rcextensions` since version 4.14 are now shipped together with |RCE| please check
 the using :ref:`integrations-rcextensions` section.
 
 
 rhodecode-gist
 --------------
 
 Use this to create, list, show, or delete gists within |RCE|. Options:
 
 .. rst-class:: dl-horizontal
 
     \ - -api-cache-only
         Requires a cache to be present when running this call
 
     \ - -api-cache-rebuild
         Replaces existing cached values with new ones from server
 
     \ - -api-cache PATH
         Use a special cache dir to read responses from instead of the server
 
     \ - -api-cert-verify
          Verify the endpoint ssl certificate
 
     \ - -api-cert PATH
          Path to alternate CA bundle.
 
     \ - -apihost <api_host>
         Set the API host value.
 
     \ - -apikey <apikey_value>
         Set the API key value.
 
     \-c, - -config <config_file>
         Create a configuration file.
         The default file is created in :file:`~/.rhoderc`
 
     \ - -create <gistname>
         create the gist
 
     \-d, - -description <str>
         Set gist description
 
     \ - -delete <gistid>
         Delete the gist
 
     \-f, - -file
         Specify the filename The file extension will enable syntax highlighting.
 
     \-F, - -format {json,pretty}
         Set the formatted representation.
 
     \ - -help
         Show help messages.
 
     \-I, - -install-dir <DIR>
         Location of application instances
 
     \ - -instance-name <instance-id>
         Set the instance name.
 
     \ - -list
         Display instance gists.
 
     \-l, --lifetime <minutes>
         Set the gist lifetime. The default value is (-1) forever
 
     \ - -show <gistname>
         Show the content of the gist
 
     \-o, - -open
         After creating Gist open it in browser
 
     \-p, - -private
         Create a private gist
 
     \ - -version
          Display your |RCT| version.
 
 Example usage:
 
 .. code-block:: bash
 
     # List the gists in an instance
     (venv)brian@ubuntu:~$ rhodecode-gist --instance-name=enterprise-1 list
     {
       "error": null,
       "id": 7102,
       "result": [
         {
           "access_id": "2",
           "content": null,
           "created_on": "2015-01-19T12:52:26.494",
           "description": "A public gust",
           "expires": -1.0,
           "gist_id": 2,
           "type": "public",
           "url": "http://127.0.0.1:10003/_admin/gists/2"
         },
         {
           "access_id": "7gs6BsSEC4pKUEPLz5AB",
           "content": null,
           "created_on": "2015-01-19T11:27:40.812",
           "description": "Gist testing API",
           "expires": -1.0,
           "gist_id": 1,
           "type": "private",
           "url": "http://127.0.0.1:10003/_admin/gists/7gs6BsSEC4pKUEPLz5AB"
         }
       ]
     }
 
     # delete a particular gist
     # You use the access_id to specify the gist to delete
     (venv)brian@ubuntu:~$ rhodecode-gist delete 2  --instance-name=enterprise-1
     {
       "error": null,
       "id": 6284,
       "result": {
         "gist": null,
         "msg": "deleted gist ID:2"
       }
     }
 
     # cat a file and pipe to new gist
     # This is if you are using virtualenv
     (venv)$ cat ~/.rhoderc | rhodecode-gist --instance-name=enterprise-1 \
         -d '.rhoderc copy' create
 
     {
       "error": null,
       "id": 5374,
       "result": {
         "gist": {
           "access_id": "7",
           "content": null,
           "created_on": "2015-01-26T11:31:58.774",
           "description": ".rhoderc copy",
           "expires": -1.0,
           "gist_id": 7,
           "type": "public",
           "url": "http://127.0.0.1:10003/_admin/gists/7"
         },
         "msg": "created new gist"
       }
     }
 
     # Cat a file and pipe to gist
     # in RCE 3.5.0 tools and above
     $ cat ~/.rhoderc | ~/.rccontrol/{instance-id}/profile/bin/rhodecode-gist \
        --instance-name=enterprise-4 -d '.rhoderc copy' create
     {
       "error": null,
       "id": 9253,
       "result": {
         "gist": {
           "access_id": "4",
           "acl_level": "acl_public",
           "content": null,
           "created_on": "2015-08-20T05:54:11.250",
           "description": ".rhoderc copy",
           "expires": -1.0,
           "gist_id": 4,
           "modified_at": "2015-08-20T05:54:11.250",
           "type": "public",
           "url": "http://127.0.0.1:10000/_admin/gists/4"
         },
         "msg": "created new gist"
       }
     }
 
 
 rhodecode-index
 ---------------
 
 More detailed information regarding setting up the indexer is available in
 the :ref:`indexing-ref` section. Options:
 
 .. rst-class:: dl-horizontal
 
     \ - -api-cache-only
         Requires a cache to be present when running this call
 
     \ - -api-cache-rebuild
         Replaces existing cached values with new ones from server
 
     \ - -api-cache PATH
         Use a special cache dir to read responses from instead of the server
 
     \ - -api-cert-verify
          Verify the endpoint ssl certificate
 
     \ - -api-cert PATH
          Path to alternate CA bundle.
 
     \ - -apihost <api_host>
         Set the API host value.
 
     \ - -apikey <apikey_value>
         Set the API key value.
 
     \-c, --config <config_file>
         Create a configuration file.
         The default file is created in :file:`~/.rhoderc`
 
     \ - -create-mapping <PATH>
          Creates an example mapping configuration for indexer.
 
     \-F, - -format {json,pretty}
          Set the formatted representation.
 
     \-h, - -help
          Show help messages.
 
     \ - -instance-name <instance-id>
         Set the instance name
 
     \-I, - -install-dir <DIR>
         Location of application instances
 
     \-m, - -mapping <file_name>
          Parse the output to the .ini mapping file.
 
     \ - -optimize
          Optimize index for performance by amalgamating multiple index files
          into one. Greatly increases incremental indexing speed.
 
     \-R, - -repo-dir <DIRECTORY>
          Location of repositories
 
     \ - -source <PATH>
          Use a special source JSON file to feed the indexer
 
     \ - -version
          Display your |RCT| version.
 
 Example usage:
 
 .. code-block:: bash
 
     # Run the indexer
     $ ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
         --instance-name=enterprise-4
 
-    # Run indexer based on mapping.ini file
+    # Run indexer based on search_mapping.ini file
     # This is using pre-350 virtualenv
     (venv)$ rhodecode-index --instance-name=enterprise-1
 
     # Index from the command line without creating
     # the .rhoderc file
     $ rhodecode-index --apikey=key --apihost=http://rhodecode.server \
         --instance-name=enterprise-2 --save-config
 
     # Create the indexing mapping file
     $ ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
-        --create-mapping mapping.ini --instance-name=enterprise-4
+        --create-mapping search_mapping.ini --instance-name=enterprise-4
 
 .. _tools-rhodecode-list-instance:
 
 rhodecode-list-instances
 ------------------------
 
 Use this command to list the instance details configured in the
 :file:`~/.rhoderc` file.
 
 .. code-block:: bash
 
    $ .rccontrol/enterprise-1/profile/bin/rhodecode-list-instances
    [instance:production] - Config only
    API-HOST: https://some.url.com
    API-KEY:  some.auth.token
 
    [instance:development] - Config only
    API-HOST: http://some.ip.address
    API-KEY:  some.auth.token
 
 
 .. _tools-setup-config:
 
 rhodecode-setup-config
 ----------------------
 
 Use this command to create the ``~.rhoderc`` file required by |RCT| to access
 remote instances.
 
 .. rst-class:: dl-horizontal
 
     \- -instance-name <name>
         Specify the instance name in the :file:`~/.rhoderc`
 
     \api_host <hostname>
         Create a configuration file. The default file is created
         in ``~/.rhoderc``
 
     \api_key <auth-token>
         Create a configuration file. The default file is created
         in ``~/.rhoderc``
 
 
 .. code-block:: bash
 
     (venv)$ rhodecode-setup-config --instance-name=tea api_host=URL api_key=xyz
     Config not found under /Users/username/.rhoderc, creating a new one
     Wrote new configuration into /Users/username/.rhoderc