Force Re-Indexing single repository

Often it's required to re-index whole repository because of some repository changes, or to remove some indexed secrets, or files. There's a special --repo-name= flag for the indexer that limits execution to a single repository. For example to force-reindex single repository such call can be made:

rhodecode-index --instance-name=enterprise-1 --force --repo-name=rhodecode-vcsserver

Removing repositories from index

The indexer automatically removes renamed repositories and builds index for new names. In case that you wish to remove indexed repository manually such call would allow that:

rhodecode-index --instance-name=enterprise-1 --remove-only --repo-name=rhodecode-vcsserver

Using search_mapping.ini file for advanced index rules

By default rhodecode-index runs for all repositories, all files with parsing limits defined by the CLI default arguments. You can change those limits by calling with different flags such as --max-filesize 2048kb or --repo-limit 10

For more advanced execution logic it's possible to use a configuration file that would define detailed rules which repositories and how should be indexed.

|RCT| provides an example index configuration file called :file:`search_mapping.ini`. This file is created by default during installation and is located at:

• :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`, using default |RCT|.
• :file:`~/venv/lib/python2.7/site-packages/rhodecode_tools/templates/mapping.ini`, when using virtualenv.

To Run the indexer with mapping file provide it using --mapping flag:

rhodecode-index --instance-name=enterprise-1 --mapping=/my/path/search_mapping.ini

Here's a detailed example of using :file:`search_mapping.ini` file.

[__DEFAULT__]
; Create index on commits data, and files data in this order. Available options
; are `commits`, `files`
index_types = commits,files

; Commit fetch limit. In what amount of chunks commits should be fetched
; via api and parsed. This allows server to transfer smaller chunks and be less loaded
commit_fetch_limit = 1000

; Commit process limit. Limit the number of commits indexer should fetch, and
; store inside the full text search index. eg. if repo has 2000 commits, and
; limit is 1000, on the first run it will process commits 0-1000 and on the
; second 1000-2000 commits. Help reduce memory usage, default is 50000
; (set -1 for unlimited)
commit_process_limit = 50000

; Limit of how many repositories each run can process, default is -1 (unlimited)
; in case of 1000s of repositories it's better to execute in chunks to not overload
; the server.
repo_limit = -1

; Default patterns for indexing files and content of files. Binary files
; are skipped by default.

; Add to index those comma separated files; globs syntax
; e.g index_files = *.py, *.c, *.h, *.js
index_files = *,

; Do not add to index those comma separated files, this excludes
; both search by name and content; globs syntax
; e.g index_files = *.key, *.sql, *.xml
skip_files = ,

; Add to index content of those comma separated files; globs syntax
; e.g index_files = *.h, *.obj
index_files_content = *,

; Do not add to index content of those comma separated files; globs syntax
; e.g index_files = *.exe, *.bin, *.log, *.dump
skip_files_content = ,

; Force rebuilding an index from scratch. Each repository will be rebuild from
; scratch with a global flag. Use --repo-name=NAME --force to rebuild single repo
force = false

; maximum file size that indexer will use, files above that limit are not going
; to have they content indexed.
; Possible options are KB (kilobytes), MB (megabytes), eg 1MB or 1024KB
max_filesize = 2MB


[__INDEX_RULES__]
; Ordered match rules for repositories. A list of all repositories will be fetched
; using API and this list will be filtered using those rules.
; Syntax for entry: `glob_pattern_OR_full_repo_name = 0 OR 1` where 0=exclude, 1=include
; When this ordered list is traversed first match will return the include/exclude marker
; For example:
;    upstream/binary_repo = 0
;    upstream/subrepo/xml_files = 0
;    upstream/* = 1
;    special-repo = 1
;    * = 0
; This will index all repositories under upstream/*, but skip upstream/binary_repo
; and upstream/sub_repo/xml_files, last * = 0 means skip all other matches

; Another example:
;    *-fork = 0
;    * = 1
; This will index all repositories, except those that have -fork as suffix.

rhodecode-vcsserver = 1
rhodecode-enterprise-ce = 1
upstream/mozilla/firefox-repo = 0
upstream/git-binaries = 0
upstream/* = 1
* = 0

; == EXPLICIT REPOSITORY INDEXING ==
; If defined this will skip using __INDEX_RULES__, and will not use API to fetch
; list of repositories, it will explicitly take names defined with [NAME] format and
; try to build the index, to build index just for repo_name_1 and special-repo use:
;    [repo_name_1]
;    [special-repo]

; == PER REPOSITORY CONFIGURATION ==
; This allows overriding the global configuration per repository.
; example to set specific file limit, and skip certain files for repository special-repo
;    [conf:special-repo]
;    max_filesize = 5mb
;    skip_files = *.xml, *.sql
;    index_types = files,

[conf:rhodecode-vcsserver]
index_types = files,
max_filesize = 5mb
skip_files = *.xml, *.sql
index_files = *.py, *.c, *.h, *.js

In case of 1000s of repositories it can be tricky to write the include/exclude rules at first. There's a special flag to test the mapping file rules and list repositories that would be indexed. Run the indexer with --show-matched-repos to list only the match rules:

rhodecode-index --instance-name=enterprise-1 --show-matched-repos --mapping=/my/path/search_mapping.ini