RhodeCode Cluster

RhodeCode Cluster is a multi-node highly-scalable setup to run
RhodeCode, Edge Router(Traefik + SSL) and Metrics Stack (Prometheus, Loki, Grafana) and
all its additional components in single environment using Docker.

Using a docker-compose this setup creates following services for RhodeCode:

Edge-Router:

• Traefik, Edge Router, SSL termination etc

Core Services:

• Database (defaults to PostgreSQL)
• Redis, acts as cache and queue exchange
• ChannelStream - live websocket communications
• Nginx (static/channelstream) proxy serving RhodeCode APPS and channelstream communication
• Elasticsearch (full text search backend)

RhodeCode

• RhodeCode CE/EE
• VCSServer for GIT/SVN/HG support
• SSH Server for cloning over SSH
• SVN webserver for HTTP support over SVN
• Celery workers for asynchronous tasks
• Celery beat for automation tasks

Metrics

• Loki, logs aggregation
• Grafana, Metrics Dashboard
• Promethues, Metrics time-series
• statsd-exporter - statsd to promethues bridge
• node-exporter - machine stats
• promtail - log scraping

Pre requisites

To Run this stack Docker engine and Docker Compose needs to be installed on the host machine.
Visit docker site and install docker (min version 20.10) and docker compose:

• https://docs.docker.com/engine/install/
• https://docs.docker.com/compose/install/

The instructions below outline installation of the current stable release (v1.29.2) of Compose.

Data structure

There are 4 volumes defined:

• /etc/rhodecode/conf

  Shared volume used for configuration files for rhodecode, vcsserver and supervisord, and some cache data

• /var/opt/rhodecode_repo_store

  Used for main repository storage where repositories would be stored

• /var/opt/rhodecode_data

  Data dir for rhodecode cache/lock files, or user sessions (for file backend)

• /var/log/rhodecode

  Used to store all logs from RhodeCode

Setup/Build options

There are 3 options to run the docker stack.

• Standalone cluster build from installer
• Standalone cluster build from source
• dev environment setup

Standalone cluster build from installer

Follow these steps to build and run the RhodeCode Cluster via Docker-compose.
Get the repository:

git clone https://code.rhodecode.com/rhodecode-enterprise-docker && cd rhodecode-enterprise-docker

Download installer binaries

First start by fetching required installer binaries. This is required to create both
simple build and full compose setup.
Please check the .env file to adjust RhodeCode version if needed.
(e.g. RC_VERSION="4.24.1")

cd .bootstrap/ && ./download-artifacts.sh && cd ../

This will download required installer files and put them into the .cache directory.
This directory should look similar to that after downloads have finish:

drwxr-xr-x   8 rcdev  rcdev   256B Feb  8 13:35 .
drwxr-xr-x  14 rcdev  rcdev   448B Feb  8 10:40 ..
-rw-r--r--   1 rcdev  rcdev     0B Feb  8 20:44 .dirkeep
-rwxr-xr-x   1 rcdev  rcdev   241M Feb  8 13:35 RhodeCode-installer-linux-build20210208_0800
-rw-r--r--   1 rcdev  rcdev   156M Feb  8 13:35 RhodeCodeCommunity-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
-rw-r--r--   1 rcdev  rcdev   171M Feb  8 13:35 RhodeCodeEnterprise-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
-rw-r--r--   1 rcdev  rcdev   145M Feb  8 13:35 RhodeCodeVCSServer-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
-rw-r--r--   1 rcdev  rcdev   109M Feb  8 13:35 locale-archive

Set License for EE version

In order to install EE edition a license file is required to be present.
It can contain your current license, or when empty license can be applied via Web interface.

To apply it during build phase save your raw license data into a file

config/rhodecode_enterprise.license

If this file is present build phase will read it and license will be applied at creation.
This file can also be empty and license can be applied via a WEB interface after first login.

Run Docker compose build:

This will build RhodeCode based on downloaded installer packages.

To create a full stack we need to run the database container, so it's ready to
build the docker image.

Disk space problems?

docker system df
docker builder prune
# optionally
docker image prune -a

Creating & building images

1) start with running the required database for the build stage in the background.

``` shell
docker-compose up --detach database
```

This will start our postgres database, and expose it to the network.  
Postgres DB is configured to output logs into a file `/var/log/rhodecode/postgres`

2) We can now run the full installation. Database needs to be running for the next build command.
This will build the rhodecode base image used for rhodecode, vcsserver, celery, ssh, svn
Then it will build all other components required.

``` shell
docker-compose build rhodecode
docker-compose build
```

3) Once we build all required containers, we can run the whole stack using docker-compose up

``` shell
docker-compose up
```

If running locally you can access Running RhodeCode via Nginx under:
http://localhost:8888

localhost can be changed to the server IP where docker is running.

In case for bigger setups docker-compose can scale more rhodecode/vcsserver workers:

docker-compose up --scale vcsserver=3 rhodecode=3

Logging is pushed to stdout from all services.

Upgrade procedure:

• pull the latest rhodecode-docker repo
• check .env file for correct update version
• re-build rhodecode
• docker-compose build rhodecode
• docker-compose build
• docker-compose stop
• docker-compose up

Standalone cluster build from source

There's an option to build the latest release from the source code as a docker installation.

If you have the 3 required projects source code already, the next step can be omitted, and the
sources can be copied to the .source/ directory. note: symlinks don't work.

• https://code.rhodecode.com/rhodecode-vcsserver
• https://code.rhodecode.com/rhodecode-enterprise-ce
• https://code.rhodecode.com/rhodecode-enterprise-ee (assuming access is granted to this)

Download the source using bootstrap script:

cd .boostrap/ && ./download-source.sh && cd ../

This step will create source code copies into the .source/ path. e.g

-rw-r--r-- 1 docker docker    0 Nov 25 12:27 .dirkeep
drwxr-xr-x 1 docker docker 1184 Nov 25 12:27 rhodecode-enterprise-ce
drwxr-xr-x 1 docker docker 1120 Nov 25 12:27 rhodecode-enterprise-ee
drwxr-xr-x 1 docker docker  800 Nov 25 12:27 rhodecode-vcsserver

With this done, you can now proceed with every step of normal source installation (Creating & building images), but instead of using
just docker-compose command it needs to be replaced with docker-compose -f docker-compose.yaml -f docker-compose.source.yaml

For example to override the installer build with the source rhodecode based image, and also setting proper version, run:

RC_VERSION="4.27.0.REL202221020" docker-compose -f docker-compose-apps.yaml -f docker-compose.source.yaml build --no-cache --progress=plain rhodecode

NOTE THAT it's recommended to keep rc_version and source_Ver the same

Simple build

Build docker RhodeCode Community without any dependencies (redis, external db) using
simple sqlite database and file based caches.
This is a fully running instance good for small use with 3-5 users.

docker build -t rhodecode/rhodecode-ce:4.23.2 -f rhodecode.dockerfile \
-e RHODECODE_TYPE=Community \
-e RHODECODE_VERSION=4.23.2 \
-e RHODECODE_DB=sqlite \
-e RHODECODE_USER_NAME=admin \
-e RHODECODE_USER_PASS=secret4 \
-e RHODECODE_USER_EMAIL=support@rhodecode.com \
.

note: for debugging better to add --progress plain into the build command to obtain all the output from the build.
To Build against existing running Postgres or MySQL you can specify:

--build-arg RHODECODE_DB=postgresql://postgres:secret@database/rhodecode
--build-arg RHODECODE_DB=mysql://root:secret@localhost/rhodecode?charset=utf8

To copy over the data into volumes use such command:

docker run -v logvolume:/data --name data_vol busybox true
docker cp . data_vol:/data
docker rm data_vol

Run the container, mounting the required volumes. By default the application would be
available at http://localhost:10020, and default login is (unless specified differently in the build command)

user: admin
password: secret4

We've not built our image using specific version. It's time to run it:
We specify the run.ini by selecting config option we have locally

docker run \
  --name rhodecode-container \
  --publish 10020:10020 \
  --restart unless-stopped \
  --volume $PWD/config:/etc/rhodecode/conf \
  --volume $PWD/config/rhodecode.optimized.ini:/etc/rhodecode/conf_build/run.ini \
  --volume $PWD/logs:/var/log/rhodecode \
  'rhodecode/rhodecode-ee:4.23.2'

Enter container

docker exec -it rhodecode-container /bin/bash

Enter interactive shell

docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-ishell /etc/rhodecode/conf/rhodecode.ini

Run Database migrations

docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-upgrade-db /etc/rhodecode/conf/rhodecode.ini --force-yes

Registry for docker swarm

docker run -d -p 5000:5000 --restart always --name registry registry:2