README.md
297 lines
| 9.4 KiB
| text/x-minidsrc
|
MarkdownLexer
r1 | # RhodeCode Cluster | ||
r23 | RhodeCode Cluster is a multi-node highly-scalable setup to run | ||
RhodeCode, Edge Router(Traefik + SSL) and Metrics Stack (Prometheus, Loki, Grafana) and | |||
r1 | all its additional components in single environment using Docker. | ||
Using a docker-compose this setup creates following services for RhodeCode: | |||
r23 | Edge-Router: | ||
- Traefik, Edge Router, SSL termination etc | |||
Core Services: | |||
- Database (defaults to PostgreSQL) | |||
- Redis, acts as cache and queue exchange | |||
- ChannelStream - live websocket communications | |||
- Nginx (static/channelstream) proxy serving RhodeCode APPS and channelstream communication | |||
- Elasticsearch (full text search backend) | |||
RhodeCode | |||
- RhodeCode CE/EE | |||
- VCSServer for GIT/SVN/HG support | |||
- SSH Server for cloning over SSH | |||
- SVN webserver for HTTP support over SVN | |||
- Celery workers for asynchronous tasks | |||
- Celery beat for automation tasks | |||
Metrics | |||
- Loki, logs aggregation | |||
- Grafana, Metrics Dashboard | |||
- Promethues, Metrics time-series | |||
- statsd-exporter - statsd to promethues bridge | |||
- node-exporter - machine stats | |||
- promtail - log scraping | |||
r1 | |||
## Pre requisites | |||
r23 | To Run this stack Docker engine and Docker Compose needs to be installed on the host machine. | ||
r1 | Visit docker site and install docker (min version 20.10) and docker compose: | ||
r23 | - https://docs.docker.com/engine/install/ | ||
- https://docs.docker.com/compose/install/ | |||
r1 | |||
r9 | The instructions below outline installation of the current stable release (v1.29.2) of Compose. | ||
r1 | |||
r4 | ## Data structure | ||
There are 4 volumes defined: | |||
- `/etc/rhodecode/conf` | |||
r23 | Shared volume used for configuration files for rhodecode, vcsserver and supervisord, and some cache data | ||
r4 | |||
- `/var/opt/rhodecode_repo_store` | |||
r23 | Used for main repository storage where repositories would be stored | ||
r4 | |||
- `/var/opt/rhodecode_data` | |||
r23 | Data dir for rhodecode cache/lock files, or user sessions (for file backend) | ||
r4 | |||
- `/var/log/rhodecode` | |||
r23 | Used to store all logs from RhodeCode | ||
r4 | |||
## Setup/Build options | |||
There are 3 options to run the docker stack. | |||
- [Standalone cluster build from installer](#standalone-cluster-build-from-installer) | |||
r8 | - [Standalone cluster build from source](#standalone-cluster-build-from-source) | ||
r23 | - [dev environment setup](#dev-env-build) | ||
r4 | |||
# Standalone cluster build from installer | |||
r2 | |||
r1 | Follow these steps to build and run the RhodeCode Cluster via Docker-compose. | ||
r2 | Get the repository: | ||
git clone https://code.rhodecode.com/rhodecode-enterprise-docker && cd rhodecode-enterprise-docker | |||
r0 | |||
## Download installer binaries | |||
First start by fetching required installer binaries. This is required to create both | |||
simple build and full compose setup. | |||
r12 | Please check the `.env` file to adjust RhodeCode version if needed. | ||
(e.g. RC_VERSION="4.24.1") | |||
r0 | |||
r4 | ```shell | ||
r8 | cd .bootstrap/ && ./download-artifacts.sh && cd ../ | ||
r4 | ``` | ||
r0 | |||
r12 | This will download required installer files and put them into the `.cache` directory. | ||
r1 | This directory should look similar to that after downloads have finish: | ||
```shell | |||
drwxr-xr-x 8 rcdev rcdev 256B Feb 8 13:35 . | |||
drwxr-xr-x 14 rcdev rcdev 448B Feb 8 10:40 .. | |||
-rw-r--r-- 1 rcdev rcdev 0B Feb 8 20:44 .dirkeep | |||
-rwxr-xr-x 1 rcdev rcdev 241M Feb 8 13:35 RhodeCode-installer-linux-build20210208_0800 | |||
-rw-r--r-- 1 rcdev rcdev 156M Feb 8 13:35 RhodeCodeCommunity-4.24.1+x86_64-linux_build20210208_0800.tar.bz2 | |||
-rw-r--r-- 1 rcdev rcdev 171M Feb 8 13:35 RhodeCodeEnterprise-4.24.1+x86_64-linux_build20210208_0800.tar.bz2 | |||
-rw-r--r-- 1 rcdev rcdev 145M Feb 8 13:35 RhodeCodeVCSServer-4.24.1+x86_64-linux_build20210208_0800.tar.bz2 | |||
-rw-r--r-- 1 rcdev rcdev 109M Feb 8 13:35 locale-archive | |||
``` | |||
r0 | |||
r1 | ## Set License for EE version | ||
r0 | |||
r23 | In order to install EE edition a license file is required to be present. | ||
It can contain your current license, or when empty license can be applied via Web interface. | |||
r12 | To apply it during build phase save your raw license data into a file | ||
r9 | |||
r23 | `config/rhodecode_enterprise.license` | ||
r9 | |||
If this file is present build phase will read it and license will be applied at creation. | |||
This file can also be empty and license can be applied via a WEB interface after first login. | |||
r1 | |||
## Run Docker compose build: | |||
r0 | |||
r8 | *This will build RhodeCode based on downloaded installer packages.* | ||
r0 | To create a full stack we need to run the database container, so it's ready to | ||
build the docker image. | |||
r12 | _Disk space problems?_ | ||
```shell | |||
docker system df | |||
docker builder prune | |||
r23 | # optionally | ||
docker image prune -a | |||
r12 | ``` | ||
r23 | ### Creating & building images | ||
r12 | |||
r0 | 1) start with running the required database for the build stage in the background. | ||
```shell | |||
docker-compose up --detach database | |||
``` | |||
r4 | This will start our postgres database, and expose it to the network. | ||
r23 | Postgres DB is configured to output logs into a file `/var/log/rhodecode/postgres` | ||
r4 | |||
2) We can now run the full installation. Database needs to be running for the next build command. | |||
r23 | This will build the rhodecode base image used for rhodecode, vcsserver, celery, ssh, svn | ||
Then it will build all other components required. | |||
r0 | ```shell | ||
r1 | docker-compose build rhodecode | ||
docker-compose build | |||
r0 | ``` | ||
r23 | 4) Once we build all required containers, we can run the whole stack using `docker-compose up` | ||
r4 | |||
```shell | |||
docker-compose up | |||
``` | |||
r0 | |||
r8 | If running locally you can access Running RhodeCode via Nginx under: | ||
r0 | http://localhost:8888 | ||
r8 | localhost can be changed to the server IP where docker is running. | ||
r0 | |||
In case for bigger setups docker-compose can scale more rhodecode/vcsserver workers: | |||
```shell | |||
docker-compose up --scale vcsserver=3 rhodecode=3 | |||
``` | |||
r23 | Logging is pushed to stdout from all services. | ||
r1 | |||
r23 | ### Upgrade procedure: | ||
r1 | |||
r8 | - pull the latest rhodecode-docker repo | ||
r1 | - check .env file for correct update version | ||
- re-build rhodecode | |||
- docker-compose build rhodecode | |||
r23 | - docker-compose build | ||
r1 | - docker-compose stop | ||
- docker-compose up | |||
r4 | # Standalone cluster build from source | ||
r1 | |||
r23 | There's an option to build the latest release from the source code as a docker installation. | ||
If you have the 3 required projects source code already, the next step can be omitted, and the | |||
sources can be copied to the `.source/` directory. note: symlinks don't work. | |||
r8 | |||
r23 | - https://code.rhodecode.com/rhodecode-vcsserver | ||
- https://code.rhodecode.com/rhodecode-enterprise-ce | |||
- https://code.rhodecode.com/rhodecode-enterprise-ee (assuming access is granted to this) | |||
Download the source using bootstrap script: | |||
r1 | |||
r4 | ```shell | ||
r8 | cd .boostrap/ && ./download-source.sh && cd ../ | ||
r4 | ``` | ||
r1 | |||
r8 | This step will create source code copies into the `.source/` path. e.g | ||
``` | |||
-rw-r--r-- 1 docker docker 0 Nov 25 12:27 .dirkeep | |||
drwxr-xr-x 1 docker docker 1184 Nov 25 12:27 rhodecode-enterprise-ce | |||
drwxr-xr-x 1 docker docker 1120 Nov 25 12:27 rhodecode-enterprise-ee | |||
drwxr-xr-x 1 docker docker 800 Nov 25 12:27 rhodecode-vcsserver | |||
``` | |||
r23 | With this done, you can now proceed with every step of normal source installation (Creating & building images), but instead of using | ||
just `docker-compose` command it needs to be replaced with `docker-compose -f docker-compose.yaml -f docker-compose.source.yaml` | |||
r8 | |||
r23 | For example to override the installer build with the source `rhodecode` based image, and also setting proper version, run: | ||
r1 | |||
r4 | ```shell | ||
r23 | RC_VERSION="4.27.0.REL202221020" docker-compose -f docker-compose-apps.yaml -f docker-compose.source.yaml build --no-cache --progress=plain rhodecode | ||
r4 | ``` | ||
r1 | |||
r23 | NOTE THAT it's recommended to keep rc_version and source_Ver the same | ||
r0 | |||
## Simple build | |||
Build docker RhodeCode `Community` without any dependencies (redis, external db) using | |||
simple sqlite database and file based caches. | |||
This is a fully running instance good for small use with 3-5 users. | |||
```shell | |||
r23 | docker build -t rhodecode/rhodecode-ce:4.23.2 -f rhodecode.dockerfile \ | ||
r1 | -e RHODECODE_TYPE=Community \ | ||
-e RHODECODE_VERSION=4.23.2 \ | |||
-e RHODECODE_DB=sqlite \ | |||
-e RHODECODE_USER_NAME=admin \ | |||
-e RHODECODE_USER_PASS=secret4 \ | |||
-e RHODECODE_USER_EMAIL=support@rhodecode.com \ | |||
r0 | . | ||
``` | |||
note: for debugging better to add `--progress plain` into the build command to obtain all the output from the build. | |||
To Build against existing running Postgres or MySQL you can specify: | |||
--build-arg RHODECODE_DB=postgresql://postgres:secret@database/rhodecode | |||
--build-arg RHODECODE_DB=mysql://root:secret@localhost/rhodecode?charset=utf8 | |||
To copy over the data into volumes use such command: | |||
```shell | |||
docker run -v logvolume:/data --name data_vol busybox true | |||
docker cp . data_vol:/data | |||
docker rm data_vol | |||
``` | |||
Run the container, mounting the required volumes. By default the application would be | |||
available at http://localhost:10020, and default login is (unless specified differently in the build command) | |||
``` | |||
user: admin | |||
password: secret4 | |||
``` | |||
We've not built our image using specific version. It's time to run it: | |||
r23 | We specify the run.ini by selecting config option we have locally | ||
r0 | |||
```shell | |||
docker run \ | |||
--name rhodecode-container \ | |||
--publish 10020:10020 \ | |||
--restart unless-stopped \ | |||
--volume $PWD/config:/etc/rhodecode/conf \ | |||
r23 | --volume $PWD/config/rhodecode.optimized.ini:/etc/rhodecode/conf_build/run.ini \ | ||
r0 | --volume $PWD/logs:/var/log/rhodecode \ | ||
r23 | 'rhodecode/rhodecode-ee:4.23.2' | ||
r0 | ``` | ||
Enter container | |||
```shell | |||
docker exec -it rhodecode-container /bin/bash | |||
``` | |||
Enter interactive shell | |||
```shell | |||
r23 | docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-ishell /etc/rhodecode/conf/rhodecode.ini | ||
r0 | ``` | ||
Run Database migrations | |||
```shell | |||
r23 | docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-upgrade-db /etc/rhodecode/conf/rhodecode.ini --force-yes | ||
``` | |||
### Registry for docker swarm | |||
docker run -d -p 5000:5000 --restart always --name registry registry:2 |