rcstack: use dl server

super-admin - - Load All Authors

File last commit:

r148:91b23071


                r148:91b23071

Download file

             README.md
        
                    392 lines
            
             | 13.4 KiB
            
                | text/x-minidsrc
            
             |
                MarkdownLexer

/ README.md

History | Annotation | Raw |Copy content |Copy permalink

RhodeCode Cluster

RhodeCode Cluster is a multi-node highly-scalable setup to run
RhodeCode, Edge Router(Traefik + SSL) and Metrics Stack (Prometheus, Loki, Grafana) and
all its additional components in single environment using Docker.

Using a docker-compose this setup creates following services for RhodeCode:

Edge-Router:

Traefik, Edge Router, SSL termination etc

Core Services:

Database (defaults to PostgreSQL)
Redis, acts as cache and queue exchange
ChannelStream - live websocket communications
Nginx (static/channelstream) proxy serving RhodeCode static files and channelstream communication
Elasticsearch (full text search backend)

RhodeCode

RhodeCode CE/EE
VCSServer for GIT/SVN/HG support
SSH Server for cloning over SSH
SVN webserver for HTTP support over SVN
Celery workers for asynchronous tasks
Celery beat for automation tasks

Metrics

Loki, logs aggregation
Grafana, metrics Dashboard
Prometheus, metrics time-series
Statsd-exporter - statsd to Prometheus bridge
Node-exporter - machine stats and usage
Promtail - log scraping

##rcstack for Linux, new docker based installer

To get started with RhodeCode get the new shell installer called rcstack

mkdir docker-rhodecode && cd docker-rhodecode
curl -L -s -o rcstack https://dls.rhodecode.com/get/master && chmod +x rcstack
./rcstack get-started

Pre requisites

To Run this stack Docker engine and Docker Compose needs to be installed on the host machine.
Please run ./rcstack init to install docker using the installer, or
visit docker site and install docker (min version 20.10) and docker compose:

./rcstack init can install docker on linux machine, but if manual installation is
required it's also possible by simply installing docker before

Quick install tutorial

Those are step-by-step installation/run steps.

create configurations / docker definitions:

./rcstack init

At this point a custom file under .custom/.runtime.env was created. Adjust it if required.

Start Traefik router that would handle all incoming traffic, load balance. A valid domain needs to be present in
.custom/.runtime.env to access the RhodeCOde

./rcstack stack router up --detach

Start the database and bootstrap it

./rcontrol stack database up --detach

Start other services required

./rcstack stack services up --detach

Start RhodeCode stack

./rcstack stack rhodecode up --detach

Check stack status

./rcstack status

Output should look similar like this:

---
CONTAINER ID   IMAGE                                   STATUS                        NAMES                                 PORTS
ef54fc528e3a   traefik:v2.9.5                          Up 2 hours             rc_cluster_router-traefik-1           0.0.0.0:80->80/tcp, :::80->80/tcp
f3ea0539e8b0   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-rhodecode-1           0.0.0.0:10020->10020/tcp, :::10020->10020/tcp
2be52ba58ffe   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-vcsserver-1
7cd730ad3263   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-celery-1
dfa231342c87   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-celery-beat-1
d3d76ce2de96   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-sshd-1
daaac329414b   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-svn-1
7b8504fb9acb   nginx:1.23.2                            Up 2 hours (healthy)   rc_cluster_services-nginx-1           80/tcp
7279c25feb6b   elasticsearch:6.8.23                    Up 2 hours (healthy)   rc_cluster_services-elasticsearch-1   9200/tcp, 9300/tcp
19fb93587493   redis:7.0.5                             Up 2 hours (healthy)   rc_cluster_services-redis-1           6379/tcp
fb77fb6496c6   channelstream/channelstream:0.7.1       Up 2 hours (healthy)   rc_cluster_services-channelstream-1   8000/tcp
cb6c5c022f5b   postgres:14.6                           Up 2 hours (healthy)   rc_cluster_services-database-1        5432/tcp

Standalone cluster build from installer

If you;d like to build your own custom image here's a quick how to.:
Follow these steps to build and run the RhodeCode Cluster via Docker-compose.

1) Run:

./rcstack init

2) Run artifacts fetch like installer and certain needed build binaries:

Start by fetching required installer binaries. This is required to create both  
simple build and full compose setup.  
Please use the `--version-name VERSION_NAME` flag to adjust RhodeCode version if needed.  
(e.g. --version-name "4.24.1")

    ./rcstack get-build-artifacts

This will download required installer files and put them into the `.cache` directory.  
This directory should look similar to that after downloads have finish:

    drwxr-xr-x   8 rcdev  rcdev   256B Feb  8 13:35 .
    drwxr-xr-x  14 rcdev  rcdev   448B Feb  8 10:40 ..
    -rw-r--r--   1 rcdev  rcdev     0B Feb  8 20:44 .dirkeep
    -rwxr-xr-x   1 rcdev  rcdev   241M Feb  8 13:35 RhodeCode-installer-linux-build20210208_0800
    -rw-r--r--   1 rcdev  rcdev   156M Feb  8 13:35 RhodeCodeCommunity-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
    -rw-r--r--   1 rcdev  rcdev   171M Feb  8 13:35 RhodeCodeEnterprise-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
    -rw-r--r--   1 rcdev  rcdev   145M Feb  8 13:35 RhodeCodeVCSServer-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
    -rw-r--r--   1 rcdev  rcdev   109M Feb  8 13:35 locale-archive

3) Create the build

./rcstack build-installer

Standalone cluster build from source code

There's an option to build the latest release from the source code as a docker installation.
If you;d like to build your own custom image based on the source code here's a quick how to.:
Follow these steps to build and run the RhodeCode Cluster via Docker-compose.

1) Run init to setup needed docker env and files:

./rcstack init

2) Run artifacts fetch like installer and certain needed build binaries:

Start by fetching required installer binaries.  
Only local-archive is required to be present, installer is disregarded for source build

    ./rcstack get-build-artifacts

This will download required installer files and put them into the `.cache` directory.  
This directory should look similar to that after downloads have finish:

    drwxr-xr-x   8 rcdev  rcdev   256B Feb  8 13:35 .
    drwxr-xr-x  14 rcdev  rcdev   448B Feb  8 10:40 ..
    -rw-r--r--   1 rcdev  rcdev     0B Feb  8 20:44 .dirkeep
    -rwxr-xr-x   1 rcdev  rcdev   241M Feb  8 13:35 RhodeCode-installer-linux-build20210208_0800
    -rw-r--r--   1 rcdev  rcdev   156M Feb  8 13:35 RhodeCodeCommunity-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
    -rw-r--r--   1 rcdev  rcdev   171M Feb  8 13:35 RhodeCodeEnterprise-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
    -rw-r--r--   1 rcdev  rcdev   145M Feb  8 13:35 RhodeCodeVCSServer-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
    -rw-r--r--   1 rcdev  rcdev   109M Feb  8 13:35 locale-archive

3) get source code needed to create a build

This step will create source code copies into the `.source/` path. So it will look like this:

``` 
 -rw-r--r-- 1 docker docker    0 Nov 25 12:27 .dirkeep
 drwxr-xr-x 1 docker docker 1184 Nov 25 12:27 rhodecode-enterprise-ce
 drwxr-xr-x 1 docker docker 1120 Nov 25 12:27 rhodecode-enterprise-ee
 drwxr-xr-x 1 docker docker  800 Nov 25 12:27 rhodecode-vcsserver
```

If you have the 3 required projects source code already, this step can be omitted, and the  
sources can be copied to the `.source/` directory. note: symlinks don't work.

  - <https://code.rhodecode.com/rhodecode-vcsserver>
  - <https://code.rhodecode.com/rhodecode-enterprise-ce>
  - <https://code.rhodecode.com/rhodecode-enterprise-ee> (assuming access is granted to this)

Run this to fetch the sources

    ./rcstack get-build-source --revision=default --auth-token=xxxx --server-url=https://code.rhodecode.com/sources

4) Create the build

    ./rcstack build-source --version-name 4.28.0.REL.2022.12.12

Operation

Data structure

There are 4 volumes defined:

/etc/rhodecode/conf

Shared volume used for configuration files for rhodecode, vcsserver and supervisord, and some cache data
/var/opt/rhodecode_repo_store

Used for main repository storage where repositories would be stored
/var/opt/rhodecode_data

Data dir for rhodecode cache/lock files, or user sessions (for file backend)

Set License for EE version

In order to install EE edition a license file is required to be present.
It can contain your current license, or when empty license can be applied via Web interface.

To apply it during build phase save your raw license data into a file

config/rhodecode_enterprise.license

If this file is present build phase will read it and license will be applied at creation.
This file can also be empty and license can be applied via a WEB interface after first login.

Run Docker compose build:

This will build RhodeCode based on downloaded installer packages.

To create a full stack we need to run the database container, so it's ready to
build the docker image.

Disk space problems?

docker system df
docker builder prune -f
# optionally
docker image prune -a

Creating & building images

1) start with running the required database for the build stage in the background.

    docker-compose up --detach database

This will start our postgres database, and expose it to the network.  
Postgres DB is configured to output logs into a stdout

2) We can now run the full installation. Database needs to be running for the next build command.
This will build the rhodecode base image used for rhodecode, vcsserver, celery, ssh, svn
Then it will build all other components required.

    docker-compose build rhodecode
    docker-compose build

3) Once we build all required containers, we can run the whole stack using docker-compose up

    docker-compose up

If running locally you can access Running RhodeCode via Nginx under:
http://localhost:8888

localhost can be changed to the server IP where docker is running.

In case for bigger setups docker-compose can scale more rhodecode/vcsserver workers:

docker-compose up --scale vcsserver=3 rhodecode=3

Logging is pushed to stdout from all services.

Upgrade procedure:

run ./rcstack self-update
run ./rcstack stack-upgrade to get upgrade instructions

With this done, you can now proceed with every step of normal source installation (Creating & building images), but instead of using
just docker-compose command it needs to be replaced with docker-compose -f docker-compose.yaml -f docker-compose.source.yaml

For example to override the installer build with the source rhodecode based image, and also setting proper version, run:

RC_VERSION="4.28.0.SRC.2022.12.12.1" docker-compose -f docker-compose-apps.yaml -f docker-compose.source.yaml build --no-cache --progress=plain rhodecode

NOTE THAT it's recommended to keep rc_version and source_Ver the same

Simple community build

Build docker RhodeCode Community without any dependencies (redis, external db) using
simple sqlite database and file based caches.
This is a fully running instance good for small use with 3-5 users.

docker build -t rhodecode/rhodecode-ce:4.23.2 -f rhodecode.dockerfile \
-e RHODECODE_TYPE=Community \
-e RHODECODE_VERSION=4.23.2 \
-e RHODECODE_DB=sqlite \
-e RHODECODE_USER_NAME=admin \
-e RHODECODE_USER_PASS=secret4 \
-e RHODECODE_USER_EMAIL=support@rhodecode.com \
.

note: for debugging better to add --progress plain into the build command to obtain all the output from the build.
To Build against existing running Postgres or MySQL you can specify:

--build-arg RHODECODE_DB=postgresql://postgres:secret@database/rhodecode
--build-arg RHODECODE_DB=mysql://root:secret@localhost/rhodecode?charset=utf8

To copy over the data into volumes use such command:

docker run -v logvolume:/data --name data_vol busybox true
docker cp . data_vol:/data
docker rm data_vol

Run the container, mounting the required volumes. By default the application would be
available at http://localhost:10020, and default login is (unless specified differently in the build command)

user: admin
password: secret4

We've not built our image using specific version. It's time to run it:
We specify the run.ini by selecting config option we have locally

docker run \
  --name rhodecode-container \
  --publish 10020:10020 \
  --restart unless-stopped \
  --volume $PWD/config:/etc/rhodecode/conf \
  --volume $PWD/config/rhodecode.ini:/etc/rhodecode/conf_build/rhodecode.ini \
  'rhodecode/rhodecode-ee:4.23.2'

Enter container

docker exec -it rhodecode-container /bin/bash

Enter interactive shell

docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-ishell /etc/rhodecode/conf/rhodecode.ini

Run Database migrations

docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-upgrade-db /etc/rhodecode/conf/rhodecode.ini --force-yes

Registry for docker swarm

docker run -d -p 5000:5000 --restart always --name registry registry:2

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages