rcstack: use dl server

README.md

@@ -1,393 +1,393 b''
 # RhodeCode Cluster
 
 RhodeCode Cluster is a multi-node highly-scalable setup to run 
 RhodeCode, Edge Router(Traefik + SSL) and Metrics Stack (Prometheus, Loki, Grafana) and
 all its additional components in single environment using Docker.
 
 Using a docker-compose this setup creates following services for RhodeCode:
 
 Edge-Router:
 
  - Traefik, Edge Router, SSL termination etc
 
 Core Services:
 
  - Database (defaults to PostgreSQL)
  - Redis, acts as cache and queue exchange
  - ChannelStream - live websocket communications
  - Nginx (static/channelstream) proxy serving RhodeCode static files and channelstream communication
  - Elasticsearch (full text search backend)
 
 RhodeCode
 
  - RhodeCode CE/EE
  - VCSServer for GIT/SVN/HG support
  - SSH Server for cloning over SSH
  - SVN webserver for HTTP support over SVN
  - Celery workers for asynchronous tasks
  - Celery beat for automation tasks
 
 Metrics
 
  - Loki, logs aggregation
  - Grafana, metrics Dashboard
  - Prometheus, metrics time-series
  - Statsd-exporter - statsd to Prometheus bridge
  - Node-exporter - machine stats and usage
  - Promtail - log scraping
 
 ##rcstack for Linux, new docker based installer
 
  To get started with RhodeCode get the new shell installer called rcstack
 
 ```
 mkdir docker-rhodecode && cd docker-rhodecode
-curl -s -o rcstack https://code.rhodecode.com/rhodecode-enterprise-docker/raw/master/scripts/rcstack/rcstack && chmod +x rcstack
+curl -L -s -o rcstack https://dls.rhodecode.com/get/master && chmod +x rcstack
 ./rcstack get-started
 ```
 
 ## Pre requisites
 
 To Run this stack Docker engine and Docker Compose needs to be installed on the host machine.
 Please run `./rcstack init` to install docker using the installer, or 
 visit docker site and install docker (min version 20.10) and docker compose:
 
 - https://docs.docker.com/engine/install/
 - https://docs.docker.com/compose/install/ 
 
 ./rcstack init can install docker on linux machine, but if manual installation is 
 required it's also possible by simply installing docker before
 
 
 
 # Quick install tutorial
 
 Those are step-by-step installation/run steps.
 
  create configurations / docker definitions:
     
     ./rcstack init
 
  At this point a custom file under .custom/.runtime.env was created. Adjust it if required.
 
  Start Traefik router that would handle all incoming traffic, load balance. A valid domain needs to be present in 
  .custom/.runtime.env to access the RhodeCOde
     
     ./rcstack stack router up --detach
  
  Start the database and bootstrap it
 
     ./rcontrol stack database up --detach
  
  Start other services required
 
     ./rcstack stack services up --detach
 
  Start RhodeCode stack
     
     ./rcstack stack rhodecode up --detach
 
  Check stack status
 
     ./rcstack status
 
  Output should look similar like this:
 
     ---
     CONTAINER ID   IMAGE                                   STATUS                        NAMES                                 PORTS
     ef54fc528e3a   traefik:v2.9.5                          Up 2 hours             rc_cluster_router-traefik-1           0.0.0.0:80->80/tcp, :::80->80/tcp
     f3ea0539e8b0   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-rhodecode-1           0.0.0.0:10020->10020/tcp, :::10020->10020/tcp
     2be52ba58ffe   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-vcsserver-1
     7cd730ad3263   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-celery-1
     dfa231342c87   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-celery-beat-1
     d3d76ce2de96   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-sshd-1
     daaac329414b   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-svn-1
     7b8504fb9acb   nginx:1.23.2                            Up 2 hours (healthy)   rc_cluster_services-nginx-1           80/tcp
     7279c25feb6b   elasticsearch:6.8.23                    Up 2 hours (healthy)   rc_cluster_services-elasticsearch-1   9200/tcp, 9300/tcp
     19fb93587493   redis:7.0.5                             Up 2 hours (healthy)   rc_cluster_services-redis-1           6379/tcp
     fb77fb6496c6   channelstream/channelstream:0.7.1       Up 2 hours (healthy)   rc_cluster_services-channelstream-1   8000/tcp
     cb6c5c022f5b   postgres:14.6                           Up 2 hours (healthy)   rc_cluster_services-database-1        5432/tcp
 
 # Standalone cluster build from installer
 
 If you;d like to build your own custom image here's a quick how to.:
 Follow these steps to build and run the RhodeCode Cluster via Docker-compose.
 
 1) Run:
 
     ./rcstack init
 
 2) Run artifacts fetch like installer and certain needed build binaries:
 
    Start by fetching required installer binaries. This is required to create both
    simple build and full compose setup.
    Please use the `--version-name VERSION_NAME` flag to adjust RhodeCode version if needed.
    (e.g. --version-name "4.24.1")
    
    ```
    ./rcstack get-build-artifacts
    ```
    
     This will download required installer files and put them into the `.cache` directory. 
     This directory should look similar to that after downloads have finish:
     
     ```
     drwxr-xr-x   8 rcdev  rcdev   256B Feb  8 13:35 .
     drwxr-xr-x  14 rcdev  rcdev   448B Feb  8 10:40 ..
     -rw-r--r--   1 rcdev  rcdev     0B Feb  8 20:44 .dirkeep
     -rwxr-xr-x   1 rcdev  rcdev   241M Feb  8 13:35 RhodeCode-installer-linux-build20210208_0800
     -rw-r--r--   1 rcdev  rcdev   156M Feb  8 13:35 RhodeCodeCommunity-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
     -rw-r--r--   1 rcdev  rcdev   171M Feb  8 13:35 RhodeCodeEnterprise-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
     -rw-r--r--   1 rcdev  rcdev   145M Feb  8 13:35 RhodeCodeVCSServer-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
     -rw-r--r--   1 rcdev  rcdev   109M Feb  8 13:35 locale-archive
     ```
 
 
 3) Create the build
 
     ./rcstack build-installer
 
 # Standalone cluster build from source code
 
 There's an option to build the latest release from the source code as a docker installation.
 If you;d like to build your own custom image based on the source code here's a quick how to.:
 Follow these steps to build and run the RhodeCode Cluster via Docker-compose.
 
 1) Run init to setup needed docker env and files:
 
     ./rcstack init
 
 2) Run artifacts fetch like installer and certain needed build binaries:
 
    Start by fetching required installer binaries.
    Only local-archive is required to be present, installer is disregarded for source build
    
    ```
    ./rcstack get-build-artifacts
    ```
    
     This will download required installer files and put them into the `.cache` directory. 
     This directory should look similar to that after downloads have finish:
     
     ```
     drwxr-xr-x   8 rcdev  rcdev   256B Feb  8 13:35 .
     drwxr-xr-x  14 rcdev  rcdev   448B Feb  8 10:40 ..
     -rw-r--r--   1 rcdev  rcdev     0B Feb  8 20:44 .dirkeep
     -rwxr-xr-x   1 rcdev  rcdev   241M Feb  8 13:35 RhodeCode-installer-linux-build20210208_0800
     -rw-r--r--   1 rcdev  rcdev   156M Feb  8 13:35 RhodeCodeCommunity-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
     -rw-r--r--   1 rcdev  rcdev   171M Feb  8 13:35 RhodeCodeEnterprise-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
     -rw-r--r--   1 rcdev  rcdev   145M Feb  8 13:35 RhodeCodeVCSServer-4.24.1+x86_64-linux_build20210208_0800.tar.bz2
     -rw-r--r--   1 rcdev  rcdev   109M Feb  8 13:35 locale-archive
     ```
 
 3) get source code needed to create a build
 
     This step will create source code copies into the `.source/` path. So it will look like this:
     
    ```
     -rw-r--r-- 1 docker docker    0 Nov 25 12:27 .dirkeep
     drwxr-xr-x 1 docker docker 1184 Nov 25 12:27 rhodecode-enterprise-ce
     drwxr-xr-x 1 docker docker 1120 Nov 25 12:27 rhodecode-enterprise-ee
     drwxr-xr-x 1 docker docker  800 Nov 25 12:27 rhodecode-vcsserver
     ```
 
     If you have the 3 required projects source code already, this step can be omitted, and the 
     sources can be copied to the `.source/` directory. note: symlinks don't work.
 
     - https://code.rhodecode.com/rhodecode-vcsserver
     - https://code.rhodecode.com/rhodecode-enterprise-ce
     - https://code.rhodecode.com/rhodecode-enterprise-ee (assuming access is granted to this)
 
     Run this to fetch the sources
 
     ```
     ./rcstack get-build-source --revision=default --auth-token=xxxx --server-url=https://code.rhodecode.com/sources
     ```
 
 
 4) Create the build
    
     ```
     ./rcstack build-source --version-name 4.28.0.REL.2022.12.12
     ```
 
 # Operation
 
 ## Data structure
 
 There are 4 volumes defined:
 
 
 - `/etc/rhodecode/conf`
   
   Shared volume used for configuration files for rhodecode, vcsserver and supervisord, and some cache data
 
 - `/var/opt/rhodecode_repo_store` 
 
   Used for main repository storage where repositories would be stored
   
 - `/var/opt/rhodecode_data` 
   
   Data dir for rhodecode cache/lock files, or user sessions (for file backend)
 
 
 ## Set License for EE version
 
 In order to install EE edition a license file is required to be present.
 It can contain your current license, or when empty license can be applied via Web interface.
 
 To apply it during build phase save your raw license data into a file 
 
 `config/rhodecode_enterprise.license` 
 
 If this file is present build phase will read it and license will be applied at creation.
 This file can also be empty and license can be applied via a WEB interface after first login.
 
 ## Run Docker compose build:
 
 *This will build RhodeCode based on downloaded installer packages.*
 
 To create a full stack we need to run the database container, so it's ready to
 build the docker image.
 
 _Disk space problems?_
      
 ```
 docker system df
 docker builder prune -f
 # optionally
 docker image prune -a
 ```
 
 ### Creating & building images
 
 1) start with running the required database for the build stage in the background.
    
     ```
     docker-compose up --detach database
     ```
    
     This will start our postgres database, and expose it to the network.
     Postgres DB is configured to output logs into a stdout
 
 2) We can now run the full installation. Database needs to be running for the next build command.
    This will build the rhodecode base image used for rhodecode, vcsserver, celery, ssh, svn
    Then it will build all other components required.
 
     ```
     docker-compose build rhodecode
     docker-compose build
     ```
 
 4) Once we build all required containers, we can run the whole stack using `docker-compose up`
 
     ```
     docker-compose up
     ```
 
 If running locally you can access Running RhodeCode via Nginx under:
 http://localhost:8888
 
 localhost can be changed to the server IP where docker is running.
 
 
 In case for bigger setups docker-compose can scale more rhodecode/vcsserver workers:
 
 ```
 docker-compose up --scale vcsserver=3 rhodecode=3
 ```
 
 Logging is pushed to stdout from all services.
 
 
 ### Upgrade procedure:
 
 - run ./rcstack self-update
 - run ./rcstack stack-upgrade to get upgrade instructions
 
 
 With this done, you can now proceed with every step of normal source installation (Creating & building images), but instead of using
 just `docker-compose` command it needs to be replaced with `docker-compose -f docker-compose.yaml -f docker-compose.source.yaml`
 
 For example to override the installer build with the source `rhodecode` based image, and also setting proper version, run:
 
 ```
 RC_VERSION="4.28.0.SRC.2022.12.12.1" docker-compose -f docker-compose-apps.yaml -f docker-compose.source.yaml build --no-cache --progress=plain rhodecode
 ```
 
 NOTE THAT it's recommended to keep rc_version and source_Ver the same
 
 ## Simple community build
 
 Build docker RhodeCode `Community` without any dependencies (redis, external db) using 
 simple sqlite database and file based caches. 
 This is a fully running instance good for small use with 3-5 users.
 
 ```
 docker build -t rhodecode/rhodecode-ce:4.23.2 -f rhodecode.dockerfile \
 -e RHODECODE_TYPE=Community \
 -e RHODECODE_VERSION=4.23.2 \
 -e RHODECODE_DB=sqlite \
 -e RHODECODE_USER_NAME=admin \
 -e RHODECODE_USER_PASS=secret4 \
 -e RHODECODE_USER_EMAIL=[e-mail] \
 .
 ```
 
 note: for debugging better to add `--progress plain` into the build command to obtain all the output from the build.
 To Build against existing running Postgres or MySQL you can specify:
 
     --build-arg RHODECODE_DB=postgresql://postgres:secret@database/rhodecode
     --build-arg RHODECODE_DB=mysql://root:secret@localhost/rhodecode?charset=utf8
 
 
 To copy over the data into volumes use such command:
 ```
 docker run -v logvolume:/data --name data_vol busybox true
 docker cp . data_vol:/data
 docker rm data_vol
 ```
 
 Run the container, mounting the required volumes. By default the application would be
 available at http://localhost:10020, and default login is (unless specified differently in the build command)
 
 ```
 user: admin
 password: secret4
 ```
 
 We've not built our image using specific version. It's time to run it:
 We specify the run.ini by selecting config option we have locally
 
 ```
 docker run \
   --name rhodecode-container \
   --publish 10020:10020 \
   --restart unless-stopped \
   --volume $PWD/config:/etc/rhodecode/conf \
   --volume $PWD/config/rhodecode.ini:/etc/rhodecode/conf_build/rhodecode.ini \
   'rhodecode/rhodecode-ee:4.23.2'
 ```
 
 Enter container
 
 ```
 docker exec -it rhodecode-container /bin/bash
 ```
 
 Enter interactive shell
 
 ```
 docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-ishell /etc/rhodecode/conf/rhodecode.ini
 ```
 
 Run Database migrations
 ```
 docker exec -it rhodecode-container /usr/local/bin/rhodecode_bin/bin/rc-upgrade-db /etc/rhodecode/conf/rhodecode.ini --force-yes
 ```
 
 
 
 ### Registry for docker swarm
 
 docker run -d -p 5000:5000 --restart always --name registry registry:2
  No newline at end of file