chore: fixed merge conflicts

docs/auth/auth-saml-azure.rst

@@ -0,0 +1,161 b''
+.. _config-saml-azure-ref:
+
+
+SAML 2.0 with Azure Entra ID
+----------------------------
+
+**This plugin is available only in EE Edition.**
+
+|RCE| supports SAML 2.0 Authentication with Azure Entra ID provider. This allows
+users to log-in to RhodeCode via SSO mechanism of external identity provider
+such as Azure AD. The login can be triggered either by the external IDP, or internally
+by clicking specific authentication button on the log-in page.
+
+
+Configuration steps
+^^^^^^^^^^^^^^^^^^^
+
+To configure Duo Security SAML authentication, use the following steps:
+
+1. From the |RCE| interface, select
+   :menuselection:`Admin --> Authentication`
+2. Activate the `Azure Entra ID` plugin and select :guilabel:`Save`
+3. Go to newly available menu option called `Azure Entra ID` on the left side.
+4. Check the `enabled` check box in the plugin configuration section,
+   and fill in the required SAML information and :guilabel:`Save`, for more details,
+   see :ref:`config-saml-azure`
+
+
+.. _config-saml-azure:
+
+
+Example SAML Azure Entra ID configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Example configuration for SAML 2.0 with Azure Entra ID provider
+
+
+Enabled
+    `True`:
+
+    .. note::
+      Enable or disable this authentication plugin.
+
+
+Auth Cache TTL
+    `30`:
+
+    .. note::
+      Amount of seconds to cache the authentication and permissions check response call for this plugin.
+      Useful for expensive calls like LDAP to improve the performance of the system (0 means disabled).
+
+Debug
+    `True`:
+
+    .. note::
+      Enable or disable debug mode that shows SAML errors in the RhodeCode logs.
+
+
+Auth button name
+    `Azure Entra ID`:
+
+    .. note::
+      Alternative authentication display name. E.g AzureAuth, CorporateID etc.
+
+
+Entity ID
+    `https://sts.windows.net/APP_ID/`:
+
+    .. note::
+      Identity Provider entity/metadata URI. Known as "Microsoft Entra Identifier"
+      E.g. https://sts.windows.net/abcd-c655-dcee-aab7-abcd/
+
+SSO URL
+    `https://login.microsoftonline.com/APP_ID/saml2`:
+
+    .. note::
+      SSO (SingleSignOn) endpoint URL of the IdP. This can be used to initialize login, Known also as Login URL
+      E.g. https://login.microsoftonline.com/abcd-c655-dcee-aab7-abcd/saml2
+
+SLO URL
+    `https://login.microsoftonline.com/APP_ID/saml2`:
+
+    .. note::
+      SLO (SingleLogout) endpoint URL of the IdP. , Known also as Logout URL
+      E.g. https://login.microsoftonline.com/abcd-c655-dcee-aab7-abcd/saml2
+
+x509cert
+    `<CERTIFICATE_STRING>`:
+
+    .. note::
+      Identity provider public x509 certificate. It will be converted to single-line format without headers.
+      Download the raw base64 encoded certificate from the Identity provider and paste it here.
+
+SAML Signature
+    `sha-256`:
+
+    .. note::
+      Type of Algorithm to use for verification of SAML signature on Identity provider side.
+
+SAML Digest
+    `sha-256`:
+
+    .. note::
+      Type of Algorithm to use for verification of SAML digest on Identity provider side.
+
+Service Provider Cert Dir
+    `/etc/rhodecode/conf/saml_ssl/`:
+
+    .. note::
+      Optional directory to store service provider certificate and private keys.
+      Expected certs for the SP should be stored in this folder as:
+
+      * sp.key     Private Key
+      * sp.crt     Public cert
+      * sp_new.crt Future Public cert
+
+      Also you can use other cert to sign the metadata of the SP using the:
+
+      * metadata.key
+      * metadata.crt
+
+Expected NameID Format
+    `nameid-format:emailAddress`:
+
+    .. note::
+      The format that specifies how the NameID is sent to the service provider.
+
+User ID Attribute
+    `user.email`:
+
+    .. note::
+      User ID Attribute name. This defines which attribute in SAML response will be used to link accounts via unique id.
+      Ensure this is returned from DuoSecurity for example via duo_username.
+
+Username Attribute
+    `user.username`:
+
+    .. note::
+      Username Attribute name. This defines which attribute in SAML response will map to a username.
+
+Email Attribute
+    `user.email`:
+
+    .. note::
+      Email Attribute name. This defines which attribute in SAML response will map to an email address.
+
+
+
+Below is example setup from Azure Administration page that can be used with above config.
+
+.. image:: ../images/saml-azure-service-provider-example.png
+   :alt: Azure SAML setup example
+   :scale: 50 %
+
+
+Below is an example attribute mapping set for IDP provider required by the above config.
+
+
+.. image:: ../images/saml-azure-attributes-example.png
+   :alt: Azure SAML setup example
+   :scale: 50 %
  No newline at end of file

docs/release-notes/release-notes-5.1.1.rst

@@ -0,0 +1,40 b''
+|RCE| 5.1.1 |RNS|
+-----------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2024-07-23
+
+
+New Features
+^^^^^^^^^^^^
+
+
+
+General
+^^^^^^^
+
+
+
+Security
+^^^^^^^^
+
+
+
+Performance
+^^^^^^^^^^^
+
+
+
+
+Fixes
+^^^^^
+
+- Fixed problems with JS static files build
+
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+- RhodeCode 5.1.1 is unscheduled bugfix release to address some build issues with 5.1 images

docs/release-notes/release-notes-5.1.2.rst

@@ -0,0 +1,41 b''
+|RCE| 5.1.2 |RNS|
+-----------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2024-09-12
+
+
+New Features
+^^^^^^^^^^^^
+
+
+
+General
+^^^^^^^
+
+
+
+Security
+^^^^^^^^
+
+
+
+Performance
+^^^^^^^^^^^
+
+
+
+
+Fixes
+^^^^^
+
+- Fixed problems with Mercurial authentication after enabling httppostargs.
+  Currently this protocol will be disabled until proper fix is in place
+
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+- RhodeCode 5.1.2 is unscheduled bugfix release to address some build issues with 5.1 images

docs/release-notes/release-notes-5.2.0.rst

@@ -0,0 +1,55 b''
+|RCE| 5.2.0 |RNS|
+-----------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2024-10-09
+
+
+New Features
+^^^^^^^^^^^^
+
+- New artifact storage engines allowing an s3 based uploads
+- Enterprise version only: Added security tab to admin interface and possibility to whitelist specific vcs client versions. Some older versions clients have known security vulnerabilities, now you can disallow them.
+- Enterprise version only: Implemented support for Azure SAML authentication
+
+
+General
+^^^^^^^
+- Bumped version of packaging, gunicorn, orjson, zope.interface and some other requirements
+- Few tweaks and changes to saml plugins to allows easier setup
+- Configs: allow json log format for gunicorn
+- Configs: deprecated old ssh wrapper command and make the v2 the default one
+- Make sure commit-caches propagate to parent repo groups
+- Configs: Moved git lfs path and path of hg large files to ini file
+
+Security
+^^^^^^^^
+
+
+
+Performance
+^^^^^^^^^^^
+
+- description escaper for better performance
+
+Fixes
+^^^^^
+
+- Email notifications not working properly
+- Removed waitress as a default runner
+- Fixed issue with branch permissions
+- Ldap: fixed nested groups extraction logic
+- Fixed possible db corruption in case of filesystem problems
+- Cleanup and improvements to documentation
+- Added Kubernetes deployment section to the documentation
+- Added default value to celery result and broker
+- Fixed broken backends function after python3 migration
+- Explicitly disable mercurial web_push ssl flag to prevent from errors about ssl required
+- VCS: fixed problems with locked repos and with branch permissions reporting
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+- RhodeCode 5.2.0 is a planned major release featuring Azure SAML, whitelist for client versions, s3 artifacts backend and more!

rhodecode/apps/admin/views/security.py

@@ -0,0 +1,46 b''
+# Copyright (C) 2010-2024 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import logging
+
+from rhodecode.apps._base import BaseAppView
+from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator
+
+log = logging.getLogger(__name__)
+
+
+class AdminSecurityView(BaseAppView):
+
+    def load_default_context(self):
+        c = self._get_local_tmpl_context()
+        return c
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    def security(self):
+        c = self.load_default_context()
+        c.active = 'security'
+        return self._get_template_context(c)
+
+
+    @LoginRequired()
+    @HasPermissionAllDecorator('hg.admin')
+    def admin_security_modify_allowed_vcs_client_versions(self):
+        c = self.load_default_context()
+        c.active = 'security'
+        return self._get_template_context(c)

rhodecode/apps/file_store/backends/base.py

@@ -0,0 +1,269 b''
+# Copyright (C) 2016-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import os
+import fsspec  # noqa
+import logging
+
+from rhodecode.lib.ext_json import json
+
+from rhodecode.apps.file_store.utils import sha256_safe, ShardFileReader, get_uid_filename
+from rhodecode.apps.file_store.extensions import resolve_extensions
+from rhodecode.apps.file_store.exceptions import FileNotAllowedException, FileOverSizeException  # noqa: F401
+
+log = logging.getLogger(__name__)
+
+
+class BaseShard:
+
+    metadata_suffix: str = '.metadata'
+    storage_type: str = ''
+    fs = None
+
+    @property
+    def storage_medium(self):
+        if not self.storage_type:
+            raise ValueError('No storage type set for this shard storage_type=""')
+        return getattr(self, self.storage_type)
+
+    def __contains__(self, key):
+        full_path = self.store_path(key)
+        return self.fs.exists(full_path)
+
+    def metadata_convert(self, uid_filename, metadata):
+        return metadata
+
+    def get_metadata_filename(self, uid_filename) -> tuple[str, str]:
+        metadata_file: str = f'{uid_filename}{self.metadata_suffix}'
+        return metadata_file, self.store_path(metadata_file)
+
+    def get_metadata(self, uid_filename, ignore_missing=False) -> dict:
+        _metadata_file, metadata_file_path = self.get_metadata_filename(uid_filename)
+        if ignore_missing and not self.fs.exists(metadata_file_path):
+            return {}
+
+        with self.fs.open(metadata_file_path, 'rb') as f:
+            metadata = json.loads(f.read())
+
+        metadata = self.metadata_convert(uid_filename, metadata)
+        return metadata
+
+    def _store(self, key: str, uid_key: str, value_reader, max_filesize: int | None = None, metadata: dict | None = None, **kwargs):
+        raise NotImplementedError
+
+    def store(self, key: str, uid_key: str, value_reader, max_filesize: int | None = None, metadata: dict | None = None, **kwargs):
+        return self._store(key, uid_key, value_reader, max_filesize, metadata, **kwargs)
+
+    def _fetch(self, key, presigned_url_expires: int = 0):
+        raise NotImplementedError
+
+    def fetch(self, key, **kwargs) -> tuple[ShardFileReader, dict]:
+        return self._fetch(key)
+
+    def _delete(self, key):
+        if key not in self:
+            log.exception(f'requested key={key} not found in {self}')
+            raise KeyError(key)
+
+        metadata = self.get_metadata(key)
+        _metadata_file, metadata_file_path = self.get_metadata_filename(key)
+        artifact_file_path = metadata['filename_uid_path']
+        self.fs.rm(artifact_file_path)
+        self.fs.rm(metadata_file_path)
+
+        return 1
+
+    def delete(self, key):
+        raise NotImplementedError
+
+    def store_path(self, uid_filename):
+        raise NotImplementedError
+
+
+class BaseFileStoreBackend:
+    _shards = tuple()
+    _shard_cls = BaseShard
+    _config: dict | None = None
+    _storage_path: str = ''
+
+    def __init__(self, settings, extension_groups=None):
+        self._config = settings
+        extension_groups = extension_groups or ['any']
+        self.extensions = resolve_extensions([], groups=extension_groups)
+
+    def __contains__(self, key):
+        return self.filename_exists(key)
+
+    def __repr__(self):
+        return f'<{self.__class__.__name__}(storage={self.storage_path})>'
+
+    @property
+    def storage_path(self):
+        return self._storage_path
+
+    @classmethod
+    def get_shard_index(cls, filename: str, num_shards) -> int:
+        # Generate a hash value from the filename
+        hash_value = sha256_safe(filename)
+
+        # Convert the hash value to an integer
+        hash_int = int(hash_value, 16)
+
+        # Map the hash integer to a shard number between 1 and num_shards
+        shard_number = (hash_int % num_shards)
+
+        return shard_number
+
+    @classmethod
+    def apply_counter(cls, counter: int, filename: str) -> str:
+        """
+        Apply a counter to the filename.
+
+        :param counter: The counter value to apply.
+        :param filename: The original filename.
+        :return: The modified filename with the counter.
+        """
+        name_counted = f'{counter:d}-{filename}'
+        return name_counted
+
+    def _get_shard(self, key) -> _shard_cls:
+        index = self.get_shard_index(key, len(self._shards))
+        shard = self._shards[index]
+        return shard
+
+    def get_conf(self, key, pop=False):
+        if key not in self._config:
+            raise ValueError(
+                f"No configuration key '{key}', please make sure it exists in filestore config")
+        val = self._config[key]
+        if pop:
+            del self._config[key]
+        return val
+
+    def filename_allowed(self, filename, extensions=None):
+        """Checks if a filename has an allowed extension
+
+        :param filename: base name of file
+        :param extensions: iterable of extensions (or self.extensions)
+        """
+        _, ext = os.path.splitext(filename)
+        return self.extension_allowed(ext, extensions)
+
+    def extension_allowed(self, ext, extensions=None):
+        """
+        Checks if an extension is permitted. Both e.g. ".jpg" and
+        "jpg" can be passed in. Extension lookup is case-insensitive.
+
+        :param ext: extension to check
+        :param extensions: iterable of extensions to validate against (or self.extensions)
+        """
+        def normalize_ext(_ext):
+            if _ext.startswith('.'):
+                _ext = _ext[1:]
+            return _ext.lower()
+
+        extensions = extensions or self.extensions
+        if not extensions:
+            return True
+
+        ext = normalize_ext(ext)
+
+        return ext in [normalize_ext(x) for x in extensions]
+
+    def filename_exists(self, uid_filename):
+        shard = self._get_shard(uid_filename)
+        return uid_filename in shard
+
+    def store_path(self, uid_filename):
+        """
+        Returns absolute file path of the uid_filename
+        """
+        shard = self._get_shard(uid_filename)
+        return shard.store_path(uid_filename)
+
+    def store_metadata(self, uid_filename):
+        shard = self._get_shard(uid_filename)
+        return shard.get_metadata_filename(uid_filename)
+
+    def store(self, filename, value_reader, extensions=None, metadata=None, max_filesize=None, randomized_name=True, **kwargs):
+        extensions = extensions or self.extensions
+
+        if not self.filename_allowed(filename, extensions):
+            msg = f'filename {filename} does not allow extensions {extensions}'
+            raise FileNotAllowedException(msg)
+
+        # # TODO: check why we need this setting ? it looks stupid...
+        # no_body_seek is used in stream mode importer somehow
+        # no_body_seek = kwargs.pop('no_body_seek', False)
+        # if no_body_seek:
+        #     pass
+        # else:
+        #     value_reader.seek(0)
+
+        uid_filename = kwargs.pop('uid_filename', None)
+        if uid_filename is None:
+            uid_filename = get_uid_filename(filename, randomized=randomized_name)
+
+        shard = self._get_shard(uid_filename)
+
+        return shard.store(filename, uid_filename, value_reader, max_filesize, metadata, **kwargs)
+
+    def import_to_store(self, value_reader, org_filename, uid_filename, metadata, **kwargs):
+        shard = self._get_shard(uid_filename)
+        max_filesize = None
+        return shard.store(org_filename, uid_filename, value_reader, max_filesize, metadata, import_mode=True)
+
+    def delete(self, uid_filename):
+        shard = self._get_shard(uid_filename)
+        return shard.delete(uid_filename)
+
+    def fetch(self, uid_filename) -> tuple[ShardFileReader, dict]:
+        shard = self._get_shard(uid_filename)
+        return shard.fetch(uid_filename)
+
+    def get_metadata(self, uid_filename, ignore_missing=False) -> dict:
+        shard = self._get_shard(uid_filename)
+        return shard.get_metadata(uid_filename, ignore_missing=ignore_missing)
+
+    def iter_keys(self):
+        for shard in self._shards:
+            if shard.fs.exists(shard.storage_medium):
+                for path, _dirs, _files in shard.fs.walk(shard.storage_medium):
+                    for key_file_path in _files:
+                        if key_file_path.endswith(shard.metadata_suffix):
+                            yield shard, key_file_path
+
+    def iter_artifacts(self):
+        for shard, key_file in self.iter_keys():
+            json_key = f"{shard.storage_medium}/{key_file}"
+            with shard.fs.open(json_key, 'rb') as f:
+                yield shard, json.loads(f.read())['filename_uid']
+
+    def get_statistics(self):
+        total_files = 0
+        total_size = 0
+        meta = {}
+
+        for shard, key_file in self.iter_keys():
+            json_key = f"{shard.storage_medium}/{key_file}"
+            with shard.fs.open(json_key, 'rb') as f:
+                total_files += 1
+                metadata = json.loads(f.read())
+                total_size += metadata['size']
+
+        return total_files, total_size, meta

rhodecode/apps/file_store/backends/filesystem.py

@@ -0,0 +1,183 b''
+# Copyright (C) 2016-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import os
+import hashlib
+import functools
+import time
+import logging
+
+from .. import config_keys
+from ..exceptions import FileOverSizeException
+from ..backends.base import BaseFileStoreBackend, fsspec, BaseShard, ShardFileReader
+
+from ....lib.ext_json import json
+
+log = logging.getLogger(__name__)
+
+
+class FileSystemShard(BaseShard):
+    METADATA_VER = 'v2'
+    BACKEND_TYPE = config_keys.backend_filesystem
+    storage_type: str = 'directory'
+
+    def __init__(self, index, directory, directory_folder, fs, **settings):
+        self._index: int = index
+        self._directory: str = directory
+        self._directory_folder: str = directory_folder
+        self.fs = fs
+
+    @property
+    def directory(self) -> str:
+        """Cache directory final path."""
+        return os.path.join(self._directory, self._directory_folder)
+
+    def _write_file(self, full_path, iterator, max_filesize, mode='wb'):
+
+        # ensure dir exists
+        destination, _ = os.path.split(full_path)
+        if not self.fs.exists(destination):
+            self.fs.makedirs(destination)
+
+        writer = self.fs.open(full_path, mode)
+
+        digest = hashlib.sha256()
+        oversize_cleanup = False
+        with writer:
+            size = 0
+            for chunk in iterator:
+                size += len(chunk)
+                digest.update(chunk)
+                writer.write(chunk)
+
+                if max_filesize and size > max_filesize:
+                    oversize_cleanup = True
+                    # free up the copied file, and raise exc
+                    break
+
+            writer.flush()
+            # Get the file descriptor
+            fd = writer.fileno()
+
+            # Sync the file descriptor to disk, helps with NFS cases...
+            os.fsync(fd)
+
+        if oversize_cleanup:
+            self.fs.rm(full_path)
+            raise FileOverSizeException(f'given file is over size limit ({max_filesize}): {full_path}')
+
+        sha256 = digest.hexdigest()
+        log.debug('written new artifact under %s, sha256: %s', full_path, sha256)
+        return size, sha256
+
+    def _store(self, key: str, uid_key, value_reader, max_filesize: int | None = None, metadata: dict | None = None, **kwargs):
+
+        filename = key
+        uid_filename = uid_key
+        full_path = self.store_path(uid_filename)
+
+        # STORE METADATA
+        _metadata = {
+            "version": self.METADATA_VER,
+            "store_type": self.BACKEND_TYPE,
+
+            "filename": filename,
+            "filename_uid_path": full_path,
+            "filename_uid": uid_filename,
+            "sha256": "",  # NOTE: filled in by reader iteration
+
+            "store_time": time.time(),
+
+            "size": 0
+        }
+
+        if metadata:
+            if kwargs.pop('import_mode', False):
+                # in import mode, we don't need to compute metadata, we just take the old version
+                _metadata["import_mode"] = True
+            else:
+                _metadata.update(metadata)
+
+        read_iterator = iter(functools.partial(value_reader.read, 2**22), b'')
+        size, sha256 = self._write_file(full_path, read_iterator, max_filesize)
+        _metadata['size'] = size
+        _metadata['sha256'] = sha256
+
+        # after storing the artifacts, we write the metadata present
+        _metadata_file, metadata_file_path = self.get_metadata_filename(uid_key)
+
+        with self.fs.open(metadata_file_path, 'wb') as f:
+            f.write(json.dumps(_metadata))
+
+        return uid_filename, _metadata
+
+    def store_path(self, uid_filename):
+        """
+        Returns absolute file path of the uid_filename
+        """
+        return os.path.join(self._directory, self._directory_folder, uid_filename)
+
+    def _fetch(self, key, presigned_url_expires: int = 0):
+        if key not in self:
+            log.exception(f'requested key={key} not found in {self}')
+            raise KeyError(key)
+
+        metadata = self.get_metadata(key)
+
+        file_path = metadata['filename_uid_path']
+        if presigned_url_expires and presigned_url_expires > 0:
+            metadata['url'] = self.fs.url(file_path, expires=presigned_url_expires)
+
+        return ShardFileReader(self.fs.open(file_path, 'rb')), metadata
+
+    def delete(self, key):
+        return self._delete(key)
+
+
+class FileSystemBackend(BaseFileStoreBackend):
+    shard_name: str = 'shard_{:03d}'
+    _shard_cls = FileSystemShard
+
+    def __init__(self, settings):
+        super().__init__(settings)
+
+        store_dir = self.get_conf(config_keys.filesystem_storage_path)
+        directory = os.path.expanduser(store_dir)
+
+        self._directory = directory
+        self._storage_path = directory  # common path for all from BaseCache
+        self._shard_count = int(self.get_conf(config_keys.filesystem_shards, pop=True))
+        if self._shard_count < 1:
+            raise ValueError(f'{config_keys.filesystem_shards} must be 1 or more')
+
+        log.debug('Initializing %s file_store instance', self)
+        fs = fsspec.filesystem('file')
+
+        if not fs.exists(self._directory):
+            fs.makedirs(self._directory, exist_ok=True)
+
+        self._shards = tuple(
+            self._shard_cls(
+                index=num,
+                directory=directory,
+                directory_folder=self.shard_name.format(num),
+                fs=fs,
+                **settings,
+            )
+            for num in range(self._shard_count)
+        )

rhodecode/apps/file_store/backends/filesystem_legacy.py

@@ -0,0 +1,278 b''
+# Copyright (C) 2016-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import errno
+import os
+import hashlib
+import functools
+import time
+import logging
+
+from .. import config_keys
+from ..exceptions import FileOverSizeException
+from ..backends.base import BaseFileStoreBackend, fsspec, BaseShard, ShardFileReader
+
+from ....lib.ext_json import json
+
+log = logging.getLogger(__name__)
+
+
+class LegacyFileSystemShard(BaseShard):
+    # legacy ver
+    METADATA_VER = 'v2'
+    BACKEND_TYPE = config_keys.backend_legacy_filesystem
+    storage_type: str = 'dir_struct'
+
+    # legacy suffix
+    metadata_suffix: str = '.meta'
+
+    @classmethod
+    def _sub_store_from_filename(cls, filename):
+        return filename[:2]
+
+    @classmethod
+    def apply_counter(cls, counter, filename):
+        name_counted = '%d-%s' % (counter, filename)
+        return name_counted
+
+    @classmethod
+    def safe_make_dirs(cls, dir_path):
+        if not os.path.exists(dir_path):
+            try:
+                os.makedirs(dir_path)
+            except OSError as e:
+                if e.errno != errno.EEXIST:
+                    raise
+                return
+
+    @classmethod
+    def resolve_name(cls, name, directory):
+        """
+        Resolves a unique name and the correct path. If a filename
+        for that path already exists then a numeric prefix with values > 0 will be
+        added, for example test.jpg -> 1-test.jpg etc. initially file would have 0 prefix.
+
+        :param name: base name of file
+        :param directory: absolute directory path
+        """
+
+        counter = 0
+        while True:
+            name_counted = cls.apply_counter(counter, name)
+
+            # sub_store prefix to optimize disk usage, e.g some_path/ab/final_file
+            sub_store: str = cls._sub_store_from_filename(name_counted)
+            sub_store_path: str = os.path.join(directory, sub_store)
+            cls.safe_make_dirs(sub_store_path)
+
+            path = os.path.join(sub_store_path, name_counted)
+            if not os.path.exists(path):
+                return name_counted, path
+            counter += 1
+
+    def __init__(self, index, directory, directory_folder, fs, **settings):
+        self._index: int = index
+        self._directory: str = directory
+        self._directory_folder: str = directory_folder
+        self.fs = fs
+
+    @property
+    def dir_struct(self) -> str:
+        """Cache directory final path."""
+        return os.path.join(self._directory, '0-')
+
+    def _write_file(self, full_path, iterator, max_filesize, mode='wb'):
+
+        # ensure dir exists
+        destination, _ = os.path.split(full_path)
+        if not self.fs.exists(destination):
+            self.fs.makedirs(destination)
+
+        writer = self.fs.open(full_path, mode)
+
+        digest = hashlib.sha256()
+        oversize_cleanup = False
+        with writer:
+            size = 0
+            for chunk in iterator:
+                size += len(chunk)
+                digest.update(chunk)
+                writer.write(chunk)
+
+                if max_filesize and size > max_filesize:
+                    # free up the copied file, and raise exc
+                    oversize_cleanup = True
+                    break
+
+            writer.flush()
+            # Get the file descriptor
+            fd = writer.fileno()
+
+            # Sync the file descriptor to disk, helps with NFS cases...
+            os.fsync(fd)
+
+        if oversize_cleanup:
+            self.fs.rm(full_path)
+            raise FileOverSizeException(f'given file is over size limit ({max_filesize}): {full_path}')
+
+        sha256 = digest.hexdigest()
+        log.debug('written new artifact under %s, sha256: %s', full_path, sha256)
+        return size, sha256
+
+    def _store(self, key: str, uid_key, value_reader, max_filesize: int | None = None, metadata: dict | None = None, **kwargs):
+
+        filename = key
+        uid_filename = uid_key
+
+        # NOTE:, also apply N- Counter...
+        uid_filename, full_path = self.resolve_name(uid_filename, self._directory)
+
+        # STORE METADATA
+        # TODO: make it compatible, and backward proof
+        _metadata = {
+            "version": self.METADATA_VER,
+
+            "filename": filename,
+            "filename_uid_path": full_path,
+            "filename_uid": uid_filename,
+            "sha256": "",  # NOTE: filled in by reader iteration
+
+            "store_time": time.time(),
+
+            "size": 0
+        }
+        if metadata:
+            _metadata.update(metadata)
+
+        read_iterator = iter(functools.partial(value_reader.read, 2**22), b'')
+        size, sha256 = self._write_file(full_path, read_iterator, max_filesize)
+        _metadata['size'] = size
+        _metadata['sha256'] = sha256
+
+        # after storing the artifacts, we write the metadata present
+        _metadata_file, metadata_file_path = self.get_metadata_filename(uid_filename)
+
+        with self.fs.open(metadata_file_path, 'wb') as f:
+            f.write(json.dumps(_metadata))
+
+        return uid_filename, _metadata
+
+    def store_path(self, uid_filename):
+        """
+        Returns absolute file path of the uid_filename
+        """
+        prefix_dir = ''
+        if '/' in uid_filename:
+            prefix_dir, filename = uid_filename.split('/')
+            sub_store = self._sub_store_from_filename(filename)
+        else:
+            sub_store = self._sub_store_from_filename(uid_filename)
+
+        return os.path.join(self._directory, prefix_dir, sub_store, uid_filename)
+
+    def metadata_convert(self, uid_filename, metadata):
+        # NOTE: backward compat mode here... this is for file created PRE 5.2 system
+        if 'meta_ver' in metadata:
+            full_path = self.store_path(uid_filename)
+            metadata = {
+                "_converted": True,
+                "_org": metadata,
+                "version": self.METADATA_VER,
+                "store_type": self.BACKEND_TYPE,
+
+                "filename": metadata['filename'],
+                "filename_uid_path": full_path,
+                "filename_uid": uid_filename,
+                "sha256": metadata['sha256'],
+
+                "store_time": metadata['time'],
+
+                "size": metadata['size']
+            }
+        return metadata
+
+    def _fetch(self, key, presigned_url_expires: int = 0):
+        if key not in self:
+            log.exception(f'requested key={key} not found in {self}')
+            raise KeyError(key)
+
+        metadata = self.get_metadata(key)
+
+        file_path = metadata['filename_uid_path']
+        if presigned_url_expires and presigned_url_expires > 0:
+            metadata['url'] = self.fs.url(file_path, expires=presigned_url_expires)
+
+        return ShardFileReader(self.fs.open(file_path, 'rb')), metadata
+
+    def delete(self, key):
+        return self._delete(key)
+
+    def _delete(self, key):
+        if key not in self:
+            log.exception(f'requested key={key} not found in {self}')
+            raise KeyError(key)
+
+        metadata = self.get_metadata(key)
+        metadata_file, metadata_file_path = self.get_metadata_filename(key)
+        artifact_file_path = metadata['filename_uid_path']
+        self.fs.rm(artifact_file_path)
+        self.fs.rm(metadata_file_path)
+
+    def get_metadata_filename(self, uid_filename) -> tuple[str, str]:
+
+        metadata_file: str = f'{uid_filename}{self.metadata_suffix}'
+        uid_path_in_store = self.store_path(uid_filename)
+
+        metadata_file_path = f'{uid_path_in_store}{self.metadata_suffix}'
+        return metadata_file, metadata_file_path
+
+
+class LegacyFileSystemBackend(BaseFileStoreBackend):
+    _shard_cls = LegacyFileSystemShard
+
+    def __init__(self, settings):
+        super().__init__(settings)
+
+        store_dir = self.get_conf(config_keys.legacy_filesystem_storage_path)
+        directory = os.path.expanduser(store_dir)
+
+        self._directory = directory
+        self._storage_path = directory  # common path for all from BaseCache
+
+        log.debug('Initializing %s file_store instance', self)
+        fs = fsspec.filesystem('file')
+
+        if not fs.exists(self._directory):
+            fs.makedirs(self._directory, exist_ok=True)
+
+        # legacy system uses single shard
+        self._shards = tuple(
+            [
+                self._shard_cls(
+                    index=0,
+                    directory=directory,
+                    directory_folder='',
+                    fs=fs,
+                    **settings,
+                )
+            ]
+        )
+
+    @classmethod
+    def get_shard_index(cls, filename: str, num_shards) -> int:
+        # legacy filesystem doesn't use shards, and always uses single shard
+        return 0

rhodecode/apps/file_store/backends/objectstore.py

@@ -0,0 +1,184 b''
+# Copyright (C) 2016-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+
+import os
+import hashlib
+import functools
+import time
+import logging
+
+from .. import config_keys
+from ..exceptions import FileOverSizeException
+from ..backends.base import BaseFileStoreBackend, fsspec, BaseShard, ShardFileReader
+
+from ....lib.ext_json import json
+
+log = logging.getLogger(__name__)
+
+
+class S3Shard(BaseShard):
+    METADATA_VER = 'v2'
+    BACKEND_TYPE = config_keys.backend_objectstore
+    storage_type: str = 'bucket'
+
+    def __init__(self, index, bucket, bucket_folder, fs, **settings):
+        self._index: int = index
+        self._bucket_main: str = bucket
+        self._bucket_folder: str = bucket_folder
+
+        self.fs = fs
+
+    @property
+    def bucket(self) -> str:
+        """Cache bucket final path."""
+        return os.path.join(self._bucket_main, self._bucket_folder)
+
+    def _write_file(self, full_path, iterator, max_filesize, mode='wb'):
+
+        # ensure dir exists
+        destination, _ = os.path.split(full_path)
+        if not self.fs.exists(destination):
+            self.fs.makedirs(destination)
+
+        writer = self.fs.open(full_path, mode)
+
+        digest = hashlib.sha256()
+        oversize_cleanup = False
+        with writer:
+            size = 0
+            for chunk in iterator:
+                size += len(chunk)
+                digest.update(chunk)
+                writer.write(chunk)
+
+                if max_filesize and size > max_filesize:
+                    oversize_cleanup = True
+                    # free up the copied file, and raise exc
+                    break
+
+        if oversize_cleanup:
+            self.fs.rm(full_path)
+            raise FileOverSizeException(f'given file is over size limit ({max_filesize}): {full_path}')
+
+        sha256 = digest.hexdigest()
+        log.debug('written new artifact under %s, sha256: %s', full_path, sha256)
+        return size, sha256
+
+    def _store(self, key: str, uid_key, value_reader, max_filesize: int | None = None, metadata: dict | None = None, **kwargs):
+
+        filename = key
+        uid_filename = uid_key
+        full_path = self.store_path(uid_filename)
+
+        # STORE METADATA
+        _metadata = {
+            "version": self.METADATA_VER,
+            "store_type": self.BACKEND_TYPE,
+
+            "filename": filename,
+            "filename_uid_path": full_path,
+            "filename_uid": uid_filename,
+            "sha256": "",  # NOTE: filled in by reader iteration
+
+            "store_time": time.time(),
+
+            "size": 0
+        }
+
+        if metadata:
+            if kwargs.pop('import_mode', False):
+                # in import mode, we don't need to compute metadata, we just take the old version
+                _metadata["import_mode"] = True
+            else:
+                _metadata.update(metadata)
+
+        read_iterator = iter(functools.partial(value_reader.read, 2**22), b'')
+        size, sha256 = self._write_file(full_path, read_iterator, max_filesize)
+        _metadata['size'] = size
+        _metadata['sha256'] = sha256
+
+        # after storing the artifacts, we write the metadata present
+        metadata_file, metadata_file_path = self.get_metadata_filename(uid_key)
+
+        with self.fs.open(metadata_file_path, 'wb') as f:
+            f.write(json.dumps(_metadata))
+
+        return uid_filename, _metadata
+
+    def store_path(self, uid_filename):
+        """
+        Returns absolute file path of the uid_filename
+        """
+        return os.path.join(self._bucket_main, self._bucket_folder, uid_filename)
+
+    def _fetch(self, key, presigned_url_expires: int = 0):
+        if key not in self:
+            log.exception(f'requested key={key} not found in {self}')
+            raise KeyError(key)
+
+        metadata_file, metadata_file_path = self.get_metadata_filename(key)
+        with self.fs.open(metadata_file_path, 'rb') as f:
+            metadata = json.loads(f.read())
+
+        file_path = metadata['filename_uid_path']
+        if presigned_url_expires and presigned_url_expires > 0:
+            metadata['url'] = self.fs.url(file_path, expires=presigned_url_expires)
+
+        return ShardFileReader(self.fs.open(file_path, 'rb')), metadata
+
+    def delete(self, key):
+        return self._delete(key)
+
+
+class ObjectStoreBackend(BaseFileStoreBackend):
+    shard_name: str = 'shard-{:03d}'
+    _shard_cls = S3Shard
+
+    def __init__(self, settings):
+        super().__init__(settings)
+
+        self._shard_count = int(self.get_conf(config_keys.objectstore_bucket_shards, pop=True))
+        if self._shard_count < 1:
+            raise ValueError('cache_shards must be 1 or more')
+
+        self._bucket = settings.pop(config_keys.objectstore_bucket)
+        if not self._bucket:
+            raise ValueError(f'{config_keys.objectstore_bucket} needs to have a value')
+
+        objectstore_url = self.get_conf(config_keys.objectstore_url)
+        key = settings.pop(config_keys.objectstore_key)
+        secret = settings.pop(config_keys.objectstore_secret)
+
+        self._storage_path = objectstore_url  # common path for all from BaseCache
+        log.debug('Initializing %s file_store instance', self)
+        fs = fsspec.filesystem('s3', anon=False, endpoint_url=objectstore_url, key=key, secret=secret)
+
+        # init main bucket
+        if not fs.exists(self._bucket):
+            fs.mkdir(self._bucket)
+
+        self._shards = tuple(
+            self._shard_cls(
+                index=num,
+                bucket=self._bucket,
+                bucket_folder=self.shard_name.format(num),
+                fs=fs,
+                **settings,
+            )
+            for num in range(self._shard_count)
+        )

rhodecode/apps/file_store/tests/test_filestore_backends.py

@@ -0,0 +1,128 b''
+# Copyright (C) 2010-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import pytest
+
+from rhodecode.apps import file_store
+from rhodecode.apps.file_store import config_keys
+from rhodecode.apps.file_store.backends.filesystem_legacy import LegacyFileSystemBackend
+from rhodecode.apps.file_store.backends.filesystem import FileSystemBackend
+from rhodecode.apps.file_store.backends.objectstore import ObjectStoreBackend
+from rhodecode.apps.file_store.exceptions import FileNotAllowedException, FileOverSizeException
+
+from rhodecode.apps.file_store import utils as store_utils
+from rhodecode.apps.file_store.tests import random_binary_file, file_store_instance
+
+
+class TestFileStoreBackends:
+
+    @pytest.mark.parametrize('backend_type, expected_instance', [
+        (config_keys.backend_legacy_filesystem, LegacyFileSystemBackend),
+        (config_keys.backend_filesystem, FileSystemBackend),
+        (config_keys.backend_objectstore, ObjectStoreBackend),
+    ])
+    def test_get_backend(self, backend_type, expected_instance, ini_settings):
+        config = ini_settings
+        config[config_keys.backend_type] = backend_type
+        f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+        assert isinstance(f_store, expected_instance)
+
+    @pytest.mark.parametrize('backend_type, expected_instance', [
+        (config_keys.backend_legacy_filesystem, LegacyFileSystemBackend),
+        (config_keys.backend_filesystem, FileSystemBackend),
+        (config_keys.backend_objectstore, ObjectStoreBackend),
+    ])
+    def test_store_and_read(self, backend_type, expected_instance, ini_settings, random_binary_file):
+        filename, temp_file = random_binary_file
+        config = ini_settings
+        config[config_keys.backend_type] = backend_type
+        f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+        metadata = {
+            'user_uploaded': {
+                'username': 'user1',
+                'user_id': 10,
+                'ip': '10.20.30.40'
+            }
+        }
+        store_fid, metadata = f_store.store(filename, temp_file, extra_metadata=metadata)
+        assert store_fid
+        assert metadata
+
+        # read-after write
+        reader, metadata2 = f_store.fetch(store_fid)
+        assert reader
+        assert metadata2['filename'] == filename
+
+    @pytest.mark.parametrize('backend_type, expected_instance', [
+        (config_keys.backend_legacy_filesystem, LegacyFileSystemBackend),
+        (config_keys.backend_filesystem, FileSystemBackend),
+        (config_keys.backend_objectstore, ObjectStoreBackend),
+    ])
+    def test_store_file_not_allowed(self, backend_type, expected_instance, ini_settings, random_binary_file):
+        filename, temp_file = random_binary_file
+        config = ini_settings
+        config[config_keys.backend_type] = backend_type
+        f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+        with pytest.raises(FileNotAllowedException):
+            f_store.store('notallowed.exe', temp_file, extensions=['.txt'])
+
+    @pytest.mark.parametrize('backend_type, expected_instance', [
+        (config_keys.backend_legacy_filesystem, LegacyFileSystemBackend),
+        (config_keys.backend_filesystem, FileSystemBackend),
+        (config_keys.backend_objectstore, ObjectStoreBackend),
+    ])
+    def test_store_file_over_size(self, backend_type, expected_instance, ini_settings, random_binary_file):
+        filename, temp_file = random_binary_file
+        config = ini_settings
+        config[config_keys.backend_type] = backend_type
+        f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+        with pytest.raises(FileOverSizeException):
+            f_store.store('toobig.exe', temp_file, extensions=['.exe'], max_filesize=124)
+
+    @pytest.mark.parametrize('backend_type, expected_instance, extra_conf', [
+        (config_keys.backend_legacy_filesystem, LegacyFileSystemBackend, {}),
+        (config_keys.backend_filesystem, FileSystemBackend, {config_keys.filesystem_storage_path: '/tmp/test-fs-store'}),
+        (config_keys.backend_objectstore, ObjectStoreBackend, {config_keys.objectstore_bucket: 'test-bucket'}),
+    ])
+    def test_store_stats_and_keys(self, backend_type, expected_instance, extra_conf, ini_settings, random_binary_file):
+        config = ini_settings
+        config[config_keys.backend_type] = backend_type
+        config.update(extra_conf)
+
+        f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+
+        # purge storage before running
+        for shard, k in f_store.iter_artifacts():
+            f_store.delete(k)
+
+        for i in range(10):
+            filename, temp_file = random_binary_file
+
+            metadata = {
+                'user_uploaded': {
+                    'username': 'user1',
+                    'user_id': 10,
+                    'ip': '10.20.30.40'
+                }
+            }
+            store_fid, metadata = f_store.store(filename, temp_file, extra_metadata=metadata)
+            assert store_fid
+            assert metadata
+
+        cnt, size, meta = f_store.get_statistics()
+        assert cnt == 10
+        assert 10 == len(list(f_store.iter_keys()))

rhodecode/apps/file_store/tests/test_filestore_filesystem_backend.py

@@ -0,0 +1,52 b''
+# Copyright (C) 2010-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import pytest
+
+from rhodecode.apps.file_store import utils as store_utils
+from rhodecode.apps.file_store import config_keys
+from rhodecode.apps.file_store.tests import generate_random_filename
+
+
[e-mail]()
+def file_store_filesystem_instance(ini_settings):
+    config = ini_settings
+    config[config_keys.backend_type] = config_keys.backend_filesystem
+    f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+    return f_store
+
+
+class TestFileStoreFileSystemBackend:
+
+    @pytest.mark.parametrize('filename', [generate_random_filename() for _ in range(10)])
+    def test_get_shard_number(self, filename, file_store_filesystem_instance):
+        shard_number = file_store_filesystem_instance.get_shard_index(filename, len(file_store_filesystem_instance._shards))
+        # Check that the shard number is between 0 and max-shards
+        assert 0 <= shard_number <= len(file_store_filesystem_instance._shards)
+
+    @pytest.mark.parametrize('filename, expected_shard_num', [
+        ('my-name-1', 3),
+        ('my-name-2', 2),
+        ('my-name-3', 4),
+        ('my-name-4', 1),
+
+        ('rhodecode-enterprise-ce', 5),
+        ('rhodecode-enterprise-ee', 6),
+    ])
+    def test_get_shard_number_consistency(self, filename, expected_shard_num, file_store_filesystem_instance):
+        shard_number = file_store_filesystem_instance.get_shard_index(filename, len(file_store_filesystem_instance._shards))
+        assert expected_shard_num == shard_number

rhodecode/apps/file_store/tests/test_filestore_legacy_and_v2_compatability.py

@@ -0,0 +1,17 b''
+# Copyright (C) 2010-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
  No newline at end of file

rhodecode/apps/file_store/tests/test_filestore_legacy_backend.py

@@ -0,0 +1,52 b''
+# Copyright (C) 2010-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import pytest
+
+from rhodecode.apps.file_store import utils as store_utils
+from rhodecode.apps.file_store import config_keys
+from rhodecode.apps.file_store.tests import generate_random_filename
+
+
[e-mail]()
+def file_store_legacy_instance(ini_settings):
+    config = ini_settings
+    config[config_keys.backend_type] = config_keys.backend_legacy_filesystem
+    f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+    return f_store
+
+
+class TestFileStoreLegacyBackend:
+
+    @pytest.mark.parametrize('filename', [generate_random_filename() for _ in range(10)])
+    def test_get_shard_number(self, filename, file_store_legacy_instance):
+        shard_number = file_store_legacy_instance.get_shard_index(filename, len(file_store_legacy_instance._shards))
+        # Check that the shard number is 0 for legacy filesystem store we don't use shards
+        assert shard_number == 0
+
+    @pytest.mark.parametrize('filename, expected_shard_num', [
+        ('my-name-1', 0),
+        ('my-name-2', 0),
+        ('my-name-3', 0),
+        ('my-name-4', 0),
+
+        ('rhodecode-enterprise-ce', 0),
+        ('rhodecode-enterprise-ee', 0),
+    ])
+    def test_get_shard_number_consistency(self, filename, expected_shard_num, file_store_legacy_instance):
+        shard_number = file_store_legacy_instance.get_shard_index(filename, len(file_store_legacy_instance._shards))
+        assert expected_shard_num == shard_number

rhodecode/apps/file_store/tests/test_filestore_objectstore_backend.py

@@ -0,0 +1,52 b''
+# Copyright (C) 2010-2023 RhodeCode GmbH
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License, version 3
+# (only), as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This program is dual-licensed. If you wish to learn more about the
+# RhodeCode Enterprise Edition, including its added features, Support services,
+# and proprietary license terms, please see https://rhodecode.com/licenses/
+import pytest
+
+from rhodecode.apps.file_store import utils as store_utils
+from rhodecode.apps.file_store import config_keys
+from rhodecode.apps.file_store.tests import generate_random_filename
+
+
[e-mail]()
+def file_store_objectstore_instance(ini_settings):
+    config = ini_settings
+    config[config_keys.backend_type] = config_keys.backend_objectstore
+    f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+    return f_store
+
+
+class TestFileStoreObjectStoreBackend:
+
+    @pytest.mark.parametrize('filename', [generate_random_filename() for _ in range(10)])
+    def test_get_shard_number(self, filename, file_store_objectstore_instance):
+        shard_number = file_store_objectstore_instance.get_shard_index(filename, len(file_store_objectstore_instance._shards))
+        # Check that the shard number is between 0 and shards
+        assert 0 <= shard_number <= len(file_store_objectstore_instance._shards)
+
+    @pytest.mark.parametrize('filename, expected_shard_num', [
+        ('my-name-1', 3),
+        ('my-name-2', 2),
+        ('my-name-3', 4),
+        ('my-name-4', 1),
+
+        ('rhodecode-enterprise-ce', 5),
+        ('rhodecode-enterprise-ee', 6),
+    ])
+    def test_get_shard_number_consistency(self, filename, expected_shard_num, file_store_objectstore_instance):
+        shard_number = file_store_objectstore_instance.get_shard_index(filename, len(file_store_objectstore_instance._shards))
+        assert expected_shard_num == shard_number

.bumpversion.cfg

@@ -1,5 +1,5 b''
 [bumpversion]
-current_version = 5.1.2
+current_version = 5.2.0
 message = release: Bump version {current_version} to {new_version}
 
 [bumpversion:file:rhodecode/VERSION]

.hgignore

@@ -1,71 +1,71 b''
 syntax: glob
 
 *.egg
 *.egg-info
 *.idea
 *.orig
 *.pyc
 *.sqlite-journal
 *.swp
 *.tox
 *.DS_Store*
 rhodecode/public/js/src/components/**/*.css
 
 syntax: regexp
 
 #.filename
 ^\.settings$
 ^\.project$
 ^\.pydevproject$
 ^\.coverage$
 ^\.cache.*$
 ^\.ruff_cache.*$
 ^\.rhodecode$
 
 ^rcextensions
 ^.dev
 ^._dev
 ^build/
 ^coverage\.xml$
 ^data$
 ^\.eggs/
 ^configs/data$
 ^dev.ini$
 ^acceptance_tests/dev.*\.ini$
 ^dist/
 ^fabfile.py
 ^htmlcov
 ^junit\.xml$
 ^node_modules/
 ^node_binaries/
 ^pylint.log$
 ^rcextensions/
 ^result$
 ^rhodecode/public/css/style.css$
 ^rhodecode/public/css/style-polymer.css$
 ^rhodecode/public/css/style-ipython.css$
 ^rhodecode/public/js/rhodecode-components.html$
 ^rhodecode/public/js/rhodecode-components.js$
 ^rhodecode/public/js/scripts.js$
 ^rhodecode/public/js/scripts.min.js$
 ^rhodecode/public/js/src/components/root-styles.gen.html$
 ^rhodecode/public/js/vendors/webcomponentsjs/
 ^rhodecode\.db$
 ^rhodecode\.log$
 ^rhodecode_dev\.log$
 ^test\.db$
-
+^venv/
 
 # ac-tests
 ^acceptance_tests/\.cache.*$
 ^acceptance_tests/externals
 ^acceptance_tests/ghostdriver.log$
 ^acceptance_tests/local(_.+)?\.ini$
 
 # docs
 ^docs/_build$
 ^docs/result$
 ^docs-internal/_build$
 
 # Cythonized things
 ^rhodecode/.*\.(c|so)$

Makefile

@@ -1,172 +1,158 b''
+.DEFAULT_GOAL := help
+
+# Pretty print values cf. https://misc.flogisoft.com/bash/tip_colors_and_formatting
+RESET := \033[0m # Reset all formatting
+GREEN := \033[0;32m # Resets before setting 16b colour (32 -- green)
+YELLOW := \033[0;33m
+ORANGE := \033[0;38;5;208m # Reset then set 256b colour (208 -- orange)
+PEACH := \033[0;38;5;216m
+
+
+## ---------------------------------------------------------------------------------- ##
+## ------------------------- Help usage builder ------------------------------------- ##
+## ---------------------------------------------------------------------------------- ##
+# use '# >>> Build commands' to create section
+# use '# target: target description' to create help for target
+.PHONY: help
+help:
+	@echo "Usage:"
+	@cat $(MAKEFILE_LIST) | grep -E '^# >>>|^# [A-Za-z0-9_.-]+:' | sed -E 's/^# //' | awk ' \
+		BEGIN { \
+			green="\033[32m"; \
+			yellow="\033[33m"; \
+			reset="\033[0m"; \
+			section=""; \
+		} \
+		/^>>>/ { \
+			section=substr($$0, 5); \
+			printf "\n" green ">>> %s" reset "\n", section; \
+			next; \
+		} \
+		/^([A-Za-z0-9_.-]+):/ { \
+			target=$$1; \
+			gsub(/:$$/, "", target); \
+			description=substr($$0, index($$0, ":") + 2); \
+			if (description == "") { description="-"; } \
+			printf " - " yellow "%-35s" reset " %s\n", target, description; \
+		} \
+	'
+
 # required for pushd to work..
 SHELL = /bin/bash
 
-
-# set by: PATH_TO_OUTDATED_PACKAGES=/some/path/outdated_packages.py
-OUTDATED_PACKAGES = ${PATH_TO_OUTDATED_PACKAGES}
+# >>> Tests commands
 
 .PHONY: clean
-## Cleanup compiled and cache py files
+# clean: Cleanup compiled and cache py files
 clean:
 	make test-clean
 	find . -type f \( -iname '*.c' -o -iname '*.pyc' -o -iname '*.so' -o -iname '*.orig' \) -exec rm '{}' ';'
 	find . -type d -name "build" -prune -exec rm -rf '{}' ';'
 
 
 .PHONY: test
-## run test-clean and tests
+# test: run test-clean and tests
 test:
 	make test-clean
-	make test-only
+	unset RC_SQLALCHEMY_DB1_URL && unset RC_DB_URL && make test-only
 
 
 .PHONY: test-clean
-## run test-clean and tests
+# test-clean: run test-clean and tests
 test-clean:
 	rm -rf coverage.xml htmlcov junit.xml pylint.log result
 	find . -type d -name "__pycache__" -prune -exec rm -rf '{}' ';'
 	find . -type f \( -iname '.coverage.*' \) -exec rm '{}' ';'
 
 
 .PHONY: test-only
-## Run tests only without cleanup
+# test-only: Run tests only without cleanup
 test-only:
 	PYTHONHASHSEED=random \
 	py.test -x -vv -r xw -p no:sugar \
 	--cov-report=term-missing --cov-report=html \
 	--cov=rhodecode rhodecode
 
+# >>> Docs commands
 
 .PHONY: docs
-## build docs
+# docs: build docs
 docs:
 	(cd docs; docker run --rm -v $(PWD):/project --workdir=/project/docs sphinx-doc-build-rc make clean html SPHINXOPTS="-W")
 
 
 .PHONY: docs-clean
-## Cleanup docs
+# docs-clean: Cleanup docs
 docs-clean:
 	(cd docs; docker run --rm -v $(PWD):/project --workdir=/project/docs sphinx-doc-build-rc make clean)
 
 
 .PHONY: docs-cleanup
-## Cleanup docs
+# docs-cleanup: Cleanup docs
 docs-cleanup:
 	(cd docs; docker run --rm -v $(PWD):/project --workdir=/project/docs sphinx-doc-build-rc make cleanup)
 
+# >>> Dev commands
 
 .PHONY: web-build
-## Build JS packages static/js
+# web-build: Build JS packages static/js
 web-build:
 	rm -rf node_modules
 	docker run -it --rm -v $(PWD):/project --workdir=/project rhodecode/static-files-build:16 -c "npm install && /project/node_modules/.bin/grunt"
 	# run static file check
 	./rhodecode/tests/scripts/static-file-check.sh rhodecode/public/
 	rm -rf node_modules
 
-.PHONY: ruff-check
-## run a ruff analysis
-ruff-check:
-	ruff check --ignore F401 --ignore I001 --ignore E402 --ignore E501 --ignore F841 --exclude rhodecode/lib/dbmigrate --exclude .eggs  --exclude .dev .
-
-.PHONY: pip-packages
-## Show outdated packages
-pip-packages:
-	python ${OUTDATED_PACKAGES}
-
-
-.PHONY: build
-## Build sdist/egg
-build:
-	python -m build
-
 
 .PHONY: dev-sh
-## make dev-sh
+# dev-sh: make dev-sh
 dev-sh:
 	sudo echo "deb [trusted=yes] https://apt.fury.io/rsteube/ /" | sudo tee -a "/etc/apt/sources.list.d/fury.list"
 	sudo apt-get update
 	sudo apt-get install -y zsh carapace-bin
 	rm -rf /home/rhodecode/.oh-my-zsh
 	curl https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh | sh
 	@echo "source <(carapace _carapace)" > /home/rhodecode/.zsrc
 	@echo "${RC_DEV_CMD_HELP}"
 	@PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# ' zsh
 
 
 .PHONY: dev-cleanup
-## Cleanup: pip freeze | grep -v "^-e" | grep -v "@" | xargs pip uninstall -y
+# dev-cleanup: Cleanup: pip freeze | grep -v "^-e" | grep -v "@" | xargs pip uninstall -y
 dev-cleanup:
 	pip freeze | grep -v "^-e" | grep -v "@" | xargs pip uninstall -y
 	rm -rf /tmp/*
 
 
 .PHONY: dev-env
-## make dev-env based on the requirements files and install develop of packages
+# dev-env: make dev-env based on the requirements files and install develop of packages
 ## Cleanup: pip freeze | grep -v "^-e" | grep -v "@" | xargs pip uninstall -y
 dev-env:
 	sudo -u root chown rhodecode:rhodecode /home/rhodecode/.cache/pip/
 	pip install build virtualenv
 	pushd ../rhodecode-vcsserver/ && make dev-env && popd
 	pip wheel --wheel-dir=/home/rhodecode/.cache/pip/wheels -r requirements.txt -r requirements_rc_tools.txt -r requirements_test.txt -r requirements_debug.txt
 	pip install --no-index --find-links=/home/rhodecode/.cache/pip/wheels -r requirements.txt -r requirements_rc_tools.txt -r requirements_test.txt -r requirements_debug.txt
 	pip install -e .
 
 
 .PHONY: sh
-## shortcut for make dev-sh dev-env
+# sh: shortcut for make dev-sh dev-env
 sh:
 	make dev-env
 	make dev-sh
 
 
 ## Allows changes of workers e.g make dev-srv-g workers=2
 workers?=1
 
 .PHONY: dev-srv
-## run gunicorn web server with reloader, use workers=N to set multiworker mode
+# dev-srv: run gunicorn web server with reloader, use workers=N to set multiworker mode, workers=N allows changes of workers
 dev-srv:
 	gunicorn --paste=.dev/dev.ini --bind=0.0.0.0:10020 --config=.dev/gunicorn_config.py --timeout=120 --reload --workers=$(workers)
 
-
-# Default command on calling make
-.DEFAULT_GOAL := show-help
+.PHONY: ruff-check
+# ruff-check: run a ruff analysis
+ruff-check:
+	ruff check --ignore F401 --ignore I001 --ignore E402 --ignore E501 --ignore F841 --exclude rhodecode/lib/dbmigrate --exclude .eggs  --exclude .dev .
 
-.PHONY: show-help
-show-help:
-	@echo "$$(tput bold)Available rules:$$(tput sgr0)"
-	@echo
-	@sed -n -e "/^## / { \
-		h; \
-		s/.*//; \
-		:doc" \
-		-e "H; \
-		n; \
-		s/^## //; \
-		t doc" \
-		-e "s/:.*//; \
-		G; \
-		s/\\n## /---/; \
-		s/\\n/ /g; \
-		p; \
-	}" ${MAKEFILE_LIST} \
-	| LC_ALL='C' sort --ignore-case \
-	| awk -F '---' \
-		-v ncol=$$(tput cols) \
-		-v indent=19 \
-		-v col_on="$$(tput setaf 6)" \
-		-v col_off="$$(tput sgr0)" \
-	'{ \
-		printf "%s%*s%s ", col_on, -indent, $$1, col_off; \
-		n = split($$2, words, " "); \
-		line_length = ncol - indent; \
-		for (i = 1; i <= n; i++) { \
-			line_length -= length(words[i]) + 1; \
-			if (line_length <= 0) { \
-				line_length = ncol - indent - length(words[i]) - 1; \
-				printf "\n%*s ", -indent, " "; \
-			} \
-			printf "%s ", words[i]; \
-		} \
-		printf "\n"; \
-	}'

configs/development.ini

@@ -1,856 +1,912 b''
 
 ; #########################################
 ; RHODECODE COMMUNITY EDITION CONFIGURATION
 ; #########################################
 
 [DEFAULT]
 ; Debug flag sets all loggers to debug, and enables request tracking
 debug = true
 
 ; ########################################################################
 ; EMAIL CONFIGURATION
 ; These settings will be used by the RhodeCode mailing system
 ; ########################################################################
 
 ; prefix all emails subjects with given prefix, helps filtering out emails
 #email_prefix = [RhodeCode]
 
 ; email FROM address all mails will be sent
 #app_email_from = rhodecode-noreply@localhost
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 
 [server:main]
 ; COMMON HOST/IP CONFIG, This applies mostly to develop setup,
 ; Host port for gunicorn are controlled by gunicorn_conf.py
 host = 127.0.0.1
 port = 10020
 
 
 ; ###########################
 ; GUNICORN APPLICATION SERVER
 ; ###########################
 
 ; run with gunicorn  --config gunicorn_conf.py --paste rhodecode.ini
 
 ; Module to use, this setting shouldn't be changed
 use = egg:gunicorn#main
 
 ; Prefix middleware for RhodeCode.
 ; recommended when using proxy setup.
 ; allows to set RhodeCode under a prefix in server.
 ; eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
 ; And set your prefix like: `prefix = /custom_prefix`
 ; be sure to also set beaker.session.cookie_path = /custom_prefix if you need
 ; to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
 
 [app:main]
 ; The %(here)s variable will be replaced with the absolute path of parent directory
 ; of this file
 ; Each option in the app:main can be override by an environmental variable
 ;
 ;To override an option:
 ;
 ;RC_<KeyName>
 ;Everything should be uppercase, . and - should be replaced by _.
 ;For example, if you have these configuration settings:
 ;rc_cache.repo_object.backend = foo
 ;can be overridden by
 ;export RC_CACHE_REPO_OBJECT_BACKEND=foo
 
 use = egg:rhodecode-enterprise-ce
 
 ; enable proxy prefix middleware, defined above
 #filter-with = proxy-prefix
 
 ; #############
 ; DEBUG OPTIONS
 ; #############
 
 pyramid.reload_templates = true
 
 # During development the we want to have the debug toolbar enabled
 pyramid.includes =
     pyramid_debugtoolbar
 
 debugtoolbar.hosts = 0.0.0.0/0
 debugtoolbar.exclude_prefixes =
     /css
     /fonts
     /images
     /js
 
 ## RHODECODE PLUGINS ##
 rhodecode.includes =
     rhodecode.api
 
 
 # api prefix url
 rhodecode.api.url = /_admin/api
 
 ; enable debug style page
 debug_style = true
 
 ; #################
 ; END DEBUG OPTIONS
 ; #################
 
 ; encryption key used to encrypt social plugin tokens,
 ; remote_urls with credentials etc, if not set it defaults to
 ; `beaker.session.secret`
 #rhodecode.encrypted_values.secret =
 
 ; decryption strict mode (enabled by default). It controls if decryption raises
 ; `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
 ; Pick algorithm for encryption. Either fernet (more secure) or aes (default)
 ; fernet is safer, and we strongly recommend switching to it.
 ; Due to backward compatibility aes is used as default.
 #rhodecode.encrypted_values.algorithm = fernet
 
 ; Return gzipped responses from RhodeCode (static files/application)
 gzip_responses = false
 
 ; Auto-generate javascript routes file on startup
 generate_js_files = false
 
 ; System global default language.
 ; All available languages: en (default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ; Perform a full repository scan and import on each server start.
 ; Settings this to true could lead to very long startup time.
 startup.import_repos = false
 
 ; URL at which the application is running. This is used for Bootstrapping
 ; requests in context when no web request is available. Used in ishell, or
 ; SSH calls. Set this for events to receive proper url for SSH calls.
 app.base_url = http://rhodecode.local
 
 ; Host at which the Service API is running.
 app.service_api.host = http://rhodecode.local:10020
 
 ; Secret for Service API authentication.
 app.service_api.token =
 
 ; Unique application ID. Should be a random unique string for security.
 app_instance_uuid = rc-production
 
 ; Cut off limit for large diffs (size in bytes). If overall diff size on
 ; commit, or pull request exceeds this limit this diff will be displayed
 ; partially. E.g 512000 == 512Kb
 cut_off_limit_diff = 512000
 
 ; Cut off limit for large files inside diffs (size in bytes). Each individual
 ; file inside diff which exceeds this limit will be displayed partially.
 ;  E.g 128000 == 128Kb
 cut_off_limit_file = 128000
 
 ; Use cached version of vcs repositories everywhere. Recommended to be `true`
 vcs_full_cache = true
 
 ; Force https in RhodeCode, fixes https redirects, assumes it's always https.
 ; Normally this is controlled by proper flags sent from http server such as Nginx or Apache
 force_https = false
 
 ; use Strict-Transport-Security headers
 use_htsts = false
 
 ; Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ; RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ; gist URL alias, used to create nicer urls for gist. This should be an
 ; url that does rewrites to _admin/gists/{gistid}.
 ; example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ; RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ; List of views (using glob pattern syntax) that AUTH TOKENS could be
 ; used for access.
 ; Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ; came from the the logged in user who own this authentication token.
 ; Additionally @TOKEN syntax can be used to bound the view to specific
 ; authentication token. Such view would be only accessible when used together
 ; with this authentication token
 ; list of all views can be found under `/_admin/permissions/auth_token_access`
 ; The list should be "," separated and on a single line.
 ; Most common views to enable:
 
 #    RepoCommitsView:repo_commit_download
 #    RepoCommitsView:repo_commit_patch
 #    RepoCommitsView:repo_commit_raw
 #    RepoCommitsView:repo_commit_raw@TOKEN
 #    RepoFilesView:repo_files_diff
 #    RepoFilesView:repo_archivefile
 #    RepoFilesView:repo_file_raw
 #    GistView:*
 api_access_controllers_whitelist =
 
 ; Default encoding used to convert from and to unicode
 ; can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ; instance-id prefix
 ; a prefix key for this instance used for cache invalidation when running
 ; multiple instances of RhodeCode, make sure it's globally unique for
 ; all running RhodeCode instances. Leave empty if you don't use it
 instance_id =
 
 ; Fallback authentication plugin. Set this to a plugin ID to force the usage
 ; of an authentication plugin also if it is disabled by it's settings.
 ; This could be useful if you are unable to log in to the system due to broken
 ; authentication settings. Then you can enable e.g. the internal RhodeCode auth
 ; module to log in again and fix the settings.
 ; Available builtin plugin IDs (hash is part of the ID):
 ; egg:rhodecode-enterprise-ce#rhodecode
 ; egg:rhodecode-enterprise-ce#pam
 ; egg:rhodecode-enterprise-ce#ldap
 ; egg:rhodecode-enterprise-ce#jasig_cas
 ; egg:rhodecode-enterprise-ce#headers
 ; egg:rhodecode-enterprise-ce#crowd
 
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ; Flag to control loading of legacy plugins in py:/path format
 auth_plugin.import_legacy_plugins = true
 
 ; alternative return HTTP header for failed authentication. Default HTTP
 ; response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ; handling that causing a series of failed authentication calls.
 ; Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ; This will be served instead of default 401 on bad authentication
 auth_ret_code =
 
 ; use special detection method when serving auth_ret_code, instead of serving
 ; ret_code directly, use 401 initially (Which triggers credentials prompt)
 ; and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ; locking return code. When repository is locked return this HTTP code. 2XX
 ; codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ; Filesystem location were repositories should be stored
 repo_store.path = /var/opt/rhodecode_repo_store
 
 ; allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ; Generated license token required for EE edition license.
 ; New generated token value can be found in Admin > settings > license page.
 license_token =
 
 ; This flag hides sensitive information on the license page such as token, and license data
 license.hide_license_info = false
 
+; Import EE license from this license path
+#license.import_path = %(here)s/rhodecode_enterprise.license
+
+; import license 'if-missing' or 'force' (always override)
+; if-missing means apply license if it doesn't exist. 'force' option always overrides it
+license.import_path_mode = if-missing
+
 ; supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 
 ; supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = dev
 
 ; Display extended labs settings
 labs_settings_active = true
 
 ; Custom exception store path, defaults to TMPDIR
 ; This is used to store exception from RhodeCode in shared directory
 #exception_tracker.store_path =
 
 ; Send email with exception details when it happens
 #exception_tracker.send_email = false
 
 ; Comma separated list of recipients for exception emails,
 ; e.g [e-mail],[e-mail]
 ; Can be left empty, then emails will be sent to ALL super-admins
 #exception_tracker.send_email_recipients =
 
 ; optional prefix to Add to email Subject
 #exception_tracker.email_prefix = [RHODECODE ERROR]
 
-; File store configuration. This is used to store and serve uploaded files
-file_store.enabled = true
+; NOTE: this setting IS DEPRECATED:
+; file_store backend is always enabled
+#file_store.enabled = true
 
+; NOTE: this setting IS DEPRECATED:
+; file_store.backend = X -> use `file_store.backend.type = filesystem_v2` instead
 ; Storage backend, available options are: local
-file_store.backend = local
+#file_store.backend = local
 
+; NOTE: this setting IS DEPRECATED:
+; file_store.storage_path = X -> use `file_store.filesystem_v2.storage_path = X` instead
 ; path to store the uploaded binaries and artifacts
-file_store.storage_path = /var/opt/rhodecode_data/file_store
+#file_store.storage_path = /var/opt/rhodecode_data/file_store
+
+; Artifacts file-store, is used to store comment attachments and artifacts uploads.
+; file_store backend type: filesystem_v1, filesystem_v2 or objectstore (s3-based) are available as options
+; filesystem_v1 is backwards compat with pre 5.1 storage changes
+; new installations should choose filesystem_v2 or objectstore (s3-based), pick filesystem when migrating from
+; previous installations to keep the artifacts without a need of migration
+#file_store.backend.type = filesystem_v2
+
+; filesystem options...
+#file_store.filesystem_v1.storage_path = /var/opt/rhodecode_data/artifacts_file_store
+
+; filesystem_v2 options...
+#file_store.filesystem_v2.storage_path = /var/opt/rhodecode_data/artifacts_file_store
+#file_store.filesystem_v2.shards = 8
 
+; objectstore options...
+; url for s3 compatible storage that allows to upload artifacts
+; e.g http://minio:9000
+#file_store.backend.type = objectstore
+#file_store.objectstore.url = http://s3-minio:9000
+
+; a top-level bucket to put all other shards in
+; objects will be stored in rhodecode-file-store/shard-N based on the bucket_shards number
+#file_store.objectstore.bucket = rhodecode-file-store
+
+; number of sharded buckets to create to distribute archives across
+; default is 8 shards
+#file_store.objectstore.bucket_shards = 8
+
+; key for s3 auth
+#file_store.objectstore.key = s3admin
+
+; secret for s3 auth
+#file_store.objectstore.secret = s3secret4
+
+;region for s3 storage
+#file_store.objectstore.region = eu-central-1
 
 ; Redis url to acquire/check generation of archives locks
 archive_cache.locking.url = redis://redis:6379/1
 
 ; Storage backend, only 'filesystem' and 'objectstore' are available now
 archive_cache.backend.type = filesystem
 
 ; url for s3 compatible storage that allows to upload artifacts
 ; e.g http://minio:9000
 archive_cache.objectstore.url = http://s3-minio:9000
 
 ; key for s3 auth
 archive_cache.objectstore.key = key
 
 ; secret for s3 auth
 archive_cache.objectstore.secret = secret
 
 ;region for s3 storage
 archive_cache.objectstore.region = eu-central-1
 
 ; number of sharded buckets to create to distribute archives across
 ; default is 8 shards
 archive_cache.objectstore.bucket_shards = 8
 
 ; a top-level bucket to put all other shards in
 ; objects will be stored in rhodecode-archive-cache/shard-N based on the bucket_shards number
 archive_cache.objectstore.bucket = rhodecode-archive-cache
 
 ; if true, this cache will try to retry with retry_attempts=N times waiting retry_backoff time
 archive_cache.objectstore.retry = false
 
 ; number of seconds to wait for next try using retry
 archive_cache.objectstore.retry_backoff = 1
 
 ; how many tries do do a retry fetch from this backend
 archive_cache.objectstore.retry_attempts = 10
 
 ; Default is $cache_dir/archive_cache if not set
 ; Generated repo archives will be cached at this location
 ; and served from the cache during subsequent requests for the same archive of
 ; the repository. This path is important to be shared across filesystems and with
 ; RhodeCode and vcsserver
 archive_cache.filesystem.store_dir = /var/opt/rhodecode_data/archive_cache
 
 ; The limit in GB sets how much data we cache before recycling last used, defaults to 10 gb
 archive_cache.filesystem.cache_size_gb = 1
 
 ; Eviction policy used to clear out after cache_size_gb limit is reached
 archive_cache.filesystem.eviction_policy = least-recently-stored
 
 ; By default cache uses sharding technique, this specifies how many shards are there
 ; default is 8 shards
 archive_cache.filesystem.cache_shards = 8
 
 ; if true, this cache will try to retry with retry_attempts=N times waiting retry_backoff time
 archive_cache.filesystem.retry = false
 
 ; number of seconds to wait for next try using retry
 archive_cache.filesystem.retry_backoff = 1
 
 ; how many tries do do a retry fetch from this backend
 archive_cache.filesystem.retry_attempts = 10
 
 
 ; #############
 ; CELERY CONFIG
 ; #############
 
 ; manually run celery: /path/to/celery worker --task-events --beat --app rhodecode.lib.celerylib.loader --scheduler rhodecode.lib.celerylib.scheduler.RcScheduler --loglevel DEBUG --ini /path/to/rhodecode.ini
 
 use_celery = true
 
 ; path to store schedule database
 #celerybeat-schedule.path =
 
 ; connection url to the message broker (default redis)
 celery.broker_url = redis://redis:6379/8
 
 ; results backend to get results for (default redis)
 celery.result_backend = redis://redis:6379/8
 
 ; rabbitmq example
 #celery.broker_url = amqp://rabbitmq:qweqwe@localhost:5672/rabbitmqhost
 
 ; maximum tasks to execute before worker restart
 celery.max_tasks_per_child = 20
 
 ; tasks will never be sent to the queue, but executed locally instead.
 celery.task_always_eager = false
 
 ; #############
 ; DOGPILE CACHE
 ; #############
 
 ; Default cache dir for caches. Putting this into a ramdisk can boost performance.
 ; eg. /tmpfs/data_ramdisk, however this directory might require large amount of space
 cache_dir = /var/opt/rhodecode_data
 
 ; *********************************************
 ; `sql_cache_short` cache for heavy SQL queries
 ; Only supported backend is `memory_lru`
 ; *********************************************
 rc_cache.sql_cache_short.backend = dogpile.cache.rc.memory_lru
 rc_cache.sql_cache_short.expiration_time = 30
 
 
 ; *****************************************************
 ; `cache_repo_longterm` cache for repo object instances
 ; Only supported backend is `memory_lru`
 ; *****************************************************
 rc_cache.cache_repo_longterm.backend = dogpile.cache.rc.memory_lru
 ; by default we use 30 Days, cache is still invalidated on push
 rc_cache.cache_repo_longterm.expiration_time = 2592000
 ; max items in LRU cache, set to smaller number to save memory, and expire last used caches
 rc_cache.cache_repo_longterm.max_size = 10000
 
 
 ; *********************************************
 ; `cache_general` cache for general purpose use
 ; for simplicity use rc.file_namespace backend,
 ; for performance and scale use rc.redis
 ; *********************************************
 rc_cache.cache_general.backend = dogpile.cache.rc.file_namespace
 rc_cache.cache_general.expiration_time = 43200
 ; file cache store path. Defaults to `cache_dir =` value or tempdir if both values are not set
 #rc_cache.cache_general.arguments.filename = /tmp/cache_general_db
 
 ; alternative `cache_general` redis backend with distributed lock
 #rc_cache.cache_general.backend = dogpile.cache.rc.redis
 #rc_cache.cache_general.expiration_time = 300
 
 ; redis_expiration_time needs to be greater then expiration_time
 #rc_cache.cache_general.arguments.redis_expiration_time = 7200
 
 #rc_cache.cache_general.arguments.host = localhost
 #rc_cache.cache_general.arguments.port = 6379
 #rc_cache.cache_general.arguments.db = 0
 #rc_cache.cache_general.arguments.socket_timeout = 30
 ; more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_general.arguments.distributed_lock = true
 
 ; auto-renew lock to prevent stale locks, slower but safer. Use only if problems happen
 #rc_cache.cache_general.arguments.lock_auto_renewal = true
 
 ; *************************************************
 ; `cache_perms` cache for permission tree, auth TTL
 ; for simplicity use rc.file_namespace backend,
 ; for performance and scale use rc.redis
 ; *************************************************
 rc_cache.cache_perms.backend = dogpile.cache.rc.file_namespace
 rc_cache.cache_perms.expiration_time = 3600
 ; file cache store path. Defaults to `cache_dir =` value or tempdir if both values are not set
 #rc_cache.cache_perms.arguments.filename = /tmp/cache_perms_db
 
 ; alternative `cache_perms` redis backend with distributed lock
 #rc_cache.cache_perms.backend = dogpile.cache.rc.redis
 #rc_cache.cache_perms.expiration_time = 300
 
 ; redis_expiration_time needs to be greater then expiration_time
 #rc_cache.cache_perms.arguments.redis_expiration_time = 7200
 
 #rc_cache.cache_perms.arguments.host = localhost
 #rc_cache.cache_perms.arguments.port = 6379
 #rc_cache.cache_perms.arguments.db = 0
 #rc_cache.cache_perms.arguments.socket_timeout = 30
 ; more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_perms.arguments.distributed_lock = true
 
 ; auto-renew lock to prevent stale locks, slower but safer. Use only if problems happen
 #rc_cache.cache_perms.arguments.lock_auto_renewal = true
 
 ; ***************************************************
 ; `cache_repo` cache for file tree, Readme, RSS FEEDS
 ; for simplicity use rc.file_namespace backend,
 ; for performance and scale use rc.redis
 ; ***************************************************
 rc_cache.cache_repo.backend = dogpile.cache.rc.file_namespace
 rc_cache.cache_repo.expiration_time = 2592000
 ; file cache store path. Defaults to `cache_dir =` value or tempdir if both values are not set
 #rc_cache.cache_repo.arguments.filename = /tmp/cache_repo_db
 
 ; alternative `cache_repo` redis backend with distributed lock
 #rc_cache.cache_repo.backend = dogpile.cache.rc.redis
 #rc_cache.cache_repo.expiration_time = 2592000
 
 ; redis_expiration_time needs to be greater then expiration_time
 #rc_cache.cache_repo.arguments.redis_expiration_time = 2678400
 
 #rc_cache.cache_repo.arguments.host = localhost
 #rc_cache.cache_repo.arguments.port = 6379
 #rc_cache.cache_repo.arguments.db = 1
 #rc_cache.cache_repo.arguments.socket_timeout = 30
 ; more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_repo.arguments.distributed_lock = true
 
 ; auto-renew lock to prevent stale locks, slower but safer. Use only if problems happen
 #rc_cache.cache_repo.arguments.lock_auto_renewal = true
 
 ; ##############
 ; BEAKER SESSION
 ; ##############
 
 ; beaker.session.type is type of storage options for the logged users sessions. Current allowed
 ; types are file, ext:redis, ext:database, ext:memcached
 ; Fastest ones are ext:redis and ext:database, DO NOT use memory type for session
 #beaker.session.type = file
 #beaker.session.data_dir = %(here)s/data/sessions
 
 ; Redis based sessions
 beaker.session.type = ext:redis
 beaker.session.url = redis://redis:6379/2
 
 ; DB based session, fast, and allows easy management over logged in users
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = develop-rc-uytcxaz
 beaker.session.lock_dir = /data_ramdisk/lock
 
 ; Secure encrypted cookie. Requires AES and AES python libraries
 ; you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = key_for_encryption
 #beaker.session.validate_key = validation_key
 
 ; Sets session as invalid (also logging out user) if it haven not been
 ; accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 
 ; Path to use for the cookie. Set to prefix if you use prefix middleware
 #beaker.session.cookie_path = /custom_prefix
 
 ; Set https secure cookie
 beaker.session.secure = false
 
 ; default cookie expiration time in seconds, set to `true` to set expire
 ; at browser close
 #beaker.session.cookie_expires = 3600
 
 ; #############################
 ; SEARCH INDEXING CONFIGURATION
 ; #############################
 
 ; Full text search indexer is available in rhodecode-tools under
 ; `rhodecode-tools index` command
 
 ; WHOOSH Backend, doesn't require additional services to run
 ; it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
 ; ####################
 ; CHANNELSTREAM CONFIG
 ; ####################
 
 ; channelstream enables persistent connections and live notification
 ; in the system. It's also used by the chat system
 
 channelstream.enabled = true
 
 ; server address for channelstream server on the backend
 channelstream.server = channelstream:9800
 
 ; location of the channelstream server from outside world
 ; use ws:// for http or wss:// for https. This address needs to be handled
 ; by external HTTP server such as Nginx or Apache
 ; see Nginx/Apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = ENV_GENERATED
 channelstream.history.location = /var/opt/rhodecode_data/channelstream_history
 
 ; Internal application path that Javascript uses to connect into.
 ; If you use proxy-prefix the prefix should be added before /_channelstream
 channelstream.proxy_path = /_channelstream
 
 
 ; ##############################
 ; MAIN RHODECODE DATABASE CONFIG
 ; ##############################
 
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode?charset=utf8
 ; pymysql is an alternative driver for MySQL, use in case of problems with default one
 #sqlalchemy.db1.url = mysql+pymysql://root:qweqwe@localhost/rhodecode
 
 sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 
 ; see sqlalchemy docs for other advanced settings
 ; print the sql statements to output
 sqlalchemy.db1.echo = false
 
 ; recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 
 ; the number of connections to keep open inside the connection pool.
 ; 0 indicates no limit
 ; the general calculus with gevent is:
 ; if your system allows 500 concurrent greenlets (max_connections) that all do database access,
 ; then increase pool size + max overflow so that they add up to 500.
 #sqlalchemy.db1.pool_size = 5
 
 ; The number of connections to allow in connection pool "overflow", that is
 ; connections that can be opened above and beyond the pool_size setting,
 ; which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 ; Connection check ping, used to detect broken database connections
 ; could be enabled to better handle cases if MySQL has gone away errors
 #sqlalchemy.db1.ping_connection = true
 
 ; ##########
 ; VCS CONFIG
 ; ##########
 vcs.server.enable = true
 vcs.server = vcsserver:10010
 
 ; Web server connectivity protocol, responsible for web based VCS operations
 ; Available protocols are:
 ; `http` - use http-rpc backend (default)
 vcs.server.protocol = http
 
 ; Push/Pull operations protocol, available options are:
 ; `http` - use http-rpc backend (default)
 vcs.scm_app_implementation = http
 
 ; Push/Pull operations hooks protocol, available options are:
 ; `http` - use http-rpc backend (default)
 ; `celery` - use celery based hooks
-vcs.hooks.protocol = http
+#DEPRECATED:vcs.hooks.protocol = http
+vcs.hooks.protocol.v2 = celery
 
 ; Host on which this instance is listening for hooks. vcsserver will call this host to pull/push hooks so it should be
 ; accessible via network.
 ; Use vcs.hooks.host = "*" to bind to current hostname (for Docker)
 vcs.hooks.host = *
 
 ; Start VCSServer with this instance as a subprocess, useful for development
 vcs.start_server = false
 
 ; List of enabled VCS backends, available options are:
 ; `hg`  - mercurial
 ; `git` - git
 ; `svn` - subversion
 vcs.backends = hg, git, svn
 
 ; Wait this number of seconds before killing connection to the vcsserver
 vcs.connection_timeout = 3600
 
 ; Cache flag to cache vcsserver remote calls locally
 ; It uses cache_region `cache_repo`
 vcs.methods.cache = true
 
+; Filesystem location where Git lfs objects should be stored
+vcs.git.lfs.storage_location = /var/opt/rhodecode_repo_store/.cache/git_lfs_store
+
+; Filesystem location where Mercurial largefile objects should be stored
+vcs.hg.largefiles.storage_location = /var/opt/rhodecode_repo_store/.cache/hg_largefiles_store
+
 ; ####################################################
 ; Subversion proxy support (mod_dav_svn)
 ; Maps RhodeCode repo groups into SVN paths for Apache
 ; ####################################################
 
 ; Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ; Set a numeric version for your current SVN e.g 1.8, or 1.12
 ; Legacy available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible, pre-1.9-compatible
 #vcs.svn.compatible_version = 1.8
 
 ; Redis connection settings for svn integrations logic
 ; This connection string needs to be the same on ce and vcsserver
 vcs.svn.redis_conn = redis://redis:6379/0
 
 ; Enable SVN proxy of requests over HTTP
 vcs.svn.proxy.enabled = true
 
 ; host to connect to running SVN subsystem
 vcs.svn.proxy.host = http://svn:8090
 
 ; Enable or disable the config file generation.
 svn.proxy.generate_config = true
 
 ; Generate config file with `SVNListParentPath` set to `On`.
 svn.proxy.list_parent_path = true
 
 ; Set location and file name of generated config file.
 svn.proxy.config_file_path = /etc/rhodecode/conf/svn/mod_dav_svn.conf
 
 ; alternative mod_dav config template. This needs to be a valid mako template
 ; Example template can be found in the source code:
 ; rhodecode/apps/svn_support/templates/mod-dav-svn.conf.mako
 #svn.proxy.config_template = ~/.rccontrol/enterprise-1/custom_svn_conf.mako
 
 ; Used as a prefix to the `Location` block in the generated config file.
 ; In most cases it should be set to `/`.
 svn.proxy.location_root = /
 
 ; Command to reload the mod dav svn configuration on change.
 ; Example: `/etc/init.d/apache2 reload` or /home/USER/apache_reload.sh
 ; Make sure user who runs RhodeCode process is allowed to reload Apache
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 
 ; If the timeout expires before the reload command finishes, the command will
 ; be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
 #svn.proxy.reload_timeout = 10
 
 ; ####################
 ; SSH Support Settings
 ; ####################
 
 ; Defines if a custom authorized_keys file should be created and written on
 ; any change user ssh keys. Setting this to false also disables possibility
 ; of adding SSH keys by users from web interface. Super admins can still
 ; manage SSH Keys.
 ssh.generate_authorized_keyfile = true
 
 ; Options for ssh, default is `no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding`
 # ssh.authorized_keys_ssh_opts =
 
 ; Path to the authorized_keys file where the generate entries are placed.
 ; It is possible to have multiple key files specified in `sshd_config` e.g.
 ; AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/authorized_keys_rhodecode
 ssh.authorized_keys_file_path = /etc/rhodecode/conf/ssh/authorized_keys_rhodecode
 
 ; Command to execute the SSH wrapper. The binary is available in the
 ; RhodeCode installation directory.
 ; legacy: /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper
 ; new rewrite: /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper-v2
-ssh.wrapper_cmd = /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper
+#DEPRECATED: ssh.wrapper_cmd = /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper
+ssh.wrapper_cmd.v2 = /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper-v2
 
 ; Allow shell when executing the ssh-wrapper command
 ssh.wrapper_cmd_allow_shell = false
 
 ; Enables logging, and detailed output send back to the client during SSH
 ; operations. Useful for debugging, shouldn't be used in production.
 ssh.enable_debug_logging = true
 
 ; Paths to binary executable, by default they are the names, but we can
 ; override them if we want to use a custom one
 ssh.executable.hg = /usr/local/bin/rhodecode_bin/vcs_bin/hg
 ssh.executable.git = /usr/local/bin/rhodecode_bin/vcs_bin/git
 ssh.executable.svn = /usr/local/bin/rhodecode_bin/vcs_bin/svnserve
 
 ; Enables SSH key generator web interface. Disabling this still allows users
 ; to add their own keys.
 ssh.enable_ui_key_generator = true
 
 ; Statsd client config, this is used to send metrics to statsd
 ; We recommend setting statsd_exported and scrape them using Prometheus
 #statsd.enabled = false
 #statsd.statsd_host = 0.0.0.0
 #statsd.statsd_port = 8125
 #statsd.statsd_prefix =
 #statsd.statsd_ipv6 = false
 
 ; configure logging automatically at server startup set to false
 ; to use the below custom logging config.
 ; RC_LOGGING_FORMATTER
 ; RC_LOGGING_LEVEL
 ; env variables can control the settings for logging in case of autoconfigure
 
 #logging.autoconfigure = true
 
 ; specify your own custom logging config file to configure logging
 #logging.logging_conf_file = /path/to/custom_logging.ini
 
 ; Dummy marker to add new entries after.
 ; Add any custom entries below. Please don't remove this marker.
 custom.conf = 1
 
 
 ; #####################
 ; LOGGING CONFIGURATION
 ; #####################
 
 [loggers]
 keys = root, sqlalchemy, beaker, celery, rhodecode, ssh_wrapper
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, json, color_formatter, color_formatter_sql
 
 ; #######
 ; LOGGERS
 ; #######
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_ssh_wrapper]
 level = DEBUG
 handlers =
 qualname = ssh_wrapper
 propagate = 1
 
 [logger_celery]
 level = DEBUG
 handlers =
 qualname = celery
 
 
 ; ########
 ; HANDLERS
 ; ########
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr, )
 level = DEBUG
 ; To enable JSON formatted logs replace 'generic/color_formatter' with 'json'
 ; This allows sending properly formatted logs to grafana loki or elasticsearch
 formatter = color_formatter
 
 [handler_console_sql]
 ; "level = DEBUG" logs SQL queries and results.
 ; "level = INFO" logs SQL queries.
 ; "level = WARN" logs neither.  (Recommended for production systems.)
 class = StreamHandler
 args = (sys.stderr, )
 level = WARN
 ; To enable JSON formatted logs replace 'generic/color_formatter_sql' with 'json'
 ; This allows sending properly formatted logs to grafana loki or elasticsearch
 formatter = color_formatter_sql
 
 ; ##########
 ; FORMATTERS
 ; ##########
 
 [formatter_generic]
 class = rhodecode.lib.logging_formatter.ExceptionAwareFormatter
 format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
 format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_json]
 format = %(timestamp)s %(levelname)s %(name)s %(message)s %(req_id)s
 class = rhodecode.lib._vendor.jsonlogger.JsonFormatter

configs/gunicorn_config.py

@@ -1,520 +1,545 b''
 """
 Gunicorn config extension and hooks. This config file adds some extra settings and memory management.
 Gunicorn configuration should be managed by .ini files entries of RhodeCode or VCSServer
 """
 
 import gc
 import os
 import sys
 import math
 import time
 import threading
 import traceback
 import random
 import socket
 import dataclasses
+import json
 from gunicorn.glogging import Logger
 
 
 def get_workers():
     import multiprocessing
     return multiprocessing.cpu_count() * 2 + 1
 
 
 bind = "127.0.0.1:10020"
 
 
 # Error logging output for gunicorn (-) is stdout
 errorlog = '-'
 
 # Access logging output for gunicorn (-) is stdout
 accesslog = '-'
 
 
 # SERVER MECHANICS
 # None == system temp dir
 # worker_tmp_dir is recommended to be set to some tmpfs
 worker_tmp_dir = None
 tmp_upload_dir = None
 
-# use re-use port logic
-#reuse_port = True
+# use re-use port logic to let linux internals load-balance the requests better.
+reuse_port = True
 
 # Custom log format
 #access_log_format = (
 #    '%(t)s %(p)s INFO  [GNCRN] %(h)-15s rqt:%(L)s %(s)s %(b)-6s "%(m)s:%(U)s %(q)s" usr:%(u)s "%(f)s" "%(a)s"')
 
 # loki format for easier parsing in grafana
-access_log_format = (
+loki_access_log_format = (
     'time="%(t)s" pid=%(p)s level="INFO" type="[GNCRN]" ip="%(h)-15s" rqt="%(L)s" response_code="%(s)s" response_bytes="%(b)-6s" uri="%(m)s:%(U)s %(q)s" user=":%(u)s" user_agent="%(a)s"')
 
+# JSON format
+json_access_log_format = json.dumps({
+    'time': r'%(t)s',
+    'pid': r'%(p)s',
+    'level': 'INFO',
+    'ip': r'%(h)s',
+    'request_time': r'%(L)s',
+    'remote_address': r'%(h)s',
+    'user_name': r'%(u)s',
+    'status': r'%(s)s',
+    'method': r'%(m)s',
+    'url_path': r'%(U)s',
+    'query_string': r'%(q)s',
+    'protocol': r'%(H)s',
+    'response_length': r'%(B)s',
+    'referer': r'%(f)s',
+    'user_agent': r'%(a)s',
+
+})
+
+access_log_format = loki_access_log_format
+if os.environ.get('RC_LOGGING_FORMATTER') == 'json':
+    access_log_format = json_access_log_format
+
 # self adjust workers based on CPU count, to use maximum of CPU and not overquota the resources
 # workers = get_workers()
 
 # Gunicorn access log level
 loglevel = 'info'
 
 # Process name visible in a process list
 proc_name = 'rhodecode_enterprise'
 
 # Type of worker class, one of `sync`, `gevent` or `gthread`
 # currently `sync` is the only option allowed for vcsserver and for rhodecode all of 3 are allowed
 # gevent:
 # In this case, the maximum number of concurrent requests is (N workers * X worker_connections)
 # e.g. workers =3 worker_connections=10 = 3*10, 30 concurrent requests can be handled
 # gthread:
 # In this case, the maximum number of concurrent requests is (N workers * X threads)
 # e.g. workers = 3 threads=3 = 3*3, 9 concurrent requests can be handled
 worker_class = 'gthread'
 
 # Sets the number of process workers. More workers means more concurrent connections
 # RhodeCode can handle at the same time. Each additional worker also it increases
 # memory usage as each has its own set of caches.
 # The Recommended value is (2 * NUMBER_OF_CPUS + 1), eg 2CPU = 5 workers, but no more
 # than 8-10 unless for huge deployments .e.g 700-1000 users.
 # `instance_id = *` must be set in the [app:main] section below (which is the default)
 # when using more than 1 worker.
 workers = 2
 
 # Threads numbers for worker class gthread
 threads = 1
 
 # The maximum number of simultaneous clients. Valid only for gevent
 # In this case, the maximum number of concurrent requests is (N workers * X worker_connections)
 # e.g workers =3 worker_connections=10 = 3*10, 30 concurrent requests can be handled
 worker_connections = 10
 
 # Max number of requests that worker will handle before being gracefully restarted.
 # Prevents memory leaks, jitter adds variability so not all workers are restarted at once.
 max_requests = 2000
 max_requests_jitter = int(max_requests * 0.2)  # 20% of max_requests
 
 # The maximum number of pending connections.
 # Exceeding this number results in the client getting an error when attempting to connect.
 backlog = 64
 
 # The Amount of time a worker can spend with handling a request before it
 # gets killed and restarted. By default, set to 21600 (6hrs)
 # Examples: 1800 (30min), 3600 (1hr), 7200 (2hr), 43200 (12h)
 timeout = 21600
 
 # The maximum size of HTTP request line in bytes.
 # 0 for unlimited
 limit_request_line = 0
 
 # Limit the number of HTTP headers fields in a request.
 # By default this value is 100 and can't be larger than 32768.
 limit_request_fields = 32768
 
 # Limit the allowed size of an HTTP request header field.
 # Value is a positive number or 0.
 # Setting it to 0 will allow unlimited header field sizes.
 limit_request_field_size = 0
 
 # Timeout for graceful workers restart.
 # After receiving a restart signal, workers have this much time to finish
 # serving requests. Workers still alive after the timeout (starting from the
 # receipt of the restart signal) are force killed.
 # Examples: 1800 (30min), 3600 (1hr), 7200 (2hr), 43200 (12h)
 graceful_timeout = 21600
 
 # The number of seconds to wait for requests on a Keep-Alive connection.
 # Generally set in the 1-5 seconds range.
 keepalive = 2
 
 # Maximum memory usage that each worker can use before it will receive a
 # graceful restart signal 0 = memory monitoring is disabled
 # Examples: 268435456 (256MB), 536870912 (512MB)
 # 1073741824 (1GB), 2147483648 (2GB), 4294967296 (4GB)
 # Dynamic formula 1024 * 1024 * 256 == 256MBs
 memory_max_usage = 0
 
 # How often in seconds to check for memory usage for each gunicorn worker
 memory_usage_check_interval = 60
 
 # Threshold value for which we don't recycle worker if GarbageCollection
 # frees up enough resources. Before each restart, we try to run GC on worker
 # in case we get enough free memory after that; restart will not happen.
 memory_usage_recovery_threshold = 0.8
 
 
 @dataclasses.dataclass
 class MemoryCheckConfig:
     max_usage: int
     check_interval: int
     recovery_threshold: float
 
 
 def _get_process_rss(pid=None):
     try:
         import psutil
         if pid:
             proc = psutil.Process(pid)
         else:
             proc = psutil.Process()
         return proc.memory_info().rss
     except Exception:
         return None
 
 
 def _get_config(ini_path):
     import configparser
 
     try:
         config = configparser.RawConfigParser()
         config.read(ini_path)
         return config
     except Exception:
         return None
 
 
 def get_memory_usage_params(config=None):
     # memory spec defaults
     _memory_max_usage = memory_max_usage
     _memory_usage_check_interval = memory_usage_check_interval
     _memory_usage_recovery_threshold = memory_usage_recovery_threshold
 
     if config:
         ini_path = os.path.abspath(config)
         conf = _get_config(ini_path)
 
         section = 'server:main'
         if conf and conf.has_section(section):
 
             if conf.has_option(section, 'memory_max_usage'):
                 _memory_max_usage = conf.getint(section, 'memory_max_usage')
 
             if conf.has_option(section, 'memory_usage_check_interval'):
                 _memory_usage_check_interval = conf.getint(section, 'memory_usage_check_interval')
 
             if conf.has_option(section, 'memory_usage_recovery_threshold'):
                 _memory_usage_recovery_threshold = conf.getfloat(section, 'memory_usage_recovery_threshold')
 
     _memory_max_usage = int(os.environ.get('RC_GUNICORN_MEMORY_MAX_USAGE', '')
                             or _memory_max_usage)
     _memory_usage_check_interval = int(os.environ.get('RC_GUNICORN_MEMORY_USAGE_CHECK_INTERVAL', '')
                                        or _memory_usage_check_interval)
     _memory_usage_recovery_threshold = float(os.environ.get('RC_GUNICORN_MEMORY_USAGE_RECOVERY_THRESHOLD', '')
                                              or _memory_usage_recovery_threshold)
 
     return MemoryCheckConfig(_memory_max_usage, _memory_usage_check_interval, _memory_usage_recovery_threshold)
 
 
 def _time_with_offset(check_interval):
     return time.time() - random.randint(0, check_interval/2.0)
 
 
 def pre_fork(server, worker):
     pass
 
 
 def post_fork(server, worker):
 
     memory_conf = get_memory_usage_params()
     _memory_max_usage = memory_conf.max_usage
     _memory_usage_check_interval = memory_conf.check_interval
     _memory_usage_recovery_threshold = memory_conf.recovery_threshold
 
     worker._memory_max_usage = int(os.environ.get('RC_GUNICORN_MEMORY_MAX_USAGE', '')
                                    or _memory_max_usage)
     worker._memory_usage_check_interval = int(os.environ.get('RC_GUNICORN_MEMORY_USAGE_CHECK_INTERVAL', '')
                                               or _memory_usage_check_interval)
     worker._memory_usage_recovery_threshold = float(os.environ.get('RC_GUNICORN_MEMORY_USAGE_RECOVERY_THRESHOLD', '')
                                                     or _memory_usage_recovery_threshold)
 
     # register memory last check time, with some random offset so we don't recycle all
     # at once
     worker._last_memory_check_time = _time_with_offset(_memory_usage_check_interval)
 
     if _memory_max_usage:
         server.log.info("pid=[%-10s] WORKER spawned with max memory set at %s", worker.pid,
                         _format_data_size(_memory_max_usage))
     else:
         server.log.info("pid=[%-10s] WORKER spawned", worker.pid)
 
 
 def pre_exec(server):
     server.log.info("Forked child, re-executing.")
 
 
 def on_starting(server):
     server_lbl = '{} {}'.format(server.proc_name, server.address)
     server.log.info("Server %s is starting.", server_lbl)
     server.log.info('Config:')
     server.log.info(f"\n{server.cfg}")
     server.log.info(get_memory_usage_params())
 
 
 def when_ready(server):
     server.log.info("Server %s is ready. Spawning workers", server)
 
 
 def on_reload(server):
     pass
 
 
 def _format_data_size(size, unit="B", precision=1, binary=True):
     """Format a number using SI units (kilo, mega, etc.).
 
     ``size``: The number as a float or int.
 
     ``unit``: The unit name in plural form. Examples: "bytes", "B".
 
     ``precision``: How many digits to the right of the decimal point. Default
     is 1.  0 suppresses the decimal point.
 
     ``binary``: If false, use base-10 decimal prefixes (kilo = K = 1000).
     If true, use base-2 binary prefixes (kibi = Ki = 1024).
 
     ``full_name``: If false (default), use the prefix abbreviation ("k" or
     "Ki").  If true, use the full prefix ("kilo" or "kibi"). If false,
     use abbreviation ("k" or "Ki").
 
     """
 
     if not binary:
         base = 1000
         multiples = ('', 'k', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y')
     else:
         base = 1024
         multiples = ('', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi', 'Yi')
 
     sign = ""
     if size > 0:
         m = int(math.log(size, base))
     elif size < 0:
         sign = "-"
         size = -size
         m = int(math.log(size, base))
     else:
         m = 0
     if m > 8:
         m = 8
 
     if m == 0:
         precision = '%.0f'
     else:
         precision = '%%.%df' % precision
 
     size = precision % (size / math.pow(base, m))
 
     return '%s%s %s%s' % (sign, size.strip(), multiples[m], unit)
 
 
 def _check_memory_usage(worker):
     _memory_max_usage = worker._memory_max_usage
     if not _memory_max_usage:
         return
 
     _memory_usage_check_interval = worker._memory_usage_check_interval
     _memory_usage_recovery_threshold = memory_max_usage * worker._memory_usage_recovery_threshold
 
     elapsed = time.time() - worker._last_memory_check_time
     if elapsed > _memory_usage_check_interval:
         mem_usage = _get_process_rss()
         if mem_usage and mem_usage > _memory_max_usage:
             worker.log.info(
                 "memory usage %s > %s, forcing gc",
                 _format_data_size(mem_usage), _format_data_size(_memory_max_usage))
             # Try to clean it up by forcing a full collection.
             gc.collect()
             mem_usage = _get_process_rss()
             if mem_usage > _memory_usage_recovery_threshold:
                 # Didn't clean up enough, we'll have to terminate.
                 worker.log.warning(
                     "memory usage %s > %s after gc, quitting",
                     _format_data_size(mem_usage), _format_data_size(_memory_max_usage))
                 # This will cause worker to auto-restart itself
                 worker.alive = False
         worker._last_memory_check_time = time.time()
 
 
 def worker_int(worker):
     worker.log.info("pid=[%-10s] worker received INT or QUIT signal", worker.pid)
 
     # get traceback info, when a worker crashes
     def get_thread_id(t_id):
         id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
         return id2name.get(t_id, "unknown_thread_id")
 
     code = []
     for thread_id, stack in sys._current_frames().items():  # noqa
         code.append(
             "\n# Thread: %s(%d)" % (get_thread_id(thread_id), thread_id))
         for fname, lineno, name, line in traceback.extract_stack(stack):
             code.append('File: "%s", line %d, in %s' % (fname, lineno, name))
             if line:
                 code.append("  %s" % (line.strip()))
     worker.log.debug("\n".join(code))
 
 
 def worker_abort(worker):
     worker.log.info("pid=[%-10s] worker received SIGABRT signal", worker.pid)
 
 
 def worker_exit(server, worker):
     worker.log.info("pid=[%-10s] worker exit", worker.pid)
 
 
 def child_exit(server, worker):
     worker.log.info("pid=[%-10s] worker child exit", worker.pid)
 
 
 def pre_request(worker, req):
     worker.start_time = time.time()
     worker.log.debug(
         "GNCRN PRE  WORKER [cnt:%s]: %s %s", worker.nr, req.method, req.path)
 
 
 def post_request(worker, req, environ, resp):
     total_time = time.time() - worker.start_time
     # Gunicorn sometimes has problems with reading the status_code
     status_code = getattr(resp, 'status_code', '')
     worker.log.debug(
         "GNCRN POST WORKER [cnt:%s]: %s %s resp: %s, Load Time: %.4fs",
         worker.nr, req.method, req.path, status_code, total_time)
     _check_memory_usage(worker)
 
 
 def _filter_proxy(ip):
     """
     Passed in IP addresses in HEADERS can be in a special format of multiple
     ips. Those comma separated IPs are passed from various proxies in the
     chain of request processing. The left-most being the original client.
     We only care about the first IP which came from the org. client.
 
     :param ip: ip string from headers
     """
     if ',' in ip:
         _ips = ip.split(',')
         _first_ip = _ips[0].strip()
         return _first_ip
     return ip
 
 
 def _filter_port(ip):
     """
     Removes a port from ip, there are 4 main cases to handle here.
     - ipv4 eg. 127.0.0.1
     - ipv6 eg. ::1
     - ipv4+port eg. 127.0.0.1:8080
     - ipv6+port eg. [::1]:8080
 
     :param ip:
     """
     def is_ipv6(ip_addr):
         if hasattr(socket, 'inet_pton'):
             try:
                 socket.inet_pton(socket.AF_INET6, ip_addr)
             except socket.error:
                 return False
         else:
             return False
         return True
 
     if ':' not in ip:  # must be ipv4 pure ip
         return ip
 
     if '[' in ip and ']' in ip:  # ipv6 with port
         return ip.split(']')[0][1:].lower()
 
     # must be ipv6 or ipv4 with port
     if is_ipv6(ip):
         return ip
     else:
         ip, _port = ip.split(':')[:2]  # means ipv4+port
         return ip
 
 
 def get_ip_addr(environ):
     proxy_key = 'HTTP_X_REAL_IP'
     proxy_key2 = 'HTTP_X_FORWARDED_FOR'
     def_key = 'REMOTE_ADDR'
 
     def _filters(x):
         return _filter_port(_filter_proxy(x))
 
     ip = environ.get(proxy_key)
     if ip:
         return _filters(ip)
 
     ip = environ.get(proxy_key2)
     if ip:
         return _filters(ip)
 
     ip = environ.get(def_key, '0.0.0.0')
     return _filters(ip)
 
 
 class RhodeCodeLogger(Logger):
     """
     Custom Logger that allows some customization that gunicorn doesn't allow
     """
 
     datefmt = r"%Y-%m-%d %H:%M:%S"
 
     def __init__(self, cfg):
         Logger.__init__(self, cfg)
 
     def now(self):
         """ return date in RhodeCode Log format """
         now = time.time()
         msecs = int((now - int(now)) * 1000)
         return time.strftime(self.datefmt, time.localtime(now)) + '.{0:03d}'.format(msecs)
 
     def atoms(self, resp, req, environ, request_time):
         """ Gets atoms for log formatting.
         """
         status = resp.status
         if isinstance(status, str):
             status = status.split(None, 1)[0]
         atoms = {
             'h': get_ip_addr(environ),
             'l': '-',
             'u': self._get_user(environ) or '-',
             't': self.now(),
             'r': "%s %s %s" % (environ['REQUEST_METHOD'],
                                environ['RAW_URI'],
                                environ["SERVER_PROTOCOL"]),
             's': status,
             'm': environ.get('REQUEST_METHOD'),
             'U': environ.get('PATH_INFO'),
             'q': environ.get('QUERY_STRING'),
             'H': environ.get('SERVER_PROTOCOL'),
             'b': getattr(resp, 'sent', None) is not None and str(resp.sent) or '-',
             'B': getattr(resp, 'sent', None),
             'f': environ.get('HTTP_REFERER', '-'),
             'a': environ.get('HTTP_USER_AGENT', '-'),
             'T': request_time.seconds,
             'D': (request_time.seconds * 1000000) + request_time.microseconds,
             'M': (request_time.seconds * 1000) + int(request_time.microseconds/1000),
             'L': "%d.%06d" % (request_time.seconds, request_time.microseconds),
             'p': "<%s>" % os.getpid()
         }
 
         # add request headers
         if hasattr(req, 'headers'):
             req_headers = req.headers
         else:
             req_headers = req
 
         if hasattr(req_headers, "items"):
             req_headers = req_headers.items()
 
         atoms.update({"{%s}i" % k.lower(): v for k, v in req_headers})
 
         resp_headers = resp.headers
         if hasattr(resp_headers, "items"):
             resp_headers = resp_headers.items()
 
         # add response headers
         atoms.update({"{%s}o" % k.lower(): v for k, v in resp_headers})
 
         # add environ variables
         environ_variables = environ.items()
         atoms.update({"{%s}e" % k.lower(): v for k, v in environ_variables})
 
         return atoms
 
 
 logger_class = RhodeCodeLogger

configs/production.ini

@@ -1,824 +1,880 b''
 
 ; #########################################
 ; RHODECODE COMMUNITY EDITION CONFIGURATION
 ; #########################################
 
 [DEFAULT]
 ; Debug flag sets all loggers to debug, and enables request tracking
 debug = false
 
 ; ########################################################################
 ; EMAIL CONFIGURATION
 ; These settings will be used by the RhodeCode mailing system
 ; ########################################################################
 
 ; prefix all emails subjects with given prefix, helps filtering out emails
 #email_prefix = [RhodeCode]
 
 ; email FROM address all mails will be sent
 #app_email_from = rhodecode-noreply@localhost
 
 #smtp_server = mail.server.com
 #smtp_username =
 #smtp_password =
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
 
 [server:main]
 ; COMMON HOST/IP CONFIG, This applies mostly to develop setup,
 ; Host port for gunicorn are controlled by gunicorn_conf.py
 host = 127.0.0.1
 port = 10020
 
 
 ; ###########################
 ; GUNICORN APPLICATION SERVER
 ; ###########################
 
 ; run with gunicorn  --config gunicorn_conf.py --paste rhodecode.ini
 
 ; Module to use, this setting shouldn't be changed
 use = egg:gunicorn#main
 
 ; Prefix middleware for RhodeCode.
 ; recommended when using proxy setup.
 ; allows to set RhodeCode under a prefix in server.
 ; eg https://server.com/custom_prefix. Enable `filter-with =` option below as well.
 ; And set your prefix like: `prefix = /custom_prefix`
 ; be sure to also set beaker.session.cookie_path = /custom_prefix if you need
 ; to make your cookies only work on prefix url
 [filter:proxy-prefix]
 use = egg:PasteDeploy#prefix
 prefix = /
 
 [app:main]
 ; The %(here)s variable will be replaced with the absolute path of parent directory
 ; of this file
 ; Each option in the app:main can be override by an environmental variable
 ;
 ;To override an option:
 ;
 ;RC_<KeyName>
 ;Everything should be uppercase, . and - should be replaced by _.
 ;For example, if you have these configuration settings:
 ;rc_cache.repo_object.backend = foo
 ;can be overridden by
 ;export RC_CACHE_REPO_OBJECT_BACKEND=foo
 
 use = egg:rhodecode-enterprise-ce
 
 ; enable proxy prefix middleware, defined above
 #filter-with = proxy-prefix
 
 ; encryption key used to encrypt social plugin tokens,
 ; remote_urls with credentials etc, if not set it defaults to
 ; `beaker.session.secret`
 #rhodecode.encrypted_values.secret =
 
 ; decryption strict mode (enabled by default). It controls if decryption raises
 ; `SignatureVerificationError` in case of wrong key, or damaged encryption data.
 #rhodecode.encrypted_values.strict = false
 
 ; Pick algorithm for encryption. Either fernet (more secure) or aes (default)
 ; fernet is safer, and we strongly recommend switching to it.
 ; Due to backward compatibility aes is used as default.
 #rhodecode.encrypted_values.algorithm = fernet
 
 ; Return gzipped responses from RhodeCode (static files/application)
 gzip_responses = false
 
 ; Auto-generate javascript routes file on startup
 generate_js_files = false
 
 ; System global default language.
 ; All available languages: en (default), be, de, es, fr, it, ja, pl, pt, ru, zh
 lang = en
 
 ; Perform a full repository scan and import on each server start.
 ; Settings this to true could lead to very long startup time.
 startup.import_repos = false
 
 ; URL at which the application is running. This is used for Bootstrapping
 ; requests in context when no web request is available. Used in ishell, or
 ; SSH calls. Set this for events to receive proper url for SSH calls.
 app.base_url = http://rhodecode.local
 
 ; Host at which the Service API is running.
 app.service_api.host = http://rhodecode.local:10020
 
 ; Secret for Service API authentication.
 app.service_api.token =
 
 ; Unique application ID. Should be a random unique string for security.
 app_instance_uuid = rc-production
 
 ; Cut off limit for large diffs (size in bytes). If overall diff size on
 ; commit, or pull request exceeds this limit this diff will be displayed
 ; partially. E.g 512000 == 512Kb
 cut_off_limit_diff = 512000
 
 ; Cut off limit for large files inside diffs (size in bytes). Each individual
 ; file inside diff which exceeds this limit will be displayed partially.
 ;  E.g 128000 == 128Kb
 cut_off_limit_file = 128000
 
 ; Use cached version of vcs repositories everywhere. Recommended to be `true`
 vcs_full_cache = true
 
 ; Force https in RhodeCode, fixes https redirects, assumes it's always https.
 ; Normally this is controlled by proper flags sent from http server such as Nginx or Apache
 force_https = false
 
 ; use Strict-Transport-Security headers
 use_htsts = false
 
 ; Set to true if your repos are exposed using the dumb protocol
 git_update_server_info = false
 
 ; RSS/ATOM feed options
 rss_cut_off_limit = 256000
 rss_items_per_page = 10
 rss_include_diff = false
 
 ; gist URL alias, used to create nicer urls for gist. This should be an
 ; url that does rewrites to _admin/gists/{gistid}.
 ; example: http://gist.rhodecode.org/{gistid}. Empty means use the internal
 ; RhodeCode url, ie. http[s]://rhodecode.server/_admin/gists/{gistid}
 gist_alias_url =
 
 ; List of views (using glob pattern syntax) that AUTH TOKENS could be
 ; used for access.
 ; Adding ?auth_token=TOKEN_HASH to the url authenticates this request as if it
 ; came from the the logged in user who own this authentication token.
 ; Additionally @TOKEN syntax can be used to bound the view to specific
 ; authentication token. Such view would be only accessible when used together
 ; with this authentication token
 ; list of all views can be found under `/_admin/permissions/auth_token_access`
 ; The list should be "," separated and on a single line.
 ; Most common views to enable:
 
 #    RepoCommitsView:repo_commit_download
 #    RepoCommitsView:repo_commit_patch
 #    RepoCommitsView:repo_commit_raw
 #    RepoCommitsView:repo_commit_raw@TOKEN
 #    RepoFilesView:repo_files_diff
 #    RepoFilesView:repo_archivefile
 #    RepoFilesView:repo_file_raw
 #    GistView:*
 api_access_controllers_whitelist =
 
 ; Default encoding used to convert from and to unicode
 ; can be also a comma separated list of encoding in case of mixed encodings
 default_encoding = UTF-8
 
 ; instance-id prefix
 ; a prefix key for this instance used for cache invalidation when running
 ; multiple instances of RhodeCode, make sure it's globally unique for
 ; all running RhodeCode instances. Leave empty if you don't use it
 instance_id =
 
 ; Fallback authentication plugin. Set this to a plugin ID to force the usage
 ; of an authentication plugin also if it is disabled by it's settings.
 ; This could be useful if you are unable to log in to the system due to broken
 ; authentication settings. Then you can enable e.g. the internal RhodeCode auth
 ; module to log in again and fix the settings.
 ; Available builtin plugin IDs (hash is part of the ID):
 ; egg:rhodecode-enterprise-ce#rhodecode
 ; egg:rhodecode-enterprise-ce#pam
 ; egg:rhodecode-enterprise-ce#ldap
 ; egg:rhodecode-enterprise-ce#jasig_cas
 ; egg:rhodecode-enterprise-ce#headers
 ; egg:rhodecode-enterprise-ce#crowd
 
 #rhodecode.auth_plugin_fallback = egg:rhodecode-enterprise-ce#rhodecode
 
 ; Flag to control loading of legacy plugins in py:/path format
 auth_plugin.import_legacy_plugins = true
 
 ; alternative return HTTP header for failed authentication. Default HTTP
 ; response is 401 HTTPUnauthorized. Currently HG clients have troubles with
 ; handling that causing a series of failed authentication calls.
 ; Set this variable to 403 to return HTTPForbidden, or any other HTTP code
 ; This will be served instead of default 401 on bad authentication
 auth_ret_code =
 
 ; use special detection method when serving auth_ret_code, instead of serving
 ; ret_code directly, use 401 initially (Which triggers credentials prompt)
 ; and then serve auth_ret_code to clients
 auth_ret_code_detection = false
 
 ; locking return code. When repository is locked return this HTTP code. 2XX
 ; codes don't break the transactions while 4XX codes do
 lock_ret_code = 423
 
 ; Filesystem location were repositories should be stored
 repo_store.path = /var/opt/rhodecode_repo_store
 
 ; allows to setup custom hooks in settings page
 allow_custom_hooks_settings = true
 
 ; Generated license token required for EE edition license.
 ; New generated token value can be found in Admin > settings > license page.
 license_token =
 
 ; This flag hides sensitive information on the license page such as token, and license data
 license.hide_license_info = false
 
+; Import EE license from this license path
+#license.import_path = %(here)s/rhodecode_enterprise.license
+
+; import license 'if-missing' or 'force' (always override)
+; if-missing means apply license if it doesn't exist. 'force' option always overrides it
+license.import_path_mode = if-missing
+
 ; supervisor connection uri, for managing supervisor and logs.
 supervisor.uri =
 
 ; supervisord group name/id we only want this RC instance to handle
 supervisor.group_id = prod
 
 ; Display extended labs settings
 labs_settings_active = true
 
 ; Custom exception store path, defaults to TMPDIR
 ; This is used to store exception from RhodeCode in shared directory
 #exception_tracker.store_path =
 
 ; Send email with exception details when it happens
 #exception_tracker.send_email = false
 
 ; Comma separated list of recipients for exception emails,
 ; e.g [e-mail],[e-mail]
 ; Can be left empty, then emails will be sent to ALL super-admins
 #exception_tracker.send_email_recipients =
 
 ; optional prefix to Add to email Subject
 #exception_tracker.email_prefix = [RHODECODE ERROR]
 
-; File store configuration. This is used to store and serve uploaded files
-file_store.enabled = true
+; NOTE: this setting IS DEPRECATED:
+; file_store backend is always enabled
+#file_store.enabled = true
 
+; NOTE: this setting IS DEPRECATED:
+; file_store.backend = X -> use `file_store.backend.type = filesystem_v2` instead
 ; Storage backend, available options are: local
-file_store.backend = local
+#file_store.backend = local
 
+; NOTE: this setting IS DEPRECATED:
+; file_store.storage_path = X -> use `file_store.filesystem_v2.storage_path = X` instead
 ; path to store the uploaded binaries and artifacts
-file_store.storage_path = /var/opt/rhodecode_data/file_store
+#file_store.storage_path = /var/opt/rhodecode_data/file_store
+
+; Artifacts file-store, is used to store comment attachments and artifacts uploads.
+; file_store backend type: filesystem_v1, filesystem_v2 or objectstore (s3-based) are available as options
+; filesystem_v1 is backwards compat with pre 5.1 storage changes
+; new installations should choose filesystem_v2 or objectstore (s3-based), pick filesystem when migrating from
+; previous installations to keep the artifacts without a need of migration
+#file_store.backend.type = filesystem_v2
+
+; filesystem options...
+#file_store.filesystem_v1.storage_path = /var/opt/rhodecode_data/artifacts_file_store
+
+; filesystem_v2 options...
+#file_store.filesystem_v2.storage_path = /var/opt/rhodecode_data/artifacts_file_store
+#file_store.filesystem_v2.shards = 8
 
+; objectstore options...
+; url for s3 compatible storage that allows to upload artifacts
+; e.g http://minio:9000
+#file_store.backend.type = objectstore
+#file_store.objectstore.url = http://s3-minio:9000
+
+; a top-level bucket to put all other shards in
+; objects will be stored in rhodecode-file-store/shard-N based on the bucket_shards number
+#file_store.objectstore.bucket = rhodecode-file-store
+
+; number of sharded buckets to create to distribute archives across
+; default is 8 shards
+#file_store.objectstore.bucket_shards = 8
+
+; key for s3 auth
+#file_store.objectstore.key = s3admin
+
+; secret for s3 auth
+#file_store.objectstore.secret = s3secret4
+
+;region for s3 storage
+#file_store.objectstore.region = eu-central-1
 
 ; Redis url to acquire/check generation of archives locks
 archive_cache.locking.url = redis://redis:6379/1
 
 ; Storage backend, only 'filesystem' and 'objectstore' are available now
 archive_cache.backend.type = filesystem
 
 ; url for s3 compatible storage that allows to upload artifacts
 ; e.g http://minio:9000
 archive_cache.objectstore.url = http://s3-minio:9000
 
 ; key for s3 auth
 archive_cache.objectstore.key = key
 
 ; secret for s3 auth
 archive_cache.objectstore.secret = secret
 
 ;region for s3 storage
 archive_cache.objectstore.region = eu-central-1
 
 ; number of sharded buckets to create to distribute archives across
 ; default is 8 shards
 archive_cache.objectstore.bucket_shards = 8
 
 ; a top-level bucket to put all other shards in
 ; objects will be stored in rhodecode-archive-cache/shard-N based on the bucket_shards number
 archive_cache.objectstore.bucket = rhodecode-archive-cache
 
 ; if true, this cache will try to retry with retry_attempts=N times waiting retry_backoff time
 archive_cache.objectstore.retry = false
 
 ; number of seconds to wait for next try using retry
 archive_cache.objectstore.retry_backoff = 1
 
 ; how many tries do do a retry fetch from this backend
 archive_cache.objectstore.retry_attempts = 10
 
 ; Default is $cache_dir/archive_cache if not set
 ; Generated repo archives will be cached at this location
 ; and served from the cache during subsequent requests for the same archive of
 ; the repository. This path is important to be shared across filesystems and with
 ; RhodeCode and vcsserver
 archive_cache.filesystem.store_dir = /var/opt/rhodecode_data/archive_cache
 
 ; The limit in GB sets how much data we cache before recycling last used, defaults to 10 gb
 archive_cache.filesystem.cache_size_gb = 40
 
 ; Eviction policy used to clear out after cache_size_gb limit is reached
 archive_cache.filesystem.eviction_policy = least-recently-stored
 
 ; By default cache uses sharding technique, this specifies how many shards are there
 ; default is 8 shards
 archive_cache.filesystem.cache_shards = 8
 
 ; if true, this cache will try to retry with retry_attempts=N times waiting retry_backoff time
 archive_cache.filesystem.retry = false
 
 ; number of seconds to wait for next try using retry
 archive_cache.filesystem.retry_backoff = 1
 
 ; how many tries do do a retry fetch from this backend
 archive_cache.filesystem.retry_attempts = 10
 
 
 ; #############
 ; CELERY CONFIG
 ; #############
 
 ; manually run celery: /path/to/celery worker --task-events --beat --app rhodecode.lib.celerylib.loader --scheduler rhodecode.lib.celerylib.scheduler.RcScheduler --loglevel DEBUG --ini /path/to/rhodecode.ini
 
 use_celery = true
 
 ; path to store schedule database
 #celerybeat-schedule.path =
 
 ; connection url to the message broker (default redis)
 celery.broker_url = redis://redis:6379/8
 
 ; results backend to get results for (default redis)
 celery.result_backend = redis://redis:6379/8
 
 ; rabbitmq example
 #celery.broker_url = amqp://rabbitmq:qweqwe@localhost:5672/rabbitmqhost
 
 ; maximum tasks to execute before worker restart
 celery.max_tasks_per_child = 20
 
 ; tasks will never be sent to the queue, but executed locally instead.
 celery.task_always_eager = false
 
 ; #############
 ; DOGPILE CACHE
 ; #############
 
 ; Default cache dir for caches. Putting this into a ramdisk can boost performance.
 ; eg. /tmpfs/data_ramdisk, however this directory might require large amount of space
 cache_dir = /var/opt/rhodecode_data
 
 ; *********************************************
 ; `sql_cache_short` cache for heavy SQL queries
 ; Only supported backend is `memory_lru`
 ; *********************************************
 rc_cache.sql_cache_short.backend = dogpile.cache.rc.memory_lru
 rc_cache.sql_cache_short.expiration_time = 30
 
 
 ; *****************************************************
 ; `cache_repo_longterm` cache for repo object instances
 ; Only supported backend is `memory_lru`
 ; *****************************************************
 rc_cache.cache_repo_longterm.backend = dogpile.cache.rc.memory_lru
 ; by default we use 30 Days, cache is still invalidated on push
 rc_cache.cache_repo_longterm.expiration_time = 2592000
 ; max items in LRU cache, set to smaller number to save memory, and expire last used caches
 rc_cache.cache_repo_longterm.max_size = 10000
 
 
 ; *********************************************
 ; `cache_general` cache for general purpose use
 ; for simplicity use rc.file_namespace backend,
 ; for performance and scale use rc.redis
 ; *********************************************
 rc_cache.cache_general.backend = dogpile.cache.rc.file_namespace
 rc_cache.cache_general.expiration_time = 43200
 ; file cache store path. Defaults to `cache_dir =` value or tempdir if both values are not set
 #rc_cache.cache_general.arguments.filename = /tmp/cache_general_db
 
 ; alternative `cache_general` redis backend with distributed lock
 #rc_cache.cache_general.backend = dogpile.cache.rc.redis
 #rc_cache.cache_general.expiration_time = 300
 
 ; redis_expiration_time needs to be greater then expiration_time
 #rc_cache.cache_general.arguments.redis_expiration_time = 7200
 
 #rc_cache.cache_general.arguments.host = localhost
 #rc_cache.cache_general.arguments.port = 6379
 #rc_cache.cache_general.arguments.db = 0
 #rc_cache.cache_general.arguments.socket_timeout = 30
 ; more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_general.arguments.distributed_lock = true
 
 ; auto-renew lock to prevent stale locks, slower but safer. Use only if problems happen
 #rc_cache.cache_general.arguments.lock_auto_renewal = true
 
 ; *************************************************
 ; `cache_perms` cache for permission tree, auth TTL
 ; for simplicity use rc.file_namespace backend,
 ; for performance and scale use rc.redis
 ; *************************************************
 rc_cache.cache_perms.backend = dogpile.cache.rc.file_namespace
 rc_cache.cache_perms.expiration_time = 3600
 ; file cache store path. Defaults to `cache_dir =` value or tempdir if both values are not set
 #rc_cache.cache_perms.arguments.filename = /tmp/cache_perms_db
 
 ; alternative `cache_perms` redis backend with distributed lock
 #rc_cache.cache_perms.backend = dogpile.cache.rc.redis
 #rc_cache.cache_perms.expiration_time = 300
 
 ; redis_expiration_time needs to be greater then expiration_time
 #rc_cache.cache_perms.arguments.redis_expiration_time = 7200
 
 #rc_cache.cache_perms.arguments.host = localhost
 #rc_cache.cache_perms.arguments.port = 6379
 #rc_cache.cache_perms.arguments.db = 0
 #rc_cache.cache_perms.arguments.socket_timeout = 30
 ; more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_perms.arguments.distributed_lock = true
 
 ; auto-renew lock to prevent stale locks, slower but safer. Use only if problems happen
 #rc_cache.cache_perms.arguments.lock_auto_renewal = true
 
 ; ***************************************************
 ; `cache_repo` cache for file tree, Readme, RSS FEEDS
 ; for simplicity use rc.file_namespace backend,
 ; for performance and scale use rc.redis
 ; ***************************************************
 rc_cache.cache_repo.backend = dogpile.cache.rc.file_namespace
 rc_cache.cache_repo.expiration_time = 2592000
 ; file cache store path. Defaults to `cache_dir =` value or tempdir if both values are not set
 #rc_cache.cache_repo.arguments.filename = /tmp/cache_repo_db
 
 ; alternative `cache_repo` redis backend with distributed lock
 #rc_cache.cache_repo.backend = dogpile.cache.rc.redis
 #rc_cache.cache_repo.expiration_time = 2592000
 
 ; redis_expiration_time needs to be greater then expiration_time
 #rc_cache.cache_repo.arguments.redis_expiration_time = 2678400
 
 #rc_cache.cache_repo.arguments.host = localhost
 #rc_cache.cache_repo.arguments.port = 6379
 #rc_cache.cache_repo.arguments.db = 1
 #rc_cache.cache_repo.arguments.socket_timeout = 30
 ; more Redis options: https://dogpilecache.sqlalchemy.org/en/latest/api.html#redis-backends
 #rc_cache.cache_repo.arguments.distributed_lock = true
 
 ; auto-renew lock to prevent stale locks, slower but safer. Use only if problems happen
 #rc_cache.cache_repo.arguments.lock_auto_renewal = true
 
 ; ##############
 ; BEAKER SESSION
 ; ##############
 
 ; beaker.session.type is type of storage options for the logged users sessions. Current allowed
 ; types are file, ext:redis, ext:database, ext:memcached
 ; Fastest ones are ext:redis and ext:database, DO NOT use memory type for session
 #beaker.session.type = file
 #beaker.session.data_dir = %(here)s/data/sessions
 
 ; Redis based sessions
 beaker.session.type = ext:redis
 beaker.session.url = redis://redis:6379/2
 
 ; DB based session, fast, and allows easy management over logged in users
 #beaker.session.type = ext:database
 #beaker.session.table_name = db_session
 #beaker.session.sa.url = postgresql://postgres:secret@localhost/rhodecode
 #beaker.session.sa.url = mysql://root:secret@127.0.0.1/rhodecode
 #beaker.session.sa.pool_recycle = 3600
 #beaker.session.sa.echo = false
 
 beaker.session.key = rhodecode
 beaker.session.secret = production-rc-uytcxaz
 beaker.session.lock_dir = /data_ramdisk/lock
 
 ; Secure encrypted cookie. Requires AES and AES python libraries
 ; you must disable beaker.session.secret to use this
 #beaker.session.encrypt_key = key_for_encryption
 #beaker.session.validate_key = validation_key
 
 ; Sets session as invalid (also logging out user) if it haven not been
 ; accessed for given amount of time in seconds
 beaker.session.timeout = 2592000
 beaker.session.httponly = true
 
 ; Path to use for the cookie. Set to prefix if you use prefix middleware
 #beaker.session.cookie_path = /custom_prefix
 
 ; Set https secure cookie
 beaker.session.secure = false
 
 ; default cookie expiration time in seconds, set to `true` to set expire
 ; at browser close
 #beaker.session.cookie_expires = 3600
 
 ; #############################
 ; SEARCH INDEXING CONFIGURATION
 ; #############################
 
 ; Full text search indexer is available in rhodecode-tools under
 ; `rhodecode-tools index` command
 
 ; WHOOSH Backend, doesn't require additional services to run
 ; it works good with few dozen repos
 search.module = rhodecode.lib.index.whoosh
 search.location = %(here)s/data/index
 
 ; ####################
 ; CHANNELSTREAM CONFIG
 ; ####################
 
 ; channelstream enables persistent connections and live notification
 ; in the system. It's also used by the chat system
 
 channelstream.enabled = true
 
 ; server address for channelstream server on the backend
 channelstream.server = channelstream:9800
 
 ; location of the channelstream server from outside world
 ; use ws:// for http or wss:// for https. This address needs to be handled
 ; by external HTTP server such as Nginx or Apache
 ; see Nginx/Apache configuration examples in our docs
 channelstream.ws_url = ws://rhodecode.yourserver.com/_channelstream
 channelstream.secret = ENV_GENERATED
 channelstream.history.location = /var/opt/rhodecode_data/channelstream_history
 
 ; Internal application path that Javascript uses to connect into.
 ; If you use proxy-prefix the prefix should be added before /_channelstream
 channelstream.proxy_path = /_channelstream
 
 
 ; ##############################
 ; MAIN RHODECODE DATABASE CONFIG
 ; ##############################
 
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 #sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode?charset=utf8
 ; pymysql is an alternative driver for MySQL, use in case of problems with default one
 #sqlalchemy.db1.url = mysql+pymysql://root:qweqwe@localhost/rhodecode
 
 sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
 
 ; see sqlalchemy docs for other advanced settings
 ; print the sql statements to output
 sqlalchemy.db1.echo = false
 
 ; recycle the connections after this amount of seconds
 sqlalchemy.db1.pool_recycle = 3600
 
 ; the number of connections to keep open inside the connection pool.
 ; 0 indicates no limit
 ; the general calculus with gevent is:
 ; if your system allows 500 concurrent greenlets (max_connections) that all do database access,
 ; then increase pool size + max overflow so that they add up to 500.
 #sqlalchemy.db1.pool_size = 5
 
 ; The number of connections to allow in connection pool "overflow", that is
 ; connections that can be opened above and beyond the pool_size setting,
 ; which defaults to five.
 #sqlalchemy.db1.max_overflow = 10
 
 ; Connection check ping, used to detect broken database connections
 ; could be enabled to better handle cases if MySQL has gone away errors
 #sqlalchemy.db1.ping_connection = true
 
 ; ##########
 ; VCS CONFIG
 ; ##########
 vcs.server.enable = true
 vcs.server = vcsserver:10010
 
 ; Web server connectivity protocol, responsible for web based VCS operations
 ; Available protocols are:
 ; `http` - use http-rpc backend (default)
 vcs.server.protocol = http
 
 ; Push/Pull operations protocol, available options are:
 ; `http` - use http-rpc backend (default)
 vcs.scm_app_implementation = http
 
 ; Push/Pull operations hooks protocol, available options are:
 ; `http` - use http-rpc backend (default)
 ; `celery` - use celery based hooks
-vcs.hooks.protocol = http
+#DEPRECATED:vcs.hooks.protocol = http
+vcs.hooks.protocol.v2 = celery
 
 ; Host on which this instance is listening for hooks. vcsserver will call this host to pull/push hooks so it should be
 ; accessible via network.
 ; Use vcs.hooks.host = "*" to bind to current hostname (for Docker)
 vcs.hooks.host = *
 
 ; Start VCSServer with this instance as a subprocess, useful for development
 vcs.start_server = false
 
 ; List of enabled VCS backends, available options are:
 ; `hg`  - mercurial
 ; `git` - git
 ; `svn` - subversion
 vcs.backends = hg, git, svn
 
 ; Wait this number of seconds before killing connection to the vcsserver
 vcs.connection_timeout = 3600
 
 ; Cache flag to cache vcsserver remote calls locally
 ; It uses cache_region `cache_repo`
 vcs.methods.cache = true
 
+; Filesystem location where Git lfs objects should be stored
+vcs.git.lfs.storage_location = /var/opt/rhodecode_repo_store/.cache/git_lfs_store
+
+; Filesystem location where Mercurial largefile objects should be stored
+vcs.hg.largefiles.storage_location = /var/opt/rhodecode_repo_store/.cache/hg_largefiles_store
+
 ; ####################################################
 ; Subversion proxy support (mod_dav_svn)
 ; Maps RhodeCode repo groups into SVN paths for Apache
 ; ####################################################
 
 ; Compatibility version when creating SVN repositories. Defaults to newest version when commented out.
 ; Set a numeric version for your current SVN e.g 1.8, or 1.12
 ; Legacy available options are: pre-1.4-compatible, pre-1.5-compatible, pre-1.6-compatible, pre-1.8-compatible, pre-1.9-compatible
 #vcs.svn.compatible_version = 1.8
 
 ; Redis connection settings for svn integrations logic
 ; This connection string needs to be the same on ce and vcsserver
 vcs.svn.redis_conn = redis://redis:6379/0
 
 ; Enable SVN proxy of requests over HTTP
 vcs.svn.proxy.enabled = true
 
 ; host to connect to running SVN subsystem
 vcs.svn.proxy.host = http://svn:8090
 
 ; Enable or disable the config file generation.
 svn.proxy.generate_config = true
 
 ; Generate config file with `SVNListParentPath` set to `On`.
 svn.proxy.list_parent_path = true
 
 ; Set location and file name of generated config file.
 svn.proxy.config_file_path = /etc/rhodecode/conf/svn/mod_dav_svn.conf
 
 ; alternative mod_dav config template. This needs to be a valid mako template
 ; Example template can be found in the source code:
 ; rhodecode/apps/svn_support/templates/mod-dav-svn.conf.mako
 #svn.proxy.config_template = ~/.rccontrol/enterprise-1/custom_svn_conf.mako
 
 ; Used as a prefix to the `Location` block in the generated config file.
 ; In most cases it should be set to `/`.
 svn.proxy.location_root = /
 
 ; Command to reload the mod dav svn configuration on change.
 ; Example: `/etc/init.d/apache2 reload` or /home/USER/apache_reload.sh
 ; Make sure user who runs RhodeCode process is allowed to reload Apache
 #svn.proxy.reload_cmd = /etc/init.d/apache2 reload
 
 ; If the timeout expires before the reload command finishes, the command will
 ; be killed. Setting it to zero means no timeout. Defaults to 10 seconds.
 #svn.proxy.reload_timeout = 10
 
 ; ####################
 ; SSH Support Settings
 ; ####################
 
 ; Defines if a custom authorized_keys file should be created and written on
 ; any change user ssh keys. Setting this to false also disables possibility
 ; of adding SSH keys by users from web interface. Super admins can still
 ; manage SSH Keys.
 ssh.generate_authorized_keyfile = true
 
 ; Options for ssh, default is `no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding`
 # ssh.authorized_keys_ssh_opts =
 
 ; Path to the authorized_keys file where the generate entries are placed.
 ; It is possible to have multiple key files specified in `sshd_config` e.g.
 ; AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/authorized_keys_rhodecode
 ssh.authorized_keys_file_path = /etc/rhodecode/conf/ssh/authorized_keys_rhodecode
 
 ; Command to execute the SSH wrapper. The binary is available in the
 ; RhodeCode installation directory.
 ; legacy: /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper
 ; new rewrite: /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper-v2
-ssh.wrapper_cmd = /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper
+#DEPRECATED: ssh.wrapper_cmd = /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper
+ssh.wrapper_cmd.v2 = /usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper-v2
 
 ; Allow shell when executing the ssh-wrapper command
 ssh.wrapper_cmd_allow_shell = false
 
 ; Enables logging, and detailed output send back to the client during SSH
 ; operations. Useful for debugging, shouldn't be used in production.
 ssh.enable_debug_logging = false
 
 ; Paths to binary executable, by default they are the names, but we can
 ; override them if we want to use a custom one
 ssh.executable.hg = /usr/local/bin/rhodecode_bin/vcs_bin/hg
 ssh.executable.git = /usr/local/bin/rhodecode_bin/vcs_bin/git
 ssh.executable.svn = /usr/local/bin/rhodecode_bin/vcs_bin/svnserve
 
 ; Enables SSH key generator web interface. Disabling this still allows users
 ; to add their own keys.
 ssh.enable_ui_key_generator = true
 
 ; Statsd client config, this is used to send metrics to statsd
 ; We recommend setting statsd_exported and scrape them using Prometheus
 #statsd.enabled = false
 #statsd.statsd_host = 0.0.0.0
 #statsd.statsd_port = 8125
 #statsd.statsd_prefix =
 #statsd.statsd_ipv6 = false
 
 ; configure logging automatically at server startup set to false
 ; to use the below custom logging config.
 ; RC_LOGGING_FORMATTER
 ; RC_LOGGING_LEVEL
 ; env variables can control the settings for logging in case of autoconfigure
 
 #logging.autoconfigure = true
 
 ; specify your own custom logging config file to configure logging
 #logging.logging_conf_file = /path/to/custom_logging.ini
 
 ; Dummy marker to add new entries after.
 ; Add any custom entries below. Please don't remove this marker.
 custom.conf = 1
 
 
 ; #####################
 ; LOGGING CONFIGURATION
 ; #####################
 
 [loggers]
 keys = root, sqlalchemy, beaker, celery, rhodecode, ssh_wrapper
 
 [handlers]
 keys = console, console_sql
 
 [formatters]
 keys = generic, json, color_formatter, color_formatter_sql
 
 ; #######
 ; LOGGERS
 ; #######
 [logger_root]
 level = NOTSET
 handlers = console
 
 [logger_sqlalchemy]
 level = INFO
 handlers = console_sql
 qualname = sqlalchemy.engine
 propagate = 0
 
 [logger_beaker]
 level = DEBUG
 handlers =
 qualname = beaker.container
 propagate = 1
 
 [logger_rhodecode]
 level = DEBUG
 handlers =
 qualname = rhodecode
 propagate = 1
 
 [logger_ssh_wrapper]
 level = DEBUG
 handlers =
 qualname = ssh_wrapper
 propagate = 1
 
 [logger_celery]
 level = DEBUG
 handlers =
 qualname = celery
 
 
 ; ########
 ; HANDLERS
 ; ########
 
 [handler_console]
 class = StreamHandler
 args = (sys.stderr, )
 level = INFO
 ; To enable JSON formatted logs replace 'generic/color_formatter' with 'json'
 ; This allows sending properly formatted logs to grafana loki or elasticsearch
 formatter = generic
 
 [handler_console_sql]
 ; "level = DEBUG" logs SQL queries and results.
 ; "level = INFO" logs SQL queries.
 ; "level = WARN" logs neither.  (Recommended for production systems.)
 class = StreamHandler
 args = (sys.stderr, )
 level = WARN
 ; To enable JSON formatted logs replace 'generic/color_formatter_sql' with 'json'
 ; This allows sending properly formatted logs to grafana loki or elasticsearch
 formatter = generic
 
 ; ##########
 ; FORMATTERS
 ; ##########
 
 [formatter_generic]
 class = rhodecode.lib.logging_formatter.ExceptionAwareFormatter
 format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter]
 class = rhodecode.lib.logging_formatter.ColorFormatter
 format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_color_formatter_sql]
 class = rhodecode.lib.logging_formatter.ColorFormatterSql
 format = %(asctime)s.%(msecs)03d [%(process)d] %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %Y-%m-%d %H:%M:%S
 
 [formatter_json]
 format = %(timestamp)s %(levelname)s %(name)s %(message)s %(req_id)s
 class = rhodecode.lib._vendor.jsonlogger.JsonFormatter

docs/Dockerfile

@@ -1,33 +1,39 b''
 FROM python:3.12.0-bullseye
 
 WORKDIR /project
 
 RUN apt-get update \
  && apt-get install --no-install-recommends --yes \
     curl \
     zip \
     graphviz \
     dvipng \
     imagemagick \
     make \
     latexmk \
     texlive-latex-recommended \
     texlive-latex-extra \
     texlive-xetex \
     fonts-freefont-otf \
     texlive-fonts-recommended \
     texlive-lang-greek \
     tex-gyre \
  && apt-get autoremove \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
 
+RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
+    unzip awscliv2.zip && \
+    ./aws/install && \
+    rm -rf ./aws && \
+    rm awscliv2.zip
+
 RUN \
     python3 -m pip install --no-cache-dir --upgrade pip && \
     python3 -m pip install --no-cache-dir Sphinx Pillow
 
 ADD requirements_docs.txt /project
 RUN \
     python3 -m pip install -r requirements_docs.txt
 
 CMD ["sphinx-build", "-M", "html", ".", "_build"]

docs/admin/system-overview.rst

@@ -1,172 +1,168 b''
 .. _system-overview-ref:
 
 System Overview
 ===============
 
 Latest Version
 --------------
 
 * |release| on Unix and Windows systems.
 
 System Architecture
 -------------------
 
 The following diagram shows a typical production architecture.
 
 .. image:: ../images/architecture-diagram.png
   :align: center
 
 Supported Operating Systems
 ---------------------------
 
 Linux
 ^^^^^
 
 * Ubuntu 14.04+
 * CentOS 6.2, 7 and 8
 * RHEL 6.2, 7 and 8
 * Debian 7.8
 * RedHat Fedora
 * Arch Linux
 * SUSE Linux
 
 Windows
 ^^^^^^^
 
 * Windows Vista Ultimate 64bit
 * Windows 7 Ultimate 64bit
 * Windows 8 Professional 64bit
 * Windows 8.1 Enterprise 64bit
 * Windows Server 2008 64bit
 * Windows Server 2008-R2 64bit
 * Windows Server 2012 64bit
 
 Supported Databases
 -------------------
 
 * SQLite
 * MySQL
 * MariaDB
 * PostgreSQL
 
 Supported Browsers
 ------------------
 
 * Chrome
 * Safari
 * Firefox
 * Internet Explorer 10 & 11
 
 System Requirements
 -------------------
 
 |RCE| performs best on machines with ultra-fast hard disks. Generally disk
 performance is more important than CPU performance. In a corporate production
 environment handling 1000s of users and |repos| you should deploy on a 12+
 core 64GB RAM server. In short, the more RAM the better.
 
 
 For example:
 
  - for team of 1 - 5 active users you can run on 1GB RAM machine with 1CPU
  - above 250 active users, |RCE| needs at least 8GB of memory.
    Number of CPUs is less important, but recommended to have at least 2-3 CPUs
 
 
 .. _config-rce-files:
 
 Configuration Files
 -------------------
 
 * :file:`config/_shared/rhodecode.ini`
 * :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`
 * :file:`/home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini`
 * :file:`/home/{user}/.rccontrol/supervisor/supervisord.ini`
 * :file:`/home/{user}/.rccontrol.ini`
 * :file:`/home/{user}/.rhoderc`
 * :file:`/home/{user}/.rccontrol/cache/MANIFEST`
 
 For more information, see the :ref:`config-files` section.
 
 Log Files
 ---------
 
 * :file:`/home/{user}/.rccontrol/{instance-id}/enterprise.log`
 * :file:`/home/{user}/.rccontrol/{vcsserver-id}/vcsserver.log`
 * :file:`/home/{user}/.rccontrol/supervisor/supervisord.log`
 * :file:`/tmp/rccontrol.log`
 * :file:`/tmp/rhodecode_tools.log`
 
 Storage Files
 -------------
 
 * :file:`/home/{user}/.rccontrol/{instance-id}/data/index/{index-file.toc}`
 * :file:`/home/{user}/repos/.rc_gist_store`
 * :file:`/home/{user}/.rccontrol/{instance-id}/rhodecode.db`
 * :file:`/opt/rhodecode/store/{unique-hash}`
 
 Default Repositories Location
 -----------------------------
 
 * :file:`/home/{user}/repos`
 
 Connection Methods
 ------------------
 
 * HTTPS
 * SSH
 * |RCE| API
 
 Internationalization Support
 ----------------------------
 
 Currently available in the following languages, see `Transifex`_ for the
 latest details. If you want a new language added, please contact us. To
 configure your language settings, see the :ref:`set-lang` section.
 
 .. hlist::
 
   * Belorussian
   * Chinese
   * French
   * German
   * Italian
   * Japanese
   * Portuguese
   * Polish
   * Russian
   * Spanish
 
 Licencing Information
 ---------------------
 
 * See licencing information `here`_
 
 Peer-to-peer Failover Support
 -----------------------------
 
 * Yes
 
-Additional Binaries
--------------------
-
-* Yes, see :ref:`rhodecode-nix-ref` for full details.
 
 Remote Connectivity
 -------------------
 
 * Available
 
 Executable Files
 ----------------
 
 Windows: :file:`RhodeCode-installer-{version}.exe`
 
 Deprecated Support
 ------------------
 
 - Internet Explorer 8 support deprecated since version 3.7.0.
 - Internet Explorer 9 support deprecated since version 3.8.0.
 
 .. _here: https://rhodecode.com/licenses/
 .. _Transifex: https://explore.transifex.com/rhodecode/RhodeCode/

docs/auth/auth-saml-bulk-enroll-users.rst

@@ -1,88 +1,90 b''
 .. _auth-saml-bulk-enroll-users-ref:
 
 
 Bulk enroll multiple existing users
 -----------------------------------
 
 
 RhodeCode Supports standard SAML 2.0 SSO for the web-application part.
 Below is an example how to enroll list of all or some users to use SAML authentication.
 This method simply enables SAML authentication for many users at once.
 
 
 From the server RhodeCode Enterprise is running run ishell on the instance which we
 want to apply the SAML migration::
 
-    rccontrol ishell enterprise-1
+    ./rcstack cli ishell
 
 Follow these steps to enable SAML authentication for multiple users.
 
 
 1) Create a user_id => attribute mapping
 
 
 `saml2user` is a mapping of external ID from SAML provider such as OneLogin, DuoSecurity, Google.
 This mapping consists of local rhodecode user_id mapped to set of required attributes needed to bind SAML
 account to internal rhodecode user.
 For example, 123 is local rhodecode user_id, and '48253211' is OneLogin ID.
 For other providers you'd have to figure out what would be the user-id, sometimes it's the email, i.e for Google
 The most important this id needs to be unique for each user.
 
 .. code-block:: python
 
     In [1]: saml2user = {
        ...: # OneLogin, uses externalID available to read from in the UI
        ...: 123: {'id': '48253211'},
        ...: # for Google/DuoSecurity email is also an option for unique ID
        ...: 124: {'id': '[e-mail]'},
        ...: }
 
 
 2) Import the plugin you want to run migration for.
 
 From available options pick only one and run the `import` statement
 
 .. code-block:: python
 
     # for Duo Security
     In [2]: from rc_auth_plugins.auth_duo_security import RhodeCodeAuthPlugin
+    # for Azure Entra
+    In [2]: from rc_auth_plugins.auth_azure import RhodeCodeAuthPlugin
     # for OneLogin
     In [2]: from rc_auth_plugins.auth_onelogin import RhodeCodeAuthPlugin
     # generic SAML plugin
     In [2]: from rc_auth_plugins.auth_saml import RhodeCodeAuthPlugin
 
 3) Run the migration based on saml2user mapping.
 
 Enter in the ishell prompt
 
 .. code-block:: python
 
     In [3]: for user in User.get_all():
        ...:     existing_identity = ExternalIdentity().query().filter(ExternalIdentity.local_user_id == user.user_id).scalar()
        ...:     attrs = saml2user.get(user.user_id)
        ...:     provider = RhodeCodeAuthPlugin.uid
        ...:     if existing_identity:
-       ...:         print('Identity for user `{}` already exists, skipping'.format(user.username))
+       ...:         print(f'Identity for user `{user.username}` already exists, skipping')
        ...:         continue
        ...:     if attrs:
        ...:         external_id = attrs['id']
        ...:         new_external_identity = ExternalIdentity()
        ...:         new_external_identity.external_id = external_id
-       ...:         new_external_identity.external_username = '{}-saml-{}'.format(user.username, user.user_id)
+       ...:         new_external_identity.external_username = f'{user.username}-saml-{user.user_id}'
        ...:         new_external_identity.provider_name = provider
        ...:         new_external_identity.local_user_id = user.user_id
        ...:         new_external_identity.access_token = ''
        ...:         new_external_identity.token_secret = ''
        ...:         new_external_identity.alt_token = ''
        ...:         Session().add(ex_identity)
        ...:         Session().commit()
-       ...:         print('Set user `{}` external identity bound to ExternalID:{}'.format(user.username, external_id))
+       ...:         print(f'Set user `{user.username}` external identity bound to ExternalID:{external_id}')
 
 .. note::
 
     saml2user can be really big and hard to maintain in ishell. It's also possible
     to load it as a JSON file prepared before and stored on disk. To do so run::
 
         import json
         saml2user = json.loads(open('/path/to/saml2user.json','rb').read())
 

docs/auth/auth-saml-duosecurity.rst

@@ -1,105 +1,161 b''
 .. _config-saml-duosecurity-ref:
 
 
 SAML 2.0 with Duo Security
 --------------------------
 
 **This plugin is available only in EE Edition.**
 
 |RCE| supports SAML 2.0 Authentication with Duo Security provider. This allows
 users to log-in to RhodeCode via SSO mechanism of external identity provider
 such as Duo. The login can be triggered either by the external IDP, or internally
 by clicking specific authentication button on the log-in page.
 
 
 Configuration steps
 ^^^^^^^^^^^^^^^^^^^
 
 To configure Duo Security SAML authentication, use the following steps:
 
 1. From the |RCE| interface, select
    :menuselection:`Admin --> Authentication`
 2. Activate the `Duo Security` plugin and select :guilabel:`Save`
 3. Go to newly available menu option called `Duo Security` on the left side.
 4. Check the `enabled` check box in the plugin configuration section,
    and fill in the required SAML information and :guilabel:`Save`, for more details,
    see :ref:`config-saml-duosecurity`
 
 
 .. _config-saml-duosecurity:
 
 
 Example SAML Duo Security configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Example configuration for SAML 2.0 with Duo Security provider::
+Example configuration for SAML 2.0 with Duo Security provider
+
+
+Enabled
+    `True`:
 
-    *option*: `enabled` => `True`
-    # Enable or disable this authentication plugin.
+    .. note::
+      Enable or disable this authentication plugin.
+
+
+Auth Cache TTL
+    `30`:
 
-    *option*: `cache_ttl` => `0`
-    # Amount of seconds to cache the authentication and permissions check response call for this plugin.
-    # Useful for expensive calls like LDAP to improve the performance of the system (0 means disabled).
+    .. note::
+      Amount of seconds to cache the authentication and permissions check response call for this plugin.
+      Useful for expensive calls like LDAP to improve the performance of the system (0 means disabled).
+
+Debug
+    `True`:
 
-    *option*: `debug` => `True`
-    # Enable or disable debug mode that shows SAML errors in the RhodeCode logs.
+    .. note::
+      Enable or disable debug mode that shows SAML errors in the RhodeCode logs.
+
+
+Auth button name
+    `Azure Entra ID`:
 
-    *option*: `entity_id` => `http://rc-app.com/dag/saml2/idp/metadata.php`
-    # Identity Provider entity/metadata URI.
-    # E.g. https://duo-gateway.com/dag/saml2/idp/metadata.php
+    .. note::
+      Alternative authentication display name. E.g AzureAuth, CorporateID etc.
+
+
+Entity ID
+    `https://my-duo-gateway.com/dag/saml2/idp/metadata.php`:
+
+    .. note::
+      Identity Provider entity/metadata URI.
+      E.g. https://duo-gateway.com/dag/saml2/idp/metadata.php
+
+SSO URL
+    `https://duo-gateway.com/dag/saml2/idp/SSOService.php?spentityid=<metadata_entity_id>`:
 
-    *option*: `sso_service_url` => `http://rc-app.com/dag/saml2/idp/SSOService.php?spentityid=http://rc.local.pl/_admin/auth/duosecurity/saml-metadata`
-    # SSO (SingleSignOn) endpoint URL of the IdP. This can be used to initialize login
-    # E.g. https://duo-gateway.com/dag/saml2/idp/SSOService.php?spentityid=<metadata_entity_id>
+    .. note::
+      SSO (SingleSignOn) endpoint URL of the IdP. This can be used to initialize login, Known also as Login URL
+      E.g. http://rc-app.com/dag/saml2/idp/SSOService.php?spentityid=https://docker-dev/_admin/auth/duosecurity/saml-metadata
+
+SLO URL
+    `https://duo-gateway.com/dag/saml2/idp/SingleLogoutService.php?ReturnTo=<return_url>`:
 
-    *option*: `slo_service_url` => `http://rc-app.com/dag/saml2/idp/SingleLogoutService.php?ReturnTo=http://rc-app.com/dag/module.php/duosecurity/logout.php`
-    # SLO (SingleLogout) endpoint URL of the IdP.
-    # E.g. https://duo-gateway.com/dag/saml2/idp/SingleLogoutService.php?ReturnTo=http://duo-gateway.com/_admin/saml/sign-out-endpoint
+    .. note::
+      SLO (SingleLogout) endpoint URL of the IdP. , Known also as Logout URL
+      E.g. http://rc-app.com/dag/saml2/idp/SingleLogoutService.php?ReturnTo=https://docker-dev/_admin/auth/duosecurity/saml-sign-out-endpoint
 
-    *option*: `x509cert` => `<CERTIFICATE_STRING>`
-    # Identity provider public x509 certificate. It will be converted to single-line format without headers
+x509cert
+    `<CERTIFICATE_STRING>`:
 
-    *option*: `name_id_format` => `sha-1`
-    # The format that specifies how the NameID is sent to the service provider.
+    .. note::
+      Identity provider public x509 certificate. It will be converted to single-line format without headers.
+      Download the raw base64 encoded certificate from the Identity provider and paste it here.
+
+SAML Signature
+    `sha-256`:
+
+    .. note::
+      Type of Algorithm to use for verification of SAML signature on Identity provider side.
+
+SAML Digest
+    `sha-256`:
 
-    *option*: `signature_algo` => `sha-256`
-    # Type of Algorithm to use for verification of SAML signature on Identity provider side
+    .. note::
+      Type of Algorithm to use for verification of SAML digest on Identity provider side.
+
+Service Provider Cert Dir
+    `/etc/rhodecode/conf/saml_ssl/`:
 
-    *option*: `digest_algo` => `sha-256`
-    # Type of Algorithm to use for verification of SAML digest on Identity provider side
+    .. note::
+      Optional directory to store service provider certificate and private keys.
+      Expected certs for the SP should be stored in this folder as:
+
+      * sp.key     Private Key
+      * sp.crt     Public cert
+      * sp_new.crt Future Public cert
+
+      Also you can use other cert to sign the metadata of the SP using the:
 
-    *option*: `cert_dir` => `/etc/saml/`
-    # Optional directory to store service provider certificate and private keys.
-    # Expected certs for the SP should be stored in this folder as:
-    #  * sp.key     Private Key
-    #  * sp.crt     Public cert
-    #  * sp_new.crt Future Public cert
-    #
-    # Also you can use other cert to sign the metadata of the SP using the:
-    #  * metadata.key
-    #  * metadata.crt
+      * metadata.key
+      * metadata.crt
+
+Expected NameID Format
+    `nameid-format:emailAddress`:
+
+    .. note::
+      The format that specifies how the NameID is sent to the service provider.
+
+User ID Attribute
+    `PersonImmutableID`:
 
-    *option*: `user_id_attribute` => `PersonImmutableID`
-    # User ID Attribute name. This defines which attribute in SAML response will be used to link accounts via unique id.
-    # Ensure this is returned from DuoSecurity for example via duo_username
+    .. note::
+      User ID Attribute name. This defines which attribute in SAML response will be used to link accounts via unique id.
+      Ensure this is returned from DuoSecurity for example via duo_username.
+
+Username Attribute
+    `User.username`:
 
-    *option*: `username_attribute` => `User.username`
-    # Username Attribute name. This defines which attribute in SAML response will map to an username.
+    .. note::
+      Username Attribute name. This defines which attribute in SAML response will map to a username.
 
-    *option*: `email_attribute` => `User.email`
-    # Email Attribute name. This defines which attribute in SAML response will map to an email address.
+Email Attribute
+    `User.email`:
+
+    .. note::
+      Email Attribute name. This defines which attribute in SAML response will map to an email address.
+
 
 
 Below is example setup from DUO Administration page that can be used with above config.
 
 .. image:: ../images/saml-duosecurity-service-provider-example.png
    :alt: DUO Security SAML setup example
    :scale: 50 %
 
 
 Below is an example attribute mapping set for IDP provider required by the above config.
 
 
 .. image:: ../images/saml-duosecurity-attributes-example.png
    :alt: DUO Security SAML setup example
    :scale: 50 %
  No newline at end of file

docs/auth/auth-saml-generic.rst

@@ -1,19 +1,20 b''
 .. _config-saml-generic-ref:
 
 
 SAML 2.0 Authentication
 -----------------------
 
 
 **This plugin is available only in EE Edition.**
 
 RhodeCode Supports standard SAML 2.0 SSO for the web-application part.
 
 Please check for reference two example providers:
 
 .. toctree::
 
    auth-saml-duosecurity
    auth-saml-onelogin
+   auth-saml-azure
    auth-saml-bulk-enroll-users
 

docs/auth/auth-saml-onelogin.rst

@@ -1,106 +1,161 b''
 .. _config-saml-onelogin-ref:
 
 
 SAML 2.0 with One Login
 -----------------------
 
 **This plugin is available only in EE Edition.**
 
 |RCE| supports SAML 2.0 Authentication with OneLogin provider. This allows
 users to log-in to RhodeCode via SSO mechanism of external identity provider
 such as OneLogin. The login can be triggered either by the external IDP, or internally
 by clicking specific authentication button on the log-in page.
 
 
 Configuration steps
 ^^^^^^^^^^^^^^^^^^^
 
 To configure OneLogin SAML authentication, use the following steps:
 
 1. From the |RCE| interface, select
    :menuselection:`Admin --> Authentication`
 2. Activate the `OneLogin` plugin and select :guilabel:`Save`
 3. Go to newly available menu option called `OneLogin` on the left side.
 4. Check the `enabled` check box in the plugin configuration section,
    and fill in the required SAML information and :guilabel:`Save`, for more details,
    see :ref:`config-saml-onelogin`
 
 
 .. _config-saml-onelogin:
 
 
 Example SAML OneLogin configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Example configuration for SAML 2.0 with OneLogin provider::
+Example configuration for SAML 2.0 with OneLogin provider
+
+
+Enabled
+    `True`:
 
-    *option*: `enabled` => `True`
-    # Enable or disable this authentication plugin.
+    .. note::
+      Enable or disable this authentication plugin.
+
+
+Auth Cache TTL
+    `30`:
 
-    *option*: `cache_ttl` => `0`
-    # Amount of seconds to cache the authentication and permissions check response call for this plugin.
-    # Useful for expensive calls like LDAP to improve the performance of the system (0 means disabled).
+    .. note::
+      Amount of seconds to cache the authentication and permissions check response call for this plugin.
+      Useful for expensive calls like LDAP to improve the performance of the system (0 means disabled).
+
+Debug
+    `True`:
 
-    *option*: `debug` => `True`
-    # Enable or disable debug mode that shows SAML errors in the RhodeCode logs.
+    .. note::
+      Enable or disable debug mode that shows SAML errors in the RhodeCode logs.
+
+
+Auth button name
+    `Azure Entra ID`:
 
-    *option*: `entity_id` => `https://app.onelogin.com/saml/metadata/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`
-    # Identity Provider entity/metadata URI.
-    # E.g. https://app.onelogin.com/saml/metadata/<onelogin_connector_id>
+    .. note::
+      Alternative authentication display name. E.g AzureAuth, CorporateID etc.
+
+
+Entity ID
+    `https://app.onelogin.com/saml/metadata/<onelogin_connector_id>`:
+
+    .. note::
+      Identity Provider entity/metadata URI.
+      E.g. https://app.onelogin.com/saml/metadata/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+
+SSO URL
+    `https://app.onelogin.com/trust/saml2/http-post/sso/<onelogin_connector_id>`:
 
-    *option*: `sso_service_url` => `https://customer-domain.onelogin.com/trust/saml2/http-post/sso/xxxxxx`
-    # SSO (SingleSignOn) endpoint URL of the IdP. This can be used to initialize login
-    # E.g. https://app.onelogin.com/trust/saml2/http-post/sso/<onelogin_connector_id>
+    .. note::
+      SSO (SingleSignOn) endpoint URL of the IdP. This can be used to initialize login, Known also as Login URL
+      E.g.  https://app.onelogin.com/trust/saml2/http-post/sso/<onelogin_connector_id>
+
+SLO URL
+    `https://app.onelogin.com/trust/saml2/http-redirect/slo/<onelogin_connector_id>`:
 
-    *option*: `slo_service_url` => `https://customer-domain.onelogin.com/trust/saml2/http-redirect/slo/xxxxxx`
-    # SLO (SingleLogout) endpoint URL of the IdP.
-    # E.g. https://app.onelogin.com/trust/saml2/http-redirect/slo/<onelogin_connector_id>
+    .. note::
+      SLO (SingleLogout) endpoint URL of the IdP. , Known also as Logout URL
+      E.g. https://app.onelogin.com/trust/saml2/http-redirect/slo/<onelogin_connector_id>
 
-    *option*: `x509cert` => `<CERTIFICATE_STRING>`
-    # Identity provider public x509 certificate. It will be converted to single-line format without headers
+x509cert
+    `<CERTIFICATE_STRING>`:
 
-    *option*: `name_id_format` => `sha-1`
-    # The format that specifies how the NameID is sent to the service provider.
+    .. note::
+      Identity provider public x509 certificate. It will be converted to single-line format without headers.
+      Download the raw base64 encoded certificate from the Identity provider and paste it here.
+
+SAML Signature
+    `sha-256`:
+
+    .. note::
+      Type of Algorithm to use for verification of SAML signature on Identity provider side.
+
+SAML Digest
+    `sha-256`:
 
-    *option*: `signature_algo` => `sha-256`
-    # Type of Algorithm to use for verification of SAML signature on Identity provider side
+    .. note::
+      Type of Algorithm to use for verification of SAML digest on Identity provider side.
+
+Service Provider Cert Dir
+    `/etc/rhodecode/conf/saml_ssl/`:
 
-    *option*: `digest_algo` => `sha-256`
-    # Type of Algorithm to use for verification of SAML digest on Identity provider side
+    .. note::
+      Optional directory to store service provider certificate and private keys.
+      Expected certs for the SP should be stored in this folder as:
+
+      * sp.key     Private Key
+      * sp.crt     Public cert
+      * sp_new.crt Future Public cert
 
-    *option*: `cert_dir` => `/etc/saml/`
-    # Optional directory to store service provider certificate and private keys.
-    # Expected certs for the SP should be stored in this folder as:
-    #  * sp.key     Private Key
-    #  * sp.crt     Public cert
-    #  * sp_new.crt Future Public cert
-    #
-    # Also you can use other cert to sign the metadata of the SP using the:
-    #  * metadata.key
-    #  * metadata.crt
+      Also you can use other cert to sign the metadata of the SP using the:
+
+      * metadata.key
+      * metadata.crt
+
+Expected NameID Format
+    `nameid-format:emailAddress`:
+
+    .. note::
+      The format that specifies how the NameID is sent to the service provider.
+
+User ID Attribute
+    `PersonImmutableID`:
 
-    *option*: `user_id_attribute` => `PersonImmutableID`
-    # User ID Attribute name. This defines which attribute in SAML response will be used to link accounts via unique id.
-    # Ensure this is returned from OneLogin for example via Internal ID
+    .. note::
+      User ID Attribute name. This defines which attribute in SAML response will be used to link accounts via unique id.
+      Ensure this is returned from DuoSecurity for example via duo_username.
+
+Username Attribute
+    `User.username`:
 
-    *option*: `username_attribute` => `User.username`
-    # Username Attribute name. This defines which attribute in SAML response will map to an username.
+    .. note::
+      Username Attribute name. This defines which attribute in SAML response will map to a username.
 
-    *option*: `email_attribute` => `User.email`
-    # Email Attribute name. This defines which attribute in SAML response will map to an email address.
+Email Attribute
+    `User.email`:
+
+    .. note::
+      Email Attribute name. This defines which attribute in SAML response will map to an email address.
 
 
 
 Below is example setup that can be used with OneLogin SAML authentication that can be used with above config..
 
 .. image:: ../images/saml-onelogin-config-example.png
    :alt: OneLogin SAML setup example
    :scale: 50 %
 
 
 Below is an example attribute mapping set for IDP provider required by the above config.
 
 
 .. image:: ../images/saml-onelogin-attributes-example.png
    :alt: OneLogin SAML setup example
    :scale: 50 %
  No newline at end of file

docs/auth/auth.rst

@@ -1,34 +1,35 b''
 .. _authentication-ref:
 
 Authentication Options
 ======================
 
 |RCE| provides a built in authentication against its own database. This is
 implemented using ``RhodeCode Internal`` plugin. This plugin is enabled by default.
 Additionally, |RCE| provides a Pluggable Authentication System. This gives the
 administrator greater control over how users authenticate with the system.
 
 .. important::
 
   You can disable the built in |RCE| authentication plugin
   ``RhodeCode Internal`` and force all authentication to go
   through your authentication plugin of choice e.g LDAP only.
   However, if you do this, and your external authentication tools fails,
   accessing |RCE| will be blocked unless a fallback plugin is
   enabled via :file: rhodecode.ini
 
 
 |RCE| comes with the following user authentication management plugins:
 
 
 .. toctree::
 
     auth-token
     auth-ldap
     auth-ldap-groups
     auth-saml-generic
     auth-saml-onelogin
     auth-saml-duosecurity
+    auth-saml-azure
     auth-crowd
     auth-pam
     ssh-connection

docs/contributing/dev-setup.rst

@@ -1,243 +1,14 b''
 .. _dev-setup:
 
 ===================
  Development setup
 ===================
 
-
-RhodeCode Enterprise runs inside a Nix managed environment. This ensures build
-environment dependencies are correctly declared and installed during setup.
-It also enables atomic upgrades, rollbacks, and multiple instances of RhodeCode
-Enterprise running with isolation.
-
-To set up RhodeCode Enterprise inside the Nix environment, use the following steps:
-
-
-
-Setup Nix Package Manager
--------------------------
-
-To install the Nix Package Manager, please run::
-
-   $ curl https://releases.nixos.org/nix/nix-2.3.4/install | sh
-
-or go to https://nixos.org/nix/ and follow the installation instructions.
-Once this is correctly set up on your system, you should be able to use the
-following commands:
-
-* `nix-env`
-
-* `nix-shell`
-
-
-.. tip::
-
-   Update your channels frequently by running ``nix-channel --update``.
-
-.. note::
-
-   To uninstall nix run the following:
-
-    remove the . "$HOME/.nix-profile/etc/profile.d/nix.sh" line in your ~/.profile or ~/.bash_profile
-    rm -rf $HOME/{.nix-channels,.nix-defexpr,.nix-profile,.config/nixpkgs}
-    sudo rm -rf /nix
-
-Switch nix to the latest STABLE channel
----------------------------------------
-
-run::
-
-   nix-channel --add https://nixos.org/channels/nixos-20.03 nixpkgs
-
-Followed by::
-
-   nix-channel --update
-   nix-env -i nix-2.3.4
-
-
-Install required binaries
--------------------------
-
-We need some handy tools first.
-
-run::
-
-    nix-env -i nix-prefetch-hg
-    nix-env -i nix-prefetch-git
-
-
-Speed up JS build by installing PhantomJS
------------------------------------------
-
-PhantomJS will be downloaded each time nix-shell is invoked. To speed this by
-setting already downloaded version do this::
-
-    nix-env -i phantomjs-2.1.1
-
-    # and set nix bin path
-    export PATH=$PATH:~/.nix-profile/bin
-
-
-Clone the required repositories
--------------------------------
-
-After Nix is set up, clone the RhodeCode Enterprise Community Edition and
-RhodeCode VCSServer repositories into the same directory.
-RhodeCode currently is using Mercurial Version Control System, please make sure
-you have it installed before continuing.
-
-To obtain the required sources, use the following commands::
-
-    mkdir rhodecode-develop && cd rhodecode-develop
-    hg clone -u default https://code.rhodecode.com/rhodecode-enterprise-ce
-    hg clone -u default https://code.rhodecode.com/rhodecode-vcsserver
-
-.. note::
-
-   If you cannot clone the repository, please contact us via [e-mail]
-
-
-Install some required libraries
--------------------------------
-
-There are some required drivers and dev libraries that we need to install to
-test RhodeCode under different types of databases. For example in Ubuntu we
-need to install the following.
-
-required libraries::
-
-    # svn related
-    sudo apt-get install libapr1-dev libaprutil1-dev
-    sudo apt-get install libsvn-dev
-    # libcurl required too
-    sudo apt-get install libcurl4-openssl-dev
-    # mysql/pg server for development, optional
-    sudo apt-get install mysql-server libmysqlclient-dev
-    sudo apt-get install postgresql postgresql-contrib libpq-dev
-
-
-
-Enter the Development Shell
----------------------------
-
-The final step is to start the development shells. To do this, run the
-following command from inside the cloned repository::
-
-   # first, the vcsserver
-   cd ~/rhodecode-vcsserver
-   nix-shell
-
-   # then enterprise sources
-   cd ~/rhodecode-enterprise-ce
-   nix-shell
-
-.. note::
-
-   On the first run, this will take a while to download and optionally compile
-   a few things. The following runs will be faster. The development shell works
-   fine on both MacOS and Linux platforms.
-
-
-Create config.nix for development
----------------------------------
-
-In order to run proper tests and setup linking across projects, a config.nix
-file needs to be setup::
-
-    # create config
-    mkdir -p ~/.nixpkgs
-    touch ~/.nixpkgs/config.nix
-
-    # put the below content into the ~/.nixpkgs/config.nix file
-    # adjusts, the path to where you cloned your repositories.
-
-    {
-      rc = {
-       sources = {
-        rhodecode-vcsserver = "/home/dev/rhodecode-vcsserver";
-        rhodecode-enterprise-ce = "/home/dev/rhodecode-enterprise-ce";
-        rhodecode-enterprise-ee = "/home/dev/rhodecode-enterprise-ee";
-       };
-      };
-    }
-
-
-
-Creating a Development Configuration
-------------------------------------
-
-To create a development environment for RhodeCode Enterprise,
-use the following steps:
-
-1. Create a copy of vcsserver config:
-    `cp ~/rhodecode-vcsserver/configs/development.ini ~/rhodecode-vcsserver/configs/dev.ini`
-2. Create a copy of rhodocode config:
-    `cp ~/rhodecode-enterprise-ce/configs/development.ini ~/rhodecode-enterprise-ce/configs/dev.ini`
-3. Adjust the configuration settings to your needs if needed.
-
-.. note::
-
-  It is recommended to use the name `dev.ini` since it's included in .hgignore file.
-
-
-Setup the Development Database
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To create a development database, use the following example. This is a one
-time operation executed from the nix-shell of rhodecode-enterprise-ce sources ::
-
-    rc-setup-app dev.ini \
-        --user=admin --password=secret \
-        --email=[e-mail] \
-        --repos=~/my_dev_repos
-
-
-Compile CSS and JavaScript
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To use the application's frontend and prepare it for production deployment,
-you will need to compile the CSS and JavaScript with Grunt.
-This is easily done from within the nix-shell using the following command::
-
-    make web-build
-
-When developing new features you will need to recompile following any
-changes made to the CSS or JavaScript files when developing the code::
-
-    grunt watch
-
-This prepares the development (with comments/whitespace) versions of files.
-
-Start the Development Servers
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-From the rhodecode-vcsserver directory, start the development server in another
-nix-shell, using the following command::
-
-      pserve configs/dev.ini
-
-In the adjacent nix-shell which you created for your development server, you may
-now start CE with the following command::
-
-
-      pserve --reload configs/dev.ini
-
-.. note::
-
-  `--reload` flag will automatically reload the server when source file changes.
-
-
-Run the Environment Tests
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Please make sure that the tests are passing to verify that your environment is
-set up correctly. RhodeCode uses py.test to run tests.
-While your instance is running, start a new nix-shell and simply run
-``make test`` to run the basic test suite.
-
+Please refer to RCstack installed documentation for instructions on setting up dev environment:
+https://docs.rhodecode.com/rcstack/dev/dev-setup.html
 
 Need Help?
 ^^^^^^^^^^
 
 Join us on Slack via https://rhodecode.com/join or post questions in our
 Community Portal at https://community.rhodecode.com

docs/index.rst

@@ -1,92 +1,104 b''
 |RCE|
 =====
 
 |RCE| is a high-performance source code management and collaboration system.
 It enables you to develop projects securely behind the firewall while
 providing collaboration tools that work with |git|, |hg|,
 and |svn| |repos|. The user interface allows you to create, edit,
 and commit files and |repos| while managing their security permissions.
 
 |RCE| provides the following features:
 
 * Source code management.
 * Extended permissions management.
 * Integrated code collaboration tools.
 * Integrated code review and notifications.
 * Scalability provided by multi-node setup.
 * Fully programmable automation API.
 * Web-based hook management.
 * Native |svn| support.
 * Migration from existing databases.
 * |RCE| SDK.
 * Built-in analytics
 * Built in integrations including: Slack, Webhooks (used for Jenkins/TeamCity and other CIs), Jira, Redmine, Hipchat
 * Pluggable authentication system.
 * Support for AD, |LDAP|, Crowd, CAS, PAM.
 * Support for external authentication via Oauth Google, Github, Bitbucket, Twitter.
 * Debug modes of operation.
 * Private and public gists.
 * Gists with limited lifetimes and within instance only sharing.
 * Fully integrated code search function.
 * Always on SSL connectivity.
 
 .. only:: html
 
    Table of Contents
    -----------------
 
 .. toctree::
    :maxdepth: 1
+   :caption: Documentation directory
+
+   Back to documentation directory <https://docs.rhodecode.com/>
+
+.. toctree::
+   :maxdepth: 1
+   :caption: RhodeCode RCstack Documentation
+
+   RhodeCode RCstack Installer <https://docs.rhodecode.com/rcstack/>
+
+.. toctree::
+   :maxdepth: 1
    :caption: Admin Documentation
 
    install/quick-start
    install/install-database
    install/install-steps
    admin/system-overview
    admin/system-admin
    admin/user-admin
    admin/repo-admin
    admin/security-tips
    auth/auth
    issue-trackers/issue-trackers
    admin/lab-settings
 
 .. toctree::
    :maxdepth: 1
    :caption: Feature Documentation
 
    collaboration/collaboration
    collaboration/review-notifications
    collaboration/pull-requests
    code-review/code-review
    integrations/integrations
 
 .. toctree::
    :maxdepth: 1
    :caption: User Documentation
 
    usage/basic-usage
    tutorials/tutorials
 
 .. toctree::
    :maxdepth: 1
    :caption: Developer Documentation
 
    api/api
    tools/rhodecode-tools
    extensions/extensions-hooks
    contributing/contributing
 
 .. toctree::
    :maxdepth: 2
    :caption: RhodeCode rcstack Documentation
 
    RhodeCode Installer <https://docs.rhodecode.com/rcstack/>
 
 .. toctree::
    :maxdepth: 1
    :caption: About
 
    release-notes/release-notes
    known-issues/known-issues
    admin/glossary

docs/install/quick-start.rst

@@ -1,92 +1,93 b''
 .. _quick-start:
 
 Quick Start Installation Guide
 ==============================
 
 .. important::
 
     These are quick start instructions. To optimize your |RCE|,
     |RCC|, and |RCT| usage, read the more detailed instructions in our guides.
     For detailed installation instructions, see
     :ref:`RhodeCode rcstack Documentation <rcstack:installation>`
 
 
 
 To get |RCE| up and running, run through the below steps:
 
 1. Register to get the latest |RCC| installer instruction from `rhodecode.com/download`_.
    If you don't have an account, sign up at `rhodecode.com/register`_.
 
 2. Run the |RCS| installer and start init process.
    following example:
 
 .. code-block:: bash
 
     mkdir docker-rhodecode && cd docker-rhodecode
     curl -L -s -o rcstack https://dls.rhodecode.com/get-rcstack && chmod +x rcstack
 
    ./rcstack init
 
 
 .. important::
 
    We recommend running RhodeCode as a non-root user, such as `rhodecode`;
    this user must have a proper home directory and sudo permissions (to start Docker)
    Either log in as that user to install the software, or do it as root
    with `sudo -i -u rhodecode ./rcstack init`
 
 
 3. Follow instructions on |RCS| documentation pages
 
    :ref:`Quick install tutorial <rcstack:quick_installation>`
 
 4. Check stack status
 
 .. code-block:: bash
 
     ./rcstack status
 
 
 Output should look similar to this:
 
 .. code-block:: bash
 
     ---
     CONTAINER ID   IMAGE                                   STATUS                        NAMES                                 PORTS
     ef54fc528e3a   traefik:v2.9.5                          Up 2 hours             rc_cluster_router-traefik-1           0.0.0.0:80->80/tcp, :::80->80/tcp
     f3ea0539e8b0   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-rhodecode-1           0.0.0.0:10020->10020/tcp, :::10020->10020/tcp
     2be52ba58ffe   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-vcsserver-1
     7cd730ad3263   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-celery-1
     dfa231342c87   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-celery-beat-1
     d3d76ce2de96   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-sshd-1
     daaac329414b   rhodecode/rhodecode-ee:4.28.0           Up 2 hours (healthy)   rc_cluster_apps-svn-1
     7b8504fb9acb   nginx:1.23.2                            Up 2 hours (healthy)   rc_cluster_services-nginx-1           80/tcp
     7279c25feb6b   elasticsearch:6.8.23                    Up 2 hours (healthy)   rc_cluster_services-elasticsearch-1   9200/tcp, 9300/tcp
     19fb93587493   redis:7.0.5                             Up 2 hours (healthy)   rc_cluster_services-redis-1           6379/tcp
     fb77fb6496c6   channelstream/channelstream:0.7.1       Up 2 hours (healthy)   rc_cluster_services-channelstream-1   8000/tcp
     cb6c5c022f5b   postgres:14.6                           Up 2 hours (healthy)   rc_cluster_services-database-1        5432/tcp
 
+
 At this point you should be able to access:
 
 - RhodeCode instance at your domain entered, e.g http://rhodecode.local, the default access
   credentials are generated and stored inside .runtime.env.
   For example::
 
     RHODECODE_USER_NAME=admin
     RHODECODE_USER_PASS=super-secret-password
 
 
+
 .. note::
 
    Recommended post quick start install instructions:
 
    * Read the documentation
    * Carry out the :ref:`rhodecode-post-install-ref`
    * Set up :ref:`indexing-ref`
    * Familiarise yourself with the :ref:`rhodecode-admin-ref` section.
 
-.. _rhodecode.com/download/: https://rhodecode.com/download/
 .. _rhodecode.com: https://rhodecode.com/
 .. _rhodecode.com/register: https://rhodecode.com/register/
 .. _rhodecode.com/download: https://rhodecode.com/download/
 

docs/install/using-sqllite.rst

@@ -1,32 +1,21 b''
 .. _install-sqlite-database:
 
-SQLite
-------
+SQLite (Deprecated)
+-------------------
 
 .. important::
 
-    We do not recommend using SQLite in a large development environment
-    as it has an internal locking mechanism which can become a performance
-    bottleneck when there are more than 5 concurrent users.
+    As of 5.x, SQLite is no longer supported, we advise to migrate to MySQL or PostgreSQL.
 
-|RCE| installs SQLite as the default database if you do not specify another
-during installation. SQLite is suitable for small teams,
-projects with a low load, and evaluation purposes since it is built into
-|RCE| and does not require any additional database server.
-
-Using MySQL or PostgreSQL in an large setup gives you much greater
-performance, and while migration tools exist to move from one database type
-to another, it is better to get it right first time and to immediately use
-MySQL or PostgreSQL when you deploy |RCE| in a production environment.
 
 Migrating From SQLite to PostgreSQL
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 If you started working with SQLite and now need to migrate your database
 to PostgreSQL, you can contact [e-mail] for some help. We have a
 set of scripts that enable SQLite to PostgreSQL migration. These scripts have
 been tested, and work with PostgreSQL 9.1+.
 
 .. note::
 
     There are no SQLite to MySQL or MariaDB scripts available.

docs/known-issues/known-issues.rst

@@ -1,95 +1,117 b''
 .. _known-issues:
 
 Known Issues
 ============
 
 Windows Upload
 --------------
 
 There can be an issue with uploading files from web interface on Windows,
 and afterwards users cannot properly clone or synchronize with the repository.
 
 Early testing shows that often uploading files via HTML forms on Windows
 includes the full path of the file being uploaded and not the name of the file.
 
 Old Format of Git Repositories
 ------------------------------
 
 There is an issue when trying to import old |git| format |repos| into recent
 versions of |RCE|. This issue can occur when importing from external |git|
 repositories or from older versions of |RCE| (<=2.2.7).
 
 To convert the old version into a current version, clone the old
 |repo| into a local machine using a recent |git| client, then push it to a new
 |repo| inside |RCE|.
 
 
 VCS Server Memory Consumption
 -----------------------------
 
 The VCS Server cache grows without limits if not configured correctly. This
 applies to |RCE| versions prior to the 3.3.2 releases, as 3.3.2
 shipped with the optimal configuration as default. See the
 :ref:`vcs-server-maintain` section for details.
 
 To fix this issue, upgrade to |RCE| 3.3.2 or greater, and if you discover
 memory consumption issues check the VCS Server settings.
 
 Newer Operating system locales
 ------------------------------
 
 |RCC| has a know problem with locales, due to changes in glibc 2.27+ which affects
 the local-archive format, which is now incompatible with our used glibc 2.26.
 
 Mostly affected are:
+
 - Fedora 23+
 - Ubuntu 18.04
 - CentOS / RHEL 8
 
 To work around this problem, you need set path to ``$LOCAL_ARCHIVE`` to the
 locale package in older pre glibc 2.27 format, or set `LC_ALL=C` in your enviroment.
 
 To use the pre 2.27 locale-archive fix follow these steps:
 
 1. Download the pre 2.27 locale-archive package
 
 .. code-block:: bash
 
     wget https://dls.rhodecode.com/assets/locale-archive
 
 
 2. Point ``$LOCAL_ARCHIVE`` to the locale package.
 
 .. code-block:: bash
 
     $ export LOCALE_ARCHIVE=/home/USER/locale-archive  # change to your path
 
 This should be added *both* in `enviroment` variable of `~/.rccontrol/supervisor/supervisord.ini`
 e.g
 
 ```
 [supervisord]
 environment = HOME=/home/user/rhodecode,LOCALE_ARCHIVE=/YOUR-PATH/locale-archive`
 ```
 
 and in user .bashrc/.zshrc etc, or via a startup script that
 runs `rccontrol self-init`
 
 If you happen to be running |RCC| from systemd, use the following
 example to pass the correct locale information on boot.
 
 .. code-block:: ini
 
     [Unit]
     Description=Rhodecode
     After=network.target
 
     [Service]
     Type=forking
     User=scm
     Environment="LOCALE_ARCHIVE=/YOUR-PATH/locale-archive"
     ExecStart=/YOUR-PATH/.rccontrol-profile/bin/rccontrol-self-init
 
     [Install]
     WantedBy=multi-user.target
 
+
+Merge stucks in "merging" status
+--------------------------------
+
+Similar issues:
+
+- Pull Request duplicated and/or stucks in "creating" status.
+
+Mostly affected are:
+
+- Kubernetes AWS EKS setup with NFS as shared storage
+- AWS EFS as shared storage
+
+Workaround:
+
+1. Manually clear the repo cache via UI: 
+:menuselection:`Repository Settings --> Caches --> Invalidate repository cache`
+
+1. Open problematic PR and reset status to "created"
+
+Now you can merge PR normally

docs/release-notes/release-notes-5.1.0.rst

@@ -1,59 +1,58 b''
 |RCE| 5.1.0 |RNS|
 -----------------
 
 Release Date
 ^^^^^^^^^^^^
 
 - 2024-07-18
 
 
 New Features
 ^^^^^^^^^^^^
 
-- We've introduced 2FA for users. Now alongside the external auth 2fa support RhodeCode allows to enable 2FA for users
+- We've introduced 2FA for users. Now alongside the external auth 2FA support RhodeCode allows to enable 2FA for users.
   2FA options will be available for each user individually, or enforced via authentication plugins like ldap, or internal.
 - Email based log-in. RhodeCode now allows to log-in using email as well as username for main authentication type.
 - Ability to replace a file using web UI. Now one can replace an existing file from the web-ui.
 - GIT LFS Sync automation. Remote push/pull commands now can also sync GIT LFS objects.
-- Added ability to remove or close branches from the web ui
-- Added ability to delete a branch automatically after merging PR for git repositories
-- Added support for S3 based archive_cache based that allows storing cached archives in S3 compatible object store.
+- Added ability to remove or close branches from the web ui.
+- Added ability to delete a branch automatically after merging PR for git repositories.
+- Added support for S3 based archive_cache that allows storing cached archives in S3 compatible object store.
 
 
 General
 ^^^^^^^
 
-- Upgraded all dependency libraries to their latest available versions
+- Upgraded all dependency libraries to their latest available versions.
 - Repository storage is no longer controlled via DB settings, but .ini file. This allows easier automated deployments.
 - Bumped mercurial to 6.7.4
 - Mercurial: enable httppostarguments for better support of large repositories with lots of heads.
 - Added explicit db-migrate step to update hooks for 5.X release.
 
 
 Security
 ^^^^^^^^
 
 
 
 Performance
 ^^^^^^^^^^^
 
 - Introduced a full rewrite of ssh backend for performance. The result is 2-5x speed improvement for operation with ssh.
-  enable new ssh wrapper by setting: `ssh.wrapper_cmd = /home/rhodecode/venv/bin/rc-ssh-wrapper-v2`
-- Introduced a new hooks subsystem that is more scalable and faster, enable it by settings: `vcs.hooks.protocol = celery`
+  Enable new ssh wrapper by setting: `ssh.wrapper_cmd = /home/rhodecode/venv/bin/rc-ssh-wrapper-v2`
+- Introduced a new hooks subsystem that is more scalable and faster, enable it by setting: `vcs.hooks.protocol = celery`
 
 
 Fixes
 ^^^^^
 
-- Archives: Zip archive download breaks when a gitmodules file is present
-- Branch permissions: fixed bug preventing to specify own rules from 4.X install
-- SVN: refactored svn events, thus fixing support for it in dockerized env
-- Fixed empty server url in PR link after push from cli
+- Archives: Zip archive download breaks when a gitmodules file is present.
+- Branch permissions: fixed bug preventing to specify own rules from 4.X install.
+- SVN: refactored svn events, thus fixing support for it in dockerized environment.
+- Fixed empty server url in PR link after push from cli.
 
 
 Upgrade notes
 ^^^^^^^^^^^^^
 
-- RhodeCode 5.1.0 is a mayor feature release after big 5.0.0 python3 migration. Happy to ship a first time feature
-  rich release
+- RhodeCode 5.1.0 is a major feature release after big 5.0.0 python3 migration. Happy to ship a first time feature-rich release.

docs/release-notes/release-notes.rst

@@ -1,173 +1,175 b''
 .. _rhodecode-release-notes-ref:
 
 Release Notes
 =============
 
 |RCE| 5.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
-
+   release-notes-5.2.0.rst
+   release-notes-5.1.2.rst
+   release-notes-5.1.1.rst
    release-notes-5.1.0.rst
    release-notes-5.0.3.rst
    release-notes-5.0.2.rst
    release-notes-5.0.1.rst
    release-notes-5.0.0.rst
 
 
 |RCE| 4.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-4.27.1.rst
    release-notes-4.27.0.rst
    release-notes-4.26.0.rst
    release-notes-4.25.2.rst
    release-notes-4.25.1.rst
    release-notes-4.25.0.rst
    release-notes-4.24.1.rst
    release-notes-4.24.0.rst
    release-notes-4.23.2.rst
    release-notes-4.23.1.rst
    release-notes-4.23.0.rst
    release-notes-4.22.0.rst
    release-notes-4.21.0.rst
    release-notes-4.20.1.rst
    release-notes-4.20.0.rst
    release-notes-4.19.3.rst
    release-notes-4.19.2.rst
    release-notes-4.19.1.rst
    release-notes-4.19.0.rst
    release-notes-4.18.3.rst
    release-notes-4.18.2.rst
    release-notes-4.18.1.rst
    release-notes-4.18.0.rst
    release-notes-4.17.4.rst
    release-notes-4.17.3.rst
    release-notes-4.17.2.rst
    release-notes-4.17.1.rst
    release-notes-4.17.0.rst
    release-notes-4.16.2.rst
    release-notes-4.16.1.rst
    release-notes-4.16.0.rst
    release-notes-4.15.2.rst
    release-notes-4.15.1.rst
    release-notes-4.15.0.rst
    release-notes-4.14.1.rst
    release-notes-4.14.0.rst
    release-notes-4.13.3.rst
    release-notes-4.13.2.rst
    release-notes-4.13.1.rst
    release-notes-4.13.0.rst
    release-notes-4.12.4.rst
    release-notes-4.12.3.rst
    release-notes-4.12.2.rst
    release-notes-4.12.1.rst
    release-notes-4.12.0.rst
    release-notes-4.11.6.rst
    release-notes-4.11.5.rst
    release-notes-4.11.4.rst
    release-notes-4.11.3.rst
    release-notes-4.11.2.rst
    release-notes-4.11.1.rst
    release-notes-4.11.0.rst
    release-notes-4.10.6.rst
    release-notes-4.10.5.rst
    release-notes-4.10.4.rst
    release-notes-4.10.3.rst
    release-notes-4.10.2.rst
    release-notes-4.10.1.rst
    release-notes-4.10.0.rst
    release-notes-4.9.1.rst
    release-notes-4.9.0.rst
    release-notes-4.8.0.rst
    release-notes-4.7.2.rst
    release-notes-4.7.1.rst
    release-notes-4.7.0.rst
    release-notes-4.6.1.rst
    release-notes-4.6.0.rst
    release-notes-4.5.2.rst
    release-notes-4.5.1.rst
    release-notes-4.5.0.rst
    release-notes-4.4.2.rst
    release-notes-4.4.1.rst
    release-notes-4.4.0.rst
    release-notes-4.3.1.rst
    release-notes-4.3.0.rst
    release-notes-4.2.1.rst
    release-notes-4.2.0.rst
    release-notes-4.1.2.rst
    release-notes-4.1.1.rst
    release-notes-4.1.0.rst
    release-notes-4.0.1.rst
    release-notes-4.0.0.rst
 
 |RCE| 3.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-3.8.4.rst
    release-notes-3.8.3.rst
    release-notes-3.8.2.rst
    release-notes-3.8.1.rst
    release-notes-3.8.0.rst
    release-notes-3.7.1.rst
    release-notes-3.7.0.rst
    release-notes-3.6.1.rst
    release-notes-3.6.0.rst
    release-notes-3.5.2.rst
    release-notes-3.5.1.rst
    release-notes-3.5.0.rst
    release-notes-3.4.1.rst
    release-notes-3.4.0.rst
    release-notes-3.3.4.rst
    release-notes-3.3.3.rst
    release-notes-3.3.2.rst
    release-notes-3.3.1.rst
    release-notes-3.3.0.rst
    release-notes-3.2.3.rst
    release-notes-3.2.2.rst
    release-notes-3.2.1.rst
    release-notes-3.2.0.rst
    release-notes-3.1.1.rst
    release-notes-3.1.0.rst
    release-notes-3.0.2.rst
    release-notes-3.0.1.rst
    release-notes-3.0.0.rst
 
 |RCE| 2.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-2.2.8.rst
    release-notes-2.2.7.rst
    release-notes-2.2.6.rst
    release-notes-2.2.5.rst
    release-notes-2.2.4.rst
    release-notes-2.2.3.rst
    release-notes-2.2.2.rst
    release-notes-2.2.1.rst
    release-notes-2.2.0.rst
    release-notes-2.1.0.rst
    release-notes-2.0.2.rst
    release-notes-2.0.1.rst
    release-notes-2.0.0.rst
 
 |RCE| 1.x Versions
 ------------------
 
 .. toctree::
    :maxdepth: 1
 
    release-notes-1.7.2.rst
    release-notes-1.7.1.rst
    release-notes-1.7.0.rst
    release-notes-1.6.0.rst

docs/requirements_docs.txt

@@ -1,11 +1,11 b''
 sphinx==7.2.6
 
 furo==2023.9.10
 sphinx-press-theme==0.8.0
 sphinx-rtd-theme==1.3.0
 
-pygments==2.16.1
+pygments==2.18.0
 
 docutils<0.19
 markupsafe==2.1.3
 jinja2==3.1.2

requirements.txt

@@ -1,313 +1,299 b''
 # deps, generated via pipdeptree --exclude setuptools,wheel,pipdeptree,pip -f | tr '[:upper:]' '[:lower:]'
 
 alembic==1.13.1
   mako==1.2.4
     markupsafe==2.1.2
   sqlalchemy==1.4.52
     greenlet==3.0.3
-  typing_extensions==4.9.0
+  typing_extensions==4.12.2
 async-timeout==4.0.3
 babel==2.12.1
 beaker==1.12.1
 celery==5.3.6
   billiard==4.2.0
   click==8.1.3
   click-didyoumean==0.3.0
     click==8.1.3
   click-plugins==1.1.1
     click==8.1.3
   click-repl==0.2.0
     click==8.1.3
-    prompt-toolkit==3.0.38
-      wcwidth==0.2.6
+    prompt_toolkit==3.0.47
+      wcwidth==0.2.13
     six==1.16.0
   kombu==5.3.5
     amqp==5.2.0
       vine==5.1.0
     vine==5.1.0
   python-dateutil==2.8.2
     six==1.16.0
   tzdata==2024.1
   vine==5.1.0
 channelstream==0.7.1
   gevent==24.2.1
     greenlet==3.0.3
     zope.event==5.0.0
-    zope.interface==6.3.0
+    zope.interface==7.0.3
   itsdangerous==1.1.0
   marshmallow==2.18.0
   pyramid==2.0.2
     hupper==1.12
     plaster==1.1.2
     plaster-pastedeploy==1.0.1
       pastedeploy==3.1.0
       plaster==1.1.2
     translationstring==1.4
     venusian==3.0.0
     webob==1.8.7
     zope.deprecation==5.0.0
-    zope.interface==6.3.0
+    zope.interface==7.0.3
   pyramid-jinja2==2.10
     jinja2==3.1.2
       markupsafe==2.1.2
     markupsafe==2.1.2
     pyramid==2.0.2
       hupper==1.12
       plaster==1.1.2
       plaster-pastedeploy==1.0.1
         pastedeploy==3.1.0
         plaster==1.1.2
       translationstring==1.4
       venusian==3.0.0
       webob==1.8.7
       zope.deprecation==5.0.0
-      zope.interface==6.3.0
+      zope.interface==7.0.3
     zope.deprecation==5.0.0
   python-dateutil==2.8.2
     six==1.16.0
   requests==2.28.2
     certifi==2022.12.7
     charset-normalizer==3.1.0
     idna==3.4
     urllib3==1.26.14
   ws4py==0.5.1
 deform==2.0.15
   chameleon==3.10.2
   colander==2.0
     iso8601==1.1.0
     translationstring==1.4
   iso8601==1.1.0
   peppercorn==0.6
   translationstring==1.4
   zope.deprecation==5.0.0
 docutils==0.19
 dogpile.cache==1.3.3
   decorator==5.1.1
   stevedore==5.1.0
     pbr==5.11.1
 formencode==2.1.0
   six==1.16.0
-fsspec==2024.6.0
-gunicorn==21.2.0
-  packaging==24.0
+fsspec==2024.9.0
+gunicorn==23.0.0
+  packaging==24.1
 gevent==24.2.1
   greenlet==3.0.3
   zope.event==5.0.0
-  zope.interface==6.3.0
-ipython==8.14.0
-  backcall==0.2.0
+  zope.interface==7.0.3
+ipython==8.26.0
   decorator==5.1.1
-  jedi==0.19.0
-    parso==0.8.3
-  matplotlib-inline==0.1.6
-    traitlets==5.9.0
-  pexpect==4.8.0
+  jedi==0.19.1
+    parso==0.8.4
+  matplotlib-inline==0.1.7
+    traitlets==5.14.3
+  pexpect==4.9.0
     ptyprocess==0.7.0
-  pickleshare==0.7.5
-  prompt-toolkit==3.0.38
-    wcwidth==0.2.6
-  pygments==2.15.1
-  stack-data==0.6.2
-    asttokens==2.2.1
+  prompt_toolkit==3.0.47
+    wcwidth==0.2.13
+  pygments==2.18.0
+  stack-data==0.6.3
+    asttokens==2.4.1
       six==1.16.0
-    executing==1.2.0
-    pure-eval==0.2.2
-  traitlets==5.9.0
+    executing==2.0.1
+    pure_eval==0.2.3
+  traitlets==5.14.3
+  typing_extensions==4.12.2
 markdown==3.4.3
 msgpack==1.0.8
 mysqlclient==2.1.1
 nbconvert==7.7.3
   beautifulsoup4==4.12.3
     soupsieve==2.5
   bleach==6.1.0
     six==1.16.0
     webencodings==0.5.1
   defusedxml==0.7.1
   jinja2==3.1.2
     markupsafe==2.1.2
   jupyter_core==5.3.1
     platformdirs==3.10.0
-    traitlets==5.9.0
+    traitlets==5.14.3
   jupyterlab-pygments==0.2.2
   markupsafe==2.1.2
   mistune==2.0.5
   nbclient==0.8.0
     jupyter_client==8.3.0
       jupyter_core==5.3.1
         platformdirs==3.10.0
-        traitlets==5.9.0
+        traitlets==5.14.3
       python-dateutil==2.8.2
         six==1.16.0
       pyzmq==25.0.0
       tornado==6.2
-      traitlets==5.9.0
+      traitlets==5.14.3
     jupyter_core==5.3.1
       platformdirs==3.10.0
-      traitlets==5.9.0
+      traitlets==5.14.3
     nbformat==5.9.2
       fastjsonschema==2.18.0
       jsonschema==4.18.6
         attrs==22.2.0
         pyrsistent==0.19.3
       jupyter_core==5.3.1
         platformdirs==3.10.0
-        traitlets==5.9.0
-      traitlets==5.9.0
-    traitlets==5.9.0
+        traitlets==5.14.3
+      traitlets==5.14.3
+    traitlets==5.14.3
   nbformat==5.9.2
     fastjsonschema==2.18.0
     jsonschema==4.18.6
       attrs==22.2.0
       pyrsistent==0.19.3
     jupyter_core==5.3.1
       platformdirs==3.10.0
-      traitlets==5.9.0
-    traitlets==5.9.0
+      traitlets==5.14.3
+    traitlets==5.14.3
   pandocfilters==1.5.0
-  pygments==2.15.1
+  pygments==2.18.0
   tinycss2==1.2.1
     webencodings==0.5.1
-  traitlets==5.9.0
-orjson==3.10.3
+  traitlets==5.14.3
+orjson==3.10.7
 paste==3.10.1
 premailer==3.10.0
   cachetools==5.3.3
   cssselect==1.2.0
   cssutils==2.6.0
-  lxml==4.9.3
+  lxml==5.3.0
   requests==2.28.2
     certifi==2022.12.7
     charset-normalizer==3.1.0
     idna==3.4
     urllib3==1.26.14
 psutil==5.9.8
 psycopg2==2.9.9
 py-bcrypt==0.4
 pycmarkgfm==1.2.0
   cffi==1.16.0
     pycparser==2.21
 pycryptodome==3.17
 pycurl==7.45.3
 pymysql==1.0.3
 pyotp==2.8.0
 pyparsing==3.1.1
-pyramid-debugtoolbar==4.12.1
-  pygments==2.15.1
+pyramid-mailer==0.15.1
   pyramid==2.0.2
     hupper==1.12
     plaster==1.1.2
     plaster-pastedeploy==1.0.1
       pastedeploy==3.1.0
       plaster==1.1.2
     translationstring==1.4
     venusian==3.0.0
     webob==1.8.7
     zope.deprecation==5.0.0
-    zope.interface==6.3.0
+    zope.interface==7.0.3
+  repoze.sendmail==4.4.1
+    transaction==5.0.0
+      zope.interface==7.0.3
+    zope.interface==7.0.3
+  transaction==5.0.0
+    zope.interface==7.0.3
   pyramid-mako==1.1.0
     mako==1.2.4
       markupsafe==2.1.2
     pyramid==2.0.2
       hupper==1.12
       plaster==1.1.2
       plaster-pastedeploy==1.0.1
         pastedeploy==3.1.0
         plaster==1.1.2
       translationstring==1.4
       venusian==3.0.0
       webob==1.8.7
       zope.deprecation==5.0.0
-      zope.interface==6.3.0
-pyramid-mailer==0.15.1
-  pyramid==2.0.2
-    hupper==1.12
-    plaster==1.1.2
-    plaster-pastedeploy==1.0.1
-      pastedeploy==3.1.0
-      plaster==1.1.2
-    translationstring==1.4
-    venusian==3.0.0
-    webob==1.8.7
-    zope.deprecation==5.0.0
-    zope.interface==6.3.0
-  repoze.sendmail==4.4.1
-    transaction==3.1.0
-      zope.interface==6.3.0
-    zope.interface==6.3.0
-  transaction==3.1.0
-    zope.interface==6.3.0
+    zope.interface==7.0.3
 python-ldap==3.4.3
   pyasn1==0.4.8
   pyasn1-modules==0.2.8
     pyasn1==0.4.8
 python-memcached==1.59
   six==1.16.0
 python-pam==2.0.2
-python3-saml==1.15.0
+python3-saml==1.16.0
   isodate==0.6.1
     six==1.16.0
-  lxml==4.9.3
-  xmlsec==1.3.13
-    lxml==4.9.3
+  lxml==5.3.0
+  xmlsec==1.3.14
+    lxml==5.3.0
 pyyaml==6.0.1
-redis==5.0.4
+redis==5.1.0
   async-timeout==4.0.3
 regex==2022.10.31
 routes==2.5.1
   repoze.lru==0.7
   six==1.16.0
-s3fs==2024.6.0
+s3fs==2024.9.0
   aiobotocore==2.13.0
     aiohttp==3.9.5
       aiosignal==1.3.1
         frozenlist==1.4.1
       attrs==22.2.0
       frozenlist==1.4.1
       multidict==6.0.5
       yarl==1.9.4
         idna==3.4
         multidict==6.0.5
     aioitertools==0.11.0
     botocore==1.34.106
       jmespath==1.0.1
       python-dateutil==2.8.2
         six==1.16.0
       urllib3==1.26.14
     wrapt==1.16.0
   aiohttp==3.9.5
     aiosignal==1.3.1
       frozenlist==1.4.1
     attrs==22.2.0
     frozenlist==1.4.1
     multidict==6.0.5
     yarl==1.9.4
       idna==3.4
       multidict==6.0.5
-  fsspec==2024.6.0
+  fsspec==2024.9.0
 simplejson==3.19.2
 sshpubkeys==3.3.1
   cryptography==40.0.2
     cffi==1.16.0
       pycparser==2.21
   ecdsa==0.18.0
     six==1.16.0
 sqlalchemy==1.4.52
   greenlet==3.0.3
-  typing_extensions==4.9.0
+  typing_extensions==4.12.2
 supervisor==4.2.5
 tzlocal==4.3
   pytz-deprecation-shim==0.1.0.post0
     tzdata==2024.1
 tempita==0.5.2
 unidecode==1.3.6
 urlobject==2.4.3
 waitress==3.0.0
 webhelpers2==2.1
   markupsafe==2.1.2
   six==1.16.0
 whoosh==2.7.4
 zope.cachedescriptors==5.0.0
 qrcode==7.4.2
 
 ## uncomment to add the debug libraries
 #-r requirements_debug.txt

requirements_debug.txt

@@ -1,28 +1,29 b''
 ## special libraries we could extend the requirements.txt file with to add some
 ## custom libraries usefull for debug and memory tracing
 
 objgraph
 memory-profiler
 pympler
 
 ## debug
 ipdb
 ipython
 rich
+pyramid-debugtoolbar
 
 # format
 flake8
 ruff
 
 pipdeptree==2.7.1
 invoke==2.0.0
 bumpversion==0.6.0
 bump2version==1.0.1
 
 docutils-stubs
 types-redis
 types-requests==2.31.0.6
 types-sqlalchemy
 types-psutil
 types-pycurl
 types-ujson

requirements_test.txt

@@ -1,48 +1,48 b''
 # test related requirements
 mock==5.1.0
 pytest-cov==4.1.0
   coverage==7.4.3
   pytest==8.1.1
     iniconfig==2.0.0
-    packaging==24.0
+    packaging==24.1
     pluggy==1.4.0
 pytest-env==1.1.3
   pytest==8.1.1
     iniconfig==2.0.0
-    packaging==24.0
+    packaging==24.1
     pluggy==1.4.0
 pytest-profiling==1.7.0
   gprof2dot==2022.7.29
   pytest==8.1.1
     iniconfig==2.0.0
-    packaging==24.0
+    packaging==24.1
     pluggy==1.4.0
   six==1.16.0
 pytest-rerunfailures==13.0
-  packaging==24.0
+  packaging==24.1
   pytest==8.1.1
     iniconfig==2.0.0
-    packaging==24.0
+    packaging==24.1
     pluggy==1.4.0
 pytest-runner==6.0.1
 pytest-sugar==1.0.0
-  packaging==24.0
+  packaging==24.1
   pytest==8.1.1
     iniconfig==2.0.0
-    packaging==24.0
+    packaging==24.1
     pluggy==1.4.0
   termcolor==2.4.0
 pytest-timeout==2.3.1
   pytest==8.1.1
     iniconfig==2.0.0
-    packaging==24.0
+    packaging==24.1
     pluggy==1.4.0
 webtest==3.0.0
   beautifulsoup4==4.12.3
     soupsieve==2.5
   waitress==3.0.0
   webob==1.8.7
 
 # RhodeCode test-data
 rc_testdata @ https://code.rhodecode.com/upstream/rc-testdata-dist/raw/77378e9097f700b4c1b9391b56199fe63566b5c9/rc_testdata-0.11.0.tar.gz#egg=rc_testdata
 rc_testdata==0.11.0

rhodecode/VERSION

@@ -1,1 +1,1 b''
-5.1.2
  No newline at end of file
+5.2.0

rhodecode/api/__init__.py

@@ -1,581 +1,581 b''
 # Copyright (C) 2011-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import itertools
 import logging
 import sys
 import fnmatch
 
 import decorator
 import venusian
 from collections import OrderedDict
 
 from pyramid.exceptions import ConfigurationError
 from pyramid.renderers import render
 from pyramid.response import Response
 from pyramid.httpexceptions import HTTPNotFound
 
 from rhodecode.api.exc import (
     JSONRPCBaseError, JSONRPCError, JSONRPCForbidden, JSONRPCValidationError)
 from rhodecode.apps._base import TemplateArgs
 from rhodecode.lib.auth import AuthUser
 from rhodecode.lib.base import get_ip_addr, attach_context_attributes
 from rhodecode.lib.exc_tracking import store_exception
 from rhodecode.lib import ext_json
 from rhodecode.lib.utils2 import safe_str
 from rhodecode.lib.plugins.utils import get_plugin_settings
 from rhodecode.model.db import User, UserApiKeys
+from rhodecode.config.patches import inspect_getargspec
 
 log = logging.getLogger(__name__)
 
 DEFAULT_RENDERER = 'jsonrpc_renderer'
 DEFAULT_URL = '/_admin/api'
 SERVICE_API_IDENTIFIER = 'service_'
 
 
 def find_methods(jsonrpc_methods, pattern):
     matches = OrderedDict()
     if not isinstance(pattern, (list, tuple)):
         pattern = [pattern]
 
     for single_pattern in pattern:
         for method_name, method in filter(
                 lambda x: not x[0].startswith(SERVICE_API_IDENTIFIER), jsonrpc_methods.items()
         ):
             if fnmatch.fnmatch(method_name, single_pattern):
                 matches[method_name] = method
     return matches
 
 
 class ExtJsonRenderer(object):
     """
     Custom renderer that makes use of our ext_json lib
 
     """
 
     def __init__(self):
         self.serializer = ext_json.formatted_json
 
     def __call__(self, info):
         """ Returns a plain JSON-encoded string with content-type
         ``application/json``. The content-type may be overridden by
         setting ``request.response.content_type``."""
 
         def _render(value, system):
             request = system.get('request')
             if request is not None:
                 response = request.response
                 ct = response.content_type
                 if ct == response.default_content_type:
                     response.content_type = 'application/json'
 
             return self.serializer(value)
 
         return _render
 
 
 def jsonrpc_response(request, result):
     rpc_id = getattr(request, 'rpc_id', None)
 
     ret_value = ''
     if rpc_id:
         ret_value = {'id': rpc_id, 'result': result, 'error': None}
 
         # fetch deprecation warnings, and store it inside results
         deprecation = getattr(request, 'rpc_deprecation', None)
         if deprecation:
             ret_value['DEPRECATION_WARNING'] = deprecation
 
     raw_body = render(DEFAULT_RENDERER, ret_value, request=request)
     content_type = 'application/json'
     content_type_header = 'Content-Type'
     headers = {
         content_type_header: content_type
     }
     return Response(
         body=raw_body,
         content_type=content_type,
         headerlist=[(k, v) for k, v in headers.items()]
     )
 
 
 def jsonrpc_error(request, message, retid=None, code: int | None = None, headers: dict | None = None):
     """
     Generate a Response object with a JSON-RPC error body
     """
     headers = headers or {}
     content_type = 'application/json'
     content_type_header = 'Content-Type'
     if content_type_header not in headers:
         headers[content_type_header] = content_type
 
     err_dict = {'id': retid, 'result': None, 'error': message}
     raw_body = render(DEFAULT_RENDERER, err_dict, request=request)
 
     return Response(
         body=raw_body,
         status=code,
         content_type=content_type,
         headerlist=[(k, v) for k, v in headers.items()]
     )
 
 
 def exception_view(exc, request):
     rpc_id = getattr(request, 'rpc_id', None)
 
     if isinstance(exc, JSONRPCError):
         fault_message = safe_str(exc)
         log.debug('json-rpc error rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, JSONRPCValidationError):
         colander_exc = exc.colander_exception
         # TODO(marcink): think maybe of nicer way to serialize errors ?
         fault_message = colander_exc.asdict()
         log.debug('json-rpc colander error rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, JSONRPCForbidden):
         fault_message = 'Access was denied to this resource.'
         log.warning('json-rpc forbidden call rpc_id:%s "%s"', rpc_id, fault_message)
     elif isinstance(exc, HTTPNotFound):
         method = request.rpc_method
         log.debug('json-rpc method `%s` not found in list of '
                   'api calls: %s, rpc_id:%s',
                   method, list(request.registry.jsonrpc_methods.keys()), rpc_id)
 
         similar = 'none'
         try:
             similar_paterns = [f'*{x}*' for x in method.split('_')]
             similar_found = find_methods(
                 request.registry.jsonrpc_methods, similar_paterns)
             similar = ', '.join(similar_found.keys()) or similar
         except Exception:
             # make the whole above block safe
             pass
 
         fault_message = f"No such method: {method}. Similar methods: {similar}"
     else:
         fault_message = 'undefined error'
         exc_info = exc.exc_info()
         store_exception(id(exc_info), exc_info, prefix='rhodecode-api')
 
         statsd = request.registry.statsd
         if statsd:
             exc_type = f"{exc.__class__.__module__}.{exc.__class__.__name__}"
             statsd.incr('rhodecode_exception_total',
                         tags=["exc_source:api", f"type:{exc_type}"])
 
     return jsonrpc_error(request, fault_message, rpc_id)
 
 
 def request_view(request):
     """
     Main request handling method. It handles all logic to call a specific
     exposed method
     """
     # cython compatible inspect
-    from rhodecode.config.patches import inspect_getargspec
     inspect = inspect_getargspec()
 
     # check if we can find this session using api_key, get_by_auth_token
     # search not expired tokens only
     try:
         if not request.rpc_method.startswith(SERVICE_API_IDENTIFIER):
             api_user = User.get_by_auth_token(request.rpc_api_key)
 
             if api_user is None:
                 return jsonrpc_error(
                     request, retid=request.rpc_id, message='Invalid API KEY')
 
             if not api_user.active:
                 return jsonrpc_error(
                     request, retid=request.rpc_id,
                     message='Request from this user not allowed')
 
             # check if we are allowed to use this IP
             auth_u = AuthUser(
                 api_user.user_id, request.rpc_api_key, ip_addr=request.rpc_ip_addr)
             if not auth_u.ip_allowed:
                 return jsonrpc_error(
                     request, retid=request.rpc_id,
                     message='Request from IP:{} not allowed'.format(
                         request.rpc_ip_addr))
             else:
                 log.info('Access for IP:%s allowed', request.rpc_ip_addr)
 
             # register our auth-user
             request.rpc_user = auth_u
             request.environ['rc_auth_user_id'] = str(auth_u.user_id)
 
             # now check if token is valid for API
             auth_token = request.rpc_api_key
             token_match = api_user.authenticate_by_token(
                 auth_token, roles=[UserApiKeys.ROLE_API])
             invalid_token = not token_match
 
             log.debug('Checking if API KEY is valid with proper role')
             if invalid_token:
                 return jsonrpc_error(
                     request, retid=request.rpc_id,
                     message='API KEY invalid or, has bad role for an API call')
         else:
             auth_u = 'service'
             if request.rpc_api_key != request.registry.settings['app.service_api.token']:
                 raise Exception("Provided service secret is not recognized!")
 
     except Exception:
         log.exception('Error on API AUTH')
         return jsonrpc_error(
             request, retid=request.rpc_id, message='Invalid API KEY')
 
     method = request.rpc_method
     func = request.registry.jsonrpc_methods[method]
 
     # now that we have a method, add request._req_params to
     # self.kargs and dispatch control to WGIController
 
     argspec = inspect.getargspec(func)
     arglist = argspec[0]
     defs = argspec[3] or []
     defaults = [type(a) for a in defs]
     default_empty = type(NotImplemented)
 
     # kw arguments required by this method
     func_kwargs = dict(itertools.zip_longest(
         reversed(arglist), reversed(defaults), fillvalue=default_empty))
 
     # This attribute will need to be first param of a method that uses
     # api_key, which is translated to instance of user at that name
     user_var = 'apiuser'
     request_var = 'request'
 
     for arg in [user_var, request_var]:
         if arg not in arglist:
             return jsonrpc_error(
                 request,
                 retid=request.rpc_id,
                 message='This method [%s] does not support '
                         'required parameter `%s`' % (func.__name__, arg))
 
     # get our arglist and check if we provided them as args
     for arg, default in func_kwargs.items():
         if arg in [user_var, request_var]:
             # user_var and request_var are pre-hardcoded parameters and we
             # don't need to do any translation
             continue
 
         # skip the required param check if it's default value is
         # NotImplementedType (default_empty)
         if default == default_empty and arg not in request.rpc_params:
             return jsonrpc_error(
                 request,
                 retid=request.rpc_id,
                 message=('Missing non optional `%s` arg in JSON DATA' % arg)
             )
 
     # sanitize extra passed arguments
     for k in list(request.rpc_params.keys()):
         if k not in func_kwargs:
             del request.rpc_params[k]
 
     call_params = request.rpc_params
     call_params.update({
         'request': request,
         'apiuser': auth_u
     })
 
     # register some common functions for usage
     rpc_user = request.rpc_user.user_id if hasattr(request, 'rpc_user') else None
     attach_context_attributes(TemplateArgs(), request, rpc_user)
 
     statsd = request.registry.statsd
 
     try:
         ret_value = func(**call_params)
         resp = jsonrpc_response(request, ret_value)
         if statsd:
             statsd.incr('rhodecode_api_call_success_total')
         return resp
     except JSONRPCBaseError:
         raise
     except Exception:
         log.exception('Unhandled exception occurred on api call: %s', func)
         exc_info = sys.exc_info()
         exc_id, exc_type_name = store_exception(
             id(exc_info), exc_info, prefix='rhodecode-api')
         error_headers = {
             'RhodeCode-Exception-Id': str(exc_id),
             'RhodeCode-Exception-Type': str(exc_type_name)
         }
         err_resp = jsonrpc_error(
             request, retid=request.rpc_id, message='Internal server error',
             headers=error_headers)
         if statsd:
             statsd.incr('rhodecode_api_call_fail_total')
         return err_resp
 
 
 def setup_request(request):
     """
     Parse a JSON-RPC request body. It's used inside the predicates method
     to validate and bootstrap requests for usage in rpc calls.
 
     We need to raise JSONRPCError here if we want to return some errors back to
     user.
     """
 
     log.debug('Executing setup request: %r', request)
     request.rpc_ip_addr = get_ip_addr(request.environ)
     # TODO(marcink): deprecate GET at some point
     if request.method not in ['POST', 'GET']:
         log.debug('unsupported request method "%s"', request.method)
         raise JSONRPCError(
             'unsupported request method "%s". Please use POST' % request.method)
 
     if 'CONTENT_LENGTH' not in request.environ:
         log.debug("No Content-Length")
         raise JSONRPCError("Empty body, No Content-Length in request")
 
     else:
         length = request.environ['CONTENT_LENGTH']
         log.debug('Content-Length: %s', length)
 
         if length == 0:
             log.debug("Content-Length is 0")
             raise JSONRPCError("Content-Length is 0")
 
     raw_body = request.body
     log.debug("Loading JSON body now")
     try:
         json_body = ext_json.json.loads(raw_body)
     except ValueError as e:
         # catch JSON errors Here
         raise JSONRPCError(f"JSON parse error ERR:{e} RAW:{raw_body!r}")
 
     request.rpc_id = json_body.get('id')
     request.rpc_method = json_body.get('method')
 
     # check required base parameters
     try:
         api_key = json_body.get('api_key')
         if not api_key:
             api_key = json_body.get('auth_token')
 
         if not api_key:
             raise KeyError('api_key or auth_token')
 
         # TODO(marcink): support passing in token in request header
 
         request.rpc_api_key = api_key
         request.rpc_id = json_body['id']
         request.rpc_method = json_body['method']
         request.rpc_params = json_body['args'] \
             if isinstance(json_body['args'], dict) else {}
 
         log.debug('method: %s, params: %.10240r', request.rpc_method, request.rpc_params)
     except KeyError as e:
         raise JSONRPCError(f'Incorrect JSON data. Missing {e}')
 
     log.debug('setup complete, now handling method:%s rpcid:%s',
               request.rpc_method, request.rpc_id, )
 
 
 class RoutePredicate(object):
     def __init__(self, val, config):
         self.val = val
 
     def text(self):
         return f'jsonrpc route = {self.val}'
 
     phash = text
 
     def __call__(self, info, request):
         if self.val:
             # potentially setup and bootstrap our call
             setup_request(request)
 
             # Always return True so that even if it isn't a valid RPC it
             # will fall through to the underlaying handlers like notfound_view
             return True
 
 
 class NotFoundPredicate(object):
     def __init__(self, val, config):
         self.val = val
         self.methods = config.registry.jsonrpc_methods
 
     def text(self):
         return f'jsonrpc method not found = {self.val}'
 
     phash = text
 
     def __call__(self, info, request):
         return hasattr(request, 'rpc_method')
 
 
 class MethodPredicate(object):
     def __init__(self, val, config):
         self.method = val
 
     def text(self):
         return f'jsonrpc method = {self.method}'
 
     phash = text
 
     def __call__(self, context, request):
         # we need to explicitly return False here, so pyramid doesn't try to
         # execute our view directly. We need our main handler to execute things
         return getattr(request, 'rpc_method') == self.method
 
 
 def add_jsonrpc_method(config, view, **kwargs):
     # pop the method name
     method = kwargs.pop('method', None)
 
     if method is None:
         raise ConfigurationError(
             'Cannot register a JSON-RPC method without specifying the "method"')
 
     # we define custom predicate, to enable to detect conflicting methods,
     # those predicates are kind of "translation" from the decorator variables
     # to internal predicates names
 
     kwargs['jsonrpc_method'] = method
 
     # register our view into global view store for validation
     config.registry.jsonrpc_methods[method] = view
 
     # we're using our main request_view handler, here, so each method
     # has a unified handler for itself
     config.add_view(request_view, route_name='apiv2', **kwargs)
 
 
 class jsonrpc_method(object):
     """
     decorator that works similar to @add_view_config decorator,
     but tailored for our JSON RPC
     """
 
     venusian = venusian  # for testing injection
 
     def __init__(self, method=None, **kwargs):
         self.method = method
         self.kwargs = kwargs
 
     def __call__(self, wrapped):
         kwargs = self.kwargs.copy()
         kwargs['method'] = self.method or wrapped.__name__
         depth = kwargs.pop('_depth', 0)
 
         def callback(context, name, ob):
             config = context.config.with_package(info.module)
             config.add_jsonrpc_method(view=ob, **kwargs)
 
         info = venusian.attach(wrapped, callback, category='pyramid',
                                depth=depth + 1)
         if info.scope == 'class':
             # ensure that attr is set if decorating a class method
             kwargs.setdefault('attr', wrapped.__name__)
 
         kwargs['_info'] = info.codeinfo  # fbo action_method
         return wrapped
 
 
 class jsonrpc_deprecated_method(object):
     """
     Marks method as deprecated, adds log.warning, and inject special key to
     the request variable to mark method as deprecated.
     Also injects special docstring that extract_docs will catch to mark
     method as deprecated.
 
     :param use_method: specify which method should be used instead of
         the decorated one
 
     Use like::
 
         @jsonrpc_method()
         @jsonrpc_deprecated_method(use_method='new_func', deprecated_at_version='3.0.0')
         def old_func(request, apiuser, arg1, arg2):
             ...
     """
 
     def __init__(self, use_method, deprecated_at_version):
         self.use_method = use_method
         self.deprecated_at_version = deprecated_at_version
         self.deprecated_msg = ''
 
     def __call__(self, func):
         self.deprecated_msg = 'Please use method `{method}` instead.'.format(
             method=self.use_method)
 
         docstring = """\n
         .. deprecated:: {version}
 
            {deprecation_message}
 
         {original_docstring}
         """
         func.__doc__ = docstring.format(
             version=self.deprecated_at_version,
             deprecation_message=self.deprecated_msg,
             original_docstring=func.__doc__)
         return decorator.decorator(self.__wrapper, func)
 
     def __wrapper(self, func, *fargs, **fkwargs):
         log.warning('DEPRECATED API CALL on function %s, please '
                     'use `%s` instead', func, self.use_method)
         # alter function docstring to mark as deprecated, this is picked up
         # via fabric file that generates API DOC.
         result = func(*fargs, **fkwargs)
 
         request = fargs[0]
         request.rpc_deprecation = 'DEPRECATED METHOD ' + self.deprecated_msg
         return result
 
 
 def add_api_methods(config):
     from rhodecode.api.views import (
         deprecated_api, gist_api, pull_request_api, repo_api, repo_group_api,
         server_api, search_api, testing_api, user_api, user_group_api)
 
     config.scan('rhodecode.api.views')
 
 
 def includeme(config):
     plugin_module = 'rhodecode.api'
     plugin_settings = get_plugin_settings(
         plugin_module, config.registry.settings)
 
     if not hasattr(config.registry, 'jsonrpc_methods'):
         config.registry.jsonrpc_methods = OrderedDict()
 
     # match filter by given method only
     config.add_view_predicate('jsonrpc_method', MethodPredicate)
     config.add_view_predicate('jsonrpc_method_not_found', NotFoundPredicate)
 
     config.add_renderer(DEFAULT_RENDERER, ExtJsonRenderer())
     config.add_directive('add_jsonrpc_method', add_jsonrpc_method)
 
     config.add_route_predicate(
         'jsonrpc_call', RoutePredicate)
 
     config.add_route(
         'apiv2', plugin_settings.get('url', DEFAULT_URL), jsonrpc_call=True)
 
     # register some exception handling view
     config.add_view(exception_view, context=JSONRPCBaseError)
     config.add_notfound_view(exception_view, jsonrpc_method_not_found=True)
 
     add_api_methods(config)

rhodecode/api/views/server_api.py

@@ -1,423 +1,423 b''
 # Copyright (C) 2011-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 import itertools
 import base64
 
 from rhodecode.api import (
     jsonrpc_method, JSONRPCError, JSONRPCForbidden, find_methods)
 
 from rhodecode.api.utils import (
     Optional, OAttr, has_superadmin_permission, get_user_or_error)
 from rhodecode.lib.utils import repo2db_mapper, get_rhodecode_repo_store_path
 from rhodecode.lib import system_info
 from rhodecode.lib import user_sessions
 from rhodecode.lib import exc_tracking
 from rhodecode.lib.ext_json import json
 from rhodecode.lib.utils2 import safe_int
 from rhodecode.model.db import UserIpMap
 from rhodecode.model.scm import ScmModel
-from rhodecode.apps.file_store import utils
+from rhodecode.apps.file_store import utils as store_utils
 from rhodecode.apps.file_store.exceptions import FileNotAllowedException, \
     FileOverSizeException
 
 log = logging.getLogger(__name__)
 
 
 @jsonrpc_method()
 def get_server_info(request, apiuser):
     """
     Returns the |RCE| server information.
 
     This includes the running version of |RCE| and all installed
     packages. This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         'modules': [<module name>,...]
         'py_version': <python version>,
         'platform': <platform type>,
         'rhodecode_version': <rhodecode version>
       }
       error :  null
     """
 
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     server_info = ScmModel().get_server_info(request.environ)
     # rhodecode-index requires those
 
     server_info['index_storage'] = server_info['search']['value']['location']
     server_info['storage'] = server_info['storage']['value']['path']
 
     return server_info
 
 
 @jsonrpc_method()
 def get_repo_store(request, apiuser):
     """
     Returns the |RCE| repository storage information.
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         'modules': [<module name>,...]
         'py_version': <python version>,
         'platform': <platform type>,
         'rhodecode_version': <rhodecode version>
       }
       error :  null
     """
 
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     path = get_rhodecode_repo_store_path()
     return {"path": path}
 
 
 @jsonrpc_method()
 def get_ip(request, apiuser, userid=Optional(OAttr('apiuser'))):
     """
     Displays the IP Address as seen from the |RCE| server.
 
     * This command displays the IP Address, as well as all the defined IP
       addresses for the specified user. If the ``userid`` is not set, the
       data returned is for the user calling the method.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from |authtoken|.
     :type apiuser: AuthUser
     :param userid: Sets the userid for which associated IP Address data
         is returned.
     :type userid: Optional(str or int)
 
     Example output:
 
     .. code-block:: bash
 
         id : <id_given_in_input>
         result : {
                      "server_ip_addr": "<ip_from_clien>",
                      "user_ips": [
                                     {
                                        "ip_addr": "<ip_with_mask>",
                                        "ip_range": ["<start_ip>", "<end_ip>"],
                                     },
                                     ...
                                  ]
         }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     userid = Optional.extract(userid, evaluate_locals=locals())
     userid = getattr(userid, 'user_id', userid)
 
     user = get_user_or_error(userid)
     ips = UserIpMap.query().filter(UserIpMap.user == user).all()
     return {
         'server_ip_addr': request.rpc_ip_addr,
         'user_ips': ips
     }
 
 
 @jsonrpc_method()
 def rescan_repos(request, apiuser, remove_obsolete=Optional(False)):
     """
     Triggers a rescan of the specified repositories.
 
     * If the ``remove_obsolete`` option is set, it also deletes repositories
       that are found in the database but not on the file system, so called
       "clean zombies".
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param remove_obsolete: Deletes repositories from the database that
         are not found on the filesystem.
     :type remove_obsolete: Optional(``True`` | ``False``)
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : {
         'added': [<added repository name>,...]
         'removed': [<removed repository name>,...]
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         'Error occurred during rescan repositories action'
       }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     try:
         rm_obsolete = Optional.extract(remove_obsolete)
         added, removed = repo2db_mapper(ScmModel().repo_scan(),
                                         remove_obsolete=rm_obsolete, force_hooks_rebuild=True)
         return {'added': added, 'removed': removed}
     except Exception:
         log.exception('Failed to run repo rescann')
         raise JSONRPCError(
             'Error occurred during rescan repositories action'
         )
 
 
 @jsonrpc_method()
 def cleanup_sessions(request, apiuser, older_then=Optional(60)):
     """
     Triggers a session cleanup action.
 
     If the ``older_then`` option is set, only sessions that hasn't been
     accessed in the given number of days will be removed.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param older_then: Deletes session that hasn't been accessed
         in given number of days.
     :type older_then: Optional(int)
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result: {
         "backend": "<type of backend>",
         "sessions_removed": <number_of_removed_sessions>
       }
       error :  null
 
     Example error output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       result : null
       error :  {
         'Error occurred during session cleanup'
       }
 
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     older_then = safe_int(Optional.extract(older_then)) or 60
     older_than_seconds = 60 * 60 * 24 * older_then
 
     config = system_info.rhodecode_config().get_value()['value']['config']
     session_model = user_sessions.get_session_handler(
         config.get('beaker.session.type', 'memory'))(config)
 
     backend = session_model.SESSION_TYPE
     try:
         cleaned = session_model.clean_sessions(
             older_than_seconds=older_than_seconds)
         return {'sessions_removed': cleaned, 'backend': backend}
     except user_sessions.CleanupCommand as msg:
         return {'cleanup_command': str(msg), 'backend': backend}
     except Exception as e:
         log.exception('Failed session cleanup')
         raise JSONRPCError(
             'Error occurred during session cleanup'
         )
 
 
 @jsonrpc_method()
 def get_method(request, apiuser, pattern=Optional('*')):
     """
     Returns list of all available API methods. By default match pattern
     os "*" but any other pattern can be specified. eg *comment* will return
     all methods with comment inside them. If just single method is matched
     returned data will also include method specification
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
     :param pattern: pattern to match method names against
     :type pattern: Optional("*")
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       "result": [
         "changeset_comment",
         "comment_pull_request",
         "comment_commit"
       ]
       error :  null
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       "result": [
         "comment_commit",
         {
           "apiuser": "<RequiredType>",
           "comment_type": "<Optional:u'note'>",
           "commit_id": "<RequiredType>",
           "message": "<RequiredType>",
           "repoid": "<RequiredType>",
           "request": "<RequiredType>",
           "resolves_comment_id": "<Optional:None>",
           "status": "<Optional:None>",
           "userid": "<Optional:<OptionalAttr:apiuser>>"
         }
       ]
       error :  null
     """
-    from rhodecode.config.patches import inspect_getargspec
-    inspect = inspect_getargspec()
+    from rhodecode.config import patches
+    inspect = patches.inspect_getargspec()
 
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     pattern = Optional.extract(pattern)
 
     matches = find_methods(request.registry.jsonrpc_methods, pattern)
 
     args_desc = []
     matches_keys = list(matches.keys())
     if len(matches_keys) == 1:
         func = matches[matches_keys[0]]
 
         argspec = inspect.getargspec(func)
         arglist = argspec[0]
         defaults = list(map(repr, argspec[3] or []))
 
         default_empty = '<RequiredType>'
 
         # kw arguments required by this method
         func_kwargs = dict(itertools.zip_longest(
             reversed(arglist), reversed(defaults), fillvalue=default_empty))
         args_desc.append(func_kwargs)
 
     return matches_keys + args_desc
 
 
 @jsonrpc_method()
 def store_exception(request, apiuser, exc_data_json, prefix=Optional('rhodecode')):
     """
     Stores sent exception inside the built-in exception tracker in |RCE| server.
 
     This command can only be run using an |authtoken| with admin rights to
     the specified repository.
 
     This command takes the following options:
 
     :param apiuser: This is filled automatically from the |authtoken|.
     :type apiuser: AuthUser
 
     :param exc_data_json: JSON data with exception e.g
         {"exc_traceback": "Value `1` is not allowed", "exc_type_name": "ValueError"}
     :type exc_data_json: JSON data
 
     :param prefix: prefix for error type, e.g 'rhodecode', 'vcsserver', 'rhodecode-tools'
     :type prefix: Optional("rhodecode")
 
     Example output:
 
     .. code-block:: bash
 
       id : <id_given_in_input>
       "result": {
         "exc_id": 139718459226384,
         "exc_url": "http://localhost:8080/_admin/settings/exceptions/139718459226384"
       }
       error :  null
     """
     if not has_superadmin_permission(apiuser):
         raise JSONRPCForbidden()
 
     prefix = Optional.extract(prefix)
     exc_id = exc_tracking.generate_id()
 
     try:
         exc_data = json.loads(exc_data_json)
     except Exception:
         log.error('Failed to parse JSON: %r', exc_data_json)
         raise JSONRPCError('Failed to parse JSON data from exc_data_json field. '
                            'Please make sure it contains a valid JSON.')
 
     try:
         exc_traceback = exc_data['exc_traceback']
         exc_type_name = exc_data['exc_type_name']
         exc_value = ''
     except KeyError as err:
         raise JSONRPCError(
             f'Missing exc_traceback, or exc_type_name '
             f'in exc_data_json field. Missing: {err}')
 
     class ExcType:
         __name__ = exc_type_name
 
     exc_info = (ExcType(), exc_value, exc_traceback)
 
     exc_tracking._store_exception(
         exc_id=exc_id, exc_info=exc_info, prefix=prefix)
 
     exc_url = request.route_url(
         'admin_settings_exception_tracker_show', exception_id=exc_id)
     return {'exc_id': exc_id, 'exc_url': exc_url}

rhodecode/apps/admin/__init__.py

@@ -1,1093 +1,1124 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 from rhodecode.apps._base import ADMIN_PREFIX
 from rhodecode.apps._base.navigation import includeme as nav_includeme
 from rhodecode.apps.admin.views.main_views import AdminMainView
 
 
 def admin_routes(config):
     """
     Admin prefixed routes
     """
     from rhodecode.apps.admin.views.audit_logs import AdminAuditLogsView
     from rhodecode.apps.admin.views.artifacts import AdminArtifactsView
     from rhodecode.apps.admin.views.automation import AdminAutomationView
     from rhodecode.apps.admin.views.scheduler import AdminSchedulerView
     from rhodecode.apps.admin.views.defaults import AdminDefaultSettingsView
     from rhodecode.apps.admin.views.exception_tracker import ExceptionsTrackerView
     from rhodecode.apps.admin.views.open_source_licenses import OpenSourceLicensesAdminSettingsView
     from rhodecode.apps.admin.views.permissions import AdminPermissionsView
     from rhodecode.apps.admin.views.process_management import AdminProcessManagementView
     from rhodecode.apps.admin.views.repo_groups import AdminRepoGroupsView 
     from rhodecode.apps.admin.views.repositories import AdminReposView
     from rhodecode.apps.admin.views.sessions import AdminSessionSettingsView
     from rhodecode.apps.admin.views.settings import AdminSettingsView
     from rhodecode.apps.admin.views.svn_config import AdminSvnConfigView
     from rhodecode.apps.admin.views.system_info import AdminSystemInfoSettingsView
     from rhodecode.apps.admin.views.user_groups import AdminUserGroupsView
     from rhodecode.apps.admin.views.users import AdminUsersView, UsersView
+    from rhodecode.apps.admin.views.security import AdminSecurityView
+
+    # Security EE feature
+
+    config.add_route(
+        'admin_security',
+        pattern='/security')
+    config.add_view(
+        AdminSecurityView,
+        attr='security',
+        route_name='admin_security', request_method='GET',
+        renderer='rhodecode:templates/admin/security/security.mako')
+
+    config.add_route(
+        name='admin_security_update',
+        pattern='/security/update')
+    config.add_view(
+        AdminSecurityView,
+        attr='security_update',
+        route_name='admin_security_update', request_method='POST',
+        renderer='rhodecode:templates/admin/security/security.mako')
+
+    config.add_route(
+        name='admin_security_modify_allowed_vcs_client_versions',
+        pattern=ADMIN_PREFIX + '/security/modify/allowed_vcs_client_versions')
+    config.add_view(
+        AdminSecurityView,
+        attr='vcs_whitelisted_client_versions_edit',
+        route_name='admin_security_modify_allowed_vcs_client_versions', request_method=('GET', 'POST'),
+        renderer='rhodecode:templates/admin/security/edit_allowed_vcs_client_versions.mako')
+
     
     config.add_route(
         name='admin_audit_logs',
         pattern='/audit_logs')
     config.add_view(
         AdminAuditLogsView,
         attr='admin_audit_logs',
         route_name='admin_audit_logs', request_method='GET',
         renderer='rhodecode:templates/admin/admin_audit_logs.mako')
 
     config.add_route(
         name='admin_audit_log_entry',
         pattern='/audit_logs/{audit_log_id}')
     config.add_view(
         AdminAuditLogsView,
         attr='admin_audit_log_entry',
         route_name='admin_audit_log_entry', request_method='GET',
         renderer='rhodecode:templates/admin/admin_audit_log_entry.mako')
 
     # Artifacts EE feature
     config.add_route(
         'admin_artifacts',
         pattern=ADMIN_PREFIX + '/artifacts')
     config.add_route(
         'admin_artifacts_show_all',
         pattern=ADMIN_PREFIX + '/artifacts')
     config.add_view(
         AdminArtifactsView,
         attr='artifacts',
         route_name='admin_artifacts', request_method='GET',
         renderer='rhodecode:templates/admin/artifacts/artifacts.mako')
     config.add_view(
         AdminArtifactsView,
         attr='artifacts',
         route_name='admin_artifacts_show_all', request_method='GET',
         renderer='rhodecode:templates/admin/artifacts/artifacts.mako')
 
     # EE views
     config.add_route(
         name='admin_artifacts_show_info',
         pattern=ADMIN_PREFIX + '/artifacts/{uid}')
     config.add_route(
         name='admin_artifacts_delete',
         pattern=ADMIN_PREFIX + '/artifacts/{uid}/delete')
     config.add_route(
         name='admin_artifacts_update',
         pattern=ADMIN_PREFIX + '/artifacts/{uid}/update')
 
     # Automation EE feature
     config.add_route(
         'admin_automation',
         pattern=ADMIN_PREFIX + '/automation')
     config.add_view(
         AdminAutomationView,
         attr='automation',
         route_name='admin_automation', request_method='GET',
         renderer='rhodecode:templates/admin/automation/automation.mako')
 
     # Scheduler EE feature
     config.add_route(
         'admin_scheduler',
         pattern=ADMIN_PREFIX + '/scheduler')
     config.add_view(
         AdminSchedulerView,
         attr='scheduler',
         route_name='admin_scheduler', request_method='GET',
         renderer='rhodecode:templates/admin/scheduler/scheduler.mako')
 
     config.add_route(
         name='admin_settings_open_source',
         pattern='/settings/open_source')
     config.add_view(
         OpenSourceLicensesAdminSettingsView,
         attr='open_source_licenses',
         route_name='admin_settings_open_source', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_vcs_svn_generate_cfg',
         pattern='/settings/vcs/svn_generate_cfg')
     config.add_view(
         AdminSvnConfigView,
         attr='vcs_svn_generate_config',
         route_name='admin_settings_vcs_svn_generate_cfg',
         request_method='POST', renderer='json')
 
     config.add_route(
         name='admin_settings_system',
         pattern='/settings/system')
     config.add_view(
         AdminSystemInfoSettingsView,
         attr='settings_system_info',
         route_name='admin_settings_system', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_system_update',
         pattern='/settings/system/updates')
     config.add_view(
         AdminSystemInfoSettingsView,
         attr='settings_system_info_check_update',
         route_name='admin_settings_system_update', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings_system_update.mako')
 
     config.add_route(
         name='admin_settings_exception_tracker',
         pattern='/settings/exceptions')
     config.add_view(
         ExceptionsTrackerView,
         attr='browse_exceptions',
         route_name='admin_settings_exception_tracker', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_exception_tracker_delete_all',
         pattern='/settings/exceptions_delete_all')
     config.add_view(
         ExceptionsTrackerView,
         attr='exception_delete_all',
         route_name='admin_settings_exception_tracker_delete_all', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_exception_tracker_show',
         pattern='/settings/exceptions/{exception_id}')
     config.add_view(
         ExceptionsTrackerView,
         attr='exception_show',
         route_name='admin_settings_exception_tracker_show', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_exception_tracker_delete',
         pattern='/settings/exceptions/{exception_id}/delete')
     config.add_view(
         ExceptionsTrackerView,
         attr='exception_delete',
         route_name='admin_settings_exception_tracker_delete', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_sessions',
         pattern='/settings/sessions')
     config.add_view(
         AdminSessionSettingsView,
         attr='settings_sessions',
         route_name='admin_settings_sessions', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_sessions_cleanup',
         pattern='/settings/sessions/cleanup')
     config.add_view(
         AdminSessionSettingsView,
         attr='settings_sessions_cleanup',
         route_name='admin_settings_sessions_cleanup', request_method='POST')
 
     config.add_route(
         name='admin_settings_process_management',
         pattern='/settings/process_management')
     config.add_view(
         AdminProcessManagementView,
         attr='process_management',
         route_name='admin_settings_process_management', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_process_management_data',
         pattern='/settings/process_management/data')
     config.add_view(
         AdminProcessManagementView,
         attr='process_management_data',
         route_name='admin_settings_process_management_data', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings_process_management_data.mako')
 
     config.add_route(
         name='admin_settings_process_management_signal',
         pattern='/settings/process_management/signal')
     config.add_view(
         AdminProcessManagementView,
         attr='process_management_signal',
         route_name='admin_settings_process_management_signal',
         request_method='POST', renderer='json_ext')
 
     config.add_route(
         name='admin_settings_process_management_master_signal',
         pattern='/settings/process_management/master_signal')
     config.add_view(
         AdminProcessManagementView,
         attr='process_management_master_signal',
         route_name='admin_settings_process_management_master_signal',
         request_method='POST', renderer='json_ext')
 
     # default settings
     config.add_route(
         name='admin_defaults_repositories',
         pattern='/defaults/repositories')
     config.add_view(
         AdminDefaultSettingsView,
         attr='defaults_repository_show',
         route_name='admin_defaults_repositories', request_method='GET',
         renderer='rhodecode:templates/admin/defaults/defaults.mako')
 
     config.add_route(
         name='admin_defaults_repositories_update',
         pattern='/defaults/repositories/update')
     config.add_view(
         AdminDefaultSettingsView,
         attr='defaults_repository_update',
         route_name='admin_defaults_repositories_update', request_method='POST',
         renderer='rhodecode:templates/admin/defaults/defaults.mako')
 
     # admin settings
 
     config.add_route(
         name='admin_settings',
         pattern='/settings')
     config.add_view(
         AdminSettingsView,
         attr='settings_global',
         route_name='admin_settings', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_update',
         pattern='/settings/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_global_update',
         route_name='admin_settings_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_global',
         pattern='/settings/global')
     config.add_view(
         AdminSettingsView,
         attr='settings_global',
         route_name='admin_settings_global', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_global_update',
         pattern='/settings/global/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_global_update',
         route_name='admin_settings_global_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_vcs',
         pattern='/settings/vcs')
     config.add_view(
         AdminSettingsView,
         attr='settings_vcs',
         route_name='admin_settings_vcs', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_vcs_update',
         pattern='/settings/vcs/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_vcs_update',
         route_name='admin_settings_vcs_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_vcs_svn_pattern_delete',
         pattern='/settings/vcs/svn_pattern_delete')
     config.add_view(
         AdminSettingsView,
         attr='settings_vcs_delete_svn_pattern',
         route_name='admin_settings_vcs_svn_pattern_delete', request_method='POST',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='admin_settings_mapping',
         pattern='/settings/mapping')
     config.add_view(
         AdminSettingsView,
         attr='settings_mapping',
         route_name='admin_settings_mapping', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_mapping_update',
         pattern='/settings/mapping/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_mapping_update',
         route_name='admin_settings_mapping_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_visual',
         pattern='/settings/visual')
     config.add_view(
         AdminSettingsView,
         attr='settings_visual',
         route_name='admin_settings_visual', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_visual_update',
         pattern='/settings/visual/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_visual_update',
         route_name='admin_settings_visual_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_issuetracker',
         pattern='/settings/issue-tracker')
     config.add_view(
         AdminSettingsView,
         attr='settings_issuetracker',
         route_name='admin_settings_issuetracker', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_issuetracker_update',
         pattern='/settings/issue-tracker/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_issuetracker_update',
         route_name='admin_settings_issuetracker_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_issuetracker_test',
         pattern='/settings/issue-tracker/test')
     config.add_view(
         AdminSettingsView,
         attr='settings_issuetracker_test',
         route_name='admin_settings_issuetracker_test', request_method='POST',
         renderer='string', xhr=True)
 
     config.add_route(
         name='admin_settings_issuetracker_delete',
         pattern='/settings/issue-tracker/delete')
     config.add_view(
         AdminSettingsView,
         attr='settings_issuetracker_delete',
         route_name='admin_settings_issuetracker_delete', request_method='POST',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='admin_settings_email',
         pattern='/settings/email')
     config.add_view(
         AdminSettingsView,
         attr='settings_email',
         route_name='admin_settings_email', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_email_update',
         pattern='/settings/email/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_email_update',
         route_name='admin_settings_email_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_hooks',
         pattern='/settings/hooks')
     config.add_view(
         AdminSettingsView,
         attr='settings_hooks',
         route_name='admin_settings_hooks', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_hooks_update',
         pattern='/settings/hooks/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_hooks_update',
         route_name='admin_settings_hooks_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_hooks_delete',
         pattern='/settings/hooks/delete')
     config.add_view(
         AdminSettingsView,
         attr='settings_hooks_update',
         route_name='admin_settings_hooks_delete', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_search',
         pattern='/settings/search')
     config.add_view(
         AdminSettingsView,
         attr='settings_search',
         route_name='admin_settings_search', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_labs',
         pattern='/settings/labs')
     config.add_view(
         AdminSettingsView,
         attr='settings_labs',
         route_name='admin_settings_labs', request_method='GET',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     config.add_route(
         name='admin_settings_labs_update',
         pattern='/settings/labs/update')
     config.add_view(
         AdminSettingsView,
         attr='settings_labs_update',
         route_name='admin_settings_labs_update', request_method='POST',
         renderer='rhodecode:templates/admin/settings/settings.mako')
 
     # global permissions
 
     config.add_route(
         name='admin_permissions_application',
         pattern='/permissions/application')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_application',
         route_name='admin_permissions_application', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_application_update',
         pattern='/permissions/application/update')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_application_update',
         route_name='admin_permissions_application_update', request_method='POST',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_global',
         pattern='/permissions/global')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_global',
         route_name='admin_permissions_global', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
     
     config.add_route(
         name='admin_permissions_global_update',
         pattern='/permissions/global/update')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_global_update',
         route_name='admin_permissions_global_update', request_method='POST',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_object',
         pattern='/permissions/object')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_objects',
         route_name='admin_permissions_object', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_object_update',
         pattern='/permissions/object/update')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_objects_update',
         route_name='admin_permissions_object_update', request_method='POST',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     # Branch perms EE feature
     config.add_route(
         name='admin_permissions_branch',
         pattern='/permissions/branch')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_branch',
         route_name='admin_permissions_branch', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_ips',
         pattern='/permissions/ips')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_ips',
         route_name='admin_permissions_ips', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_overview',
         pattern='/permissions/overview')
     config.add_view(
         AdminPermissionsView,
         attr='permissions_overview',
         route_name='admin_permissions_overview', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_auth_token_access',
         pattern='/permissions/auth_token_access')
     config.add_view(
         AdminPermissionsView,
         attr='auth_token_access',
         route_name='admin_permissions_auth_token_access', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_ssh_keys',
         pattern='/permissions/ssh_keys')
     config.add_view(
         AdminPermissionsView,
         attr='ssh_keys',
         route_name='admin_permissions_ssh_keys', request_method='GET',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     config.add_route(
         name='admin_permissions_ssh_keys_data',
         pattern='/permissions/ssh_keys/data')
     config.add_view(
         AdminPermissionsView,
         attr='ssh_keys_data',
         route_name='admin_permissions_ssh_keys_data', request_method='GET',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='admin_permissions_ssh_keys_update',
         pattern='/permissions/ssh_keys/update')
     config.add_view(
         AdminPermissionsView,
         attr='ssh_keys_update',
         route_name='admin_permissions_ssh_keys_update', request_method='POST',
         renderer='rhodecode:templates/admin/permissions/permissions.mako')
 
     # users admin
     config.add_route(
         name='users',
         pattern='/users')
     config.add_view(
         AdminUsersView,
         attr='users_list',
         route_name='users', request_method='GET',
         renderer='rhodecode:templates/admin/users/users.mako')
 
     config.add_route(
         name='users_data',
         pattern='/users_data')
     config.add_view(
         AdminUsersView,
         attr='users_list_data',
         # renderer defined below
         route_name='users_data', request_method='GET',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='users_create',
         pattern='/users/create')
     config.add_view(
         AdminUsersView,
         attr='users_create',
         route_name='users_create', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_add.mako')
 
     config.add_route(
         name='users_new',
         pattern='/users/new')
     config.add_view(
         AdminUsersView,
         attr='users_new',
         route_name='users_new', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_add.mako')
 
     # user management
     config.add_route(
         name='user_edit',
         pattern=r'/users/{user_id:\d+}/edit',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_edit',
         route_name='user_edit', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_edit_advanced',
         pattern=r'/users/{user_id:\d+}/edit/advanced',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_edit_advanced',
         route_name='user_edit_advanced', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_edit_global_perms',
         pattern=r'/users/{user_id:\d+}/edit/global_permissions',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_edit_global_perms',
         route_name='user_edit_global_perms', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_edit_global_perms_update',
         pattern=r'/users/{user_id:\d+}/edit/global_permissions/update',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_edit_global_perms_update',
         route_name='user_edit_global_perms_update', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_update',
         pattern=r'/users/{user_id:\d+}/update',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_update',
         route_name='user_update', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_delete',
         pattern=r'/users/{user_id:\d+}/delete',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_delete',
         route_name='user_delete', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_enable_force_password_reset',
         pattern=r'/users/{user_id:\d+}/password_reset_enable',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_enable_force_password_reset',
         route_name='user_enable_force_password_reset', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_disable_force_password_reset',
         pattern=r'/users/{user_id:\d+}/password_reset_disable',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_disable_force_password_reset',
         route_name='user_disable_force_password_reset', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='user_create_personal_repo_group',
         pattern=r'/users/{user_id:\d+}/create_repo_group',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_create_personal_repo_group',
         route_name='user_create_personal_repo_group', request_method='POST',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     # user notice
     config.add_route(
         name='user_notice_dismiss',
         pattern=r'/users/{user_id:\d+}/notice_dismiss',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_notice_dismiss',
         route_name='user_notice_dismiss', request_method='POST',
         renderer='json_ext', xhr=True)
 
     # user auth tokens
     config.add_route(
         name='edit_user_auth_tokens',
         pattern=r'/users/{user_id:\d+}/edit/auth_tokens',
         user_route=True)
     config.add_view(
         UsersView,
         attr='auth_tokens',
         route_name='edit_user_auth_tokens', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_auth_tokens_view',
         pattern=r'/users/{user_id:\d+}/edit/auth_tokens/view',
         user_route=True)
     config.add_view(
         UsersView,
         attr='auth_tokens_view',
         route_name='edit_user_auth_tokens_view', request_method='POST',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='edit_user_auth_tokens_add',
         pattern=r'/users/{user_id:\d+}/edit/auth_tokens/new',
         user_route=True)
     config.add_view(
         UsersView,
         attr='auth_tokens_add',
         route_name='edit_user_auth_tokens_add', request_method='POST')
 
     config.add_route(
         name='edit_user_auth_tokens_delete',
         pattern=r'/users/{user_id:\d+}/edit/auth_tokens/delete',
         user_route=True)
     config.add_view(
         UsersView,
         attr='auth_tokens_delete',
         route_name='edit_user_auth_tokens_delete', request_method='POST')
 
     # user ssh keys
     config.add_route(
         name='edit_user_ssh_keys',
         pattern=r'/users/{user_id:\d+}/edit/ssh_keys',
         user_route=True)
     config.add_view(
         UsersView,
         attr='ssh_keys',
         route_name='edit_user_ssh_keys', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_ssh_keys_generate_keypair',
         pattern=r'/users/{user_id:\d+}/edit/ssh_keys/generate',
         user_route=True)
     config.add_view(
         UsersView,
         attr='ssh_keys_generate_keypair',
         route_name='edit_user_ssh_keys_generate_keypair', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_ssh_keys_add',
         pattern=r'/users/{user_id:\d+}/edit/ssh_keys/new',
         user_route=True)
     config.add_view(
         UsersView,
         attr='ssh_keys_add',
         route_name='edit_user_ssh_keys_add', request_method='POST')
 
     config.add_route(
         name='edit_user_ssh_keys_delete',
         pattern=r'/users/{user_id:\d+}/edit/ssh_keys/delete',
         user_route=True)
     config.add_view(
         UsersView,
         attr='ssh_keys_delete',
         route_name='edit_user_ssh_keys_delete', request_method='POST')
 
     # user emails
     config.add_route(
         name='edit_user_emails',
         pattern=r'/users/{user_id:\d+}/edit/emails',
         user_route=True)
     config.add_view(
         UsersView,
         attr='emails',
         route_name='edit_user_emails', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_emails_add',
         pattern=r'/users/{user_id:\d+}/edit/emails/new',
         user_route=True)
     config.add_view(
         UsersView,
         attr='emails_add',
         route_name='edit_user_emails_add', request_method='POST')
 
     config.add_route(
         name='edit_user_emails_delete',
         pattern=r'/users/{user_id:\d+}/edit/emails/delete',
         user_route=True)
     config.add_view(
         UsersView,
         attr='emails_delete',
         route_name='edit_user_emails_delete', request_method='POST')
 
     # user IPs
     config.add_route(
         name='edit_user_ips',
         pattern=r'/users/{user_id:\d+}/edit/ips',
         user_route=True)
     config.add_view(
         UsersView,
         attr='ips',
         route_name='edit_user_ips', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_ips_add',
         pattern=r'/users/{user_id:\d+}/edit/ips/new',
         user_route_with_default=True)  # enabled for default user too
     config.add_view(
         UsersView,
         attr='ips_add',
         route_name='edit_user_ips_add', request_method='POST')
 
     config.add_route(
         name='edit_user_ips_delete',
         pattern=r'/users/{user_id:\d+}/edit/ips/delete',
         user_route_with_default=True)  # enabled for default user too
     config.add_view(
         UsersView,
         attr='ips_delete',
         route_name='edit_user_ips_delete', request_method='POST')
 
     # user perms
     config.add_route(
         name='edit_user_perms_summary',
         pattern=r'/users/{user_id:\d+}/edit/permissions_summary',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_perms_summary',
         route_name='edit_user_perms_summary', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_perms_summary_json',
         pattern=r'/users/{user_id:\d+}/edit/permissions_summary/json',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_perms_summary_json',
         route_name='edit_user_perms_summary_json', request_method='GET',
         renderer='json_ext')
 
     # user user groups management
     config.add_route(
         name='edit_user_groups_management',
         pattern=r'/users/{user_id:\d+}/edit/groups_management',
         user_route=True)
     config.add_view(
         UsersView,
         attr='groups_management',
         route_name='edit_user_groups_management', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_groups_management_updates',
         pattern=r'/users/{user_id:\d+}/edit/edit_user_groups_management/updates',
         user_route=True)
     config.add_view(
         UsersView,
         attr='groups_management_updates',
         route_name='edit_user_groups_management_updates', request_method='POST')
 
     # user audit logs
     config.add_route(
         name='edit_user_audit_logs',
         pattern=r'/users/{user_id:\d+}/edit/audit', user_route=True)
     config.add_view(
         UsersView,
         attr='user_audit_logs',
         route_name='edit_user_audit_logs', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_audit_logs_download',
         pattern=r'/users/{user_id:\d+}/edit/audit/download', user_route=True)
     config.add_view(
         UsersView,
         attr='user_audit_logs_download',
         route_name='edit_user_audit_logs_download', request_method='GET',
         renderer='string')
 
     # user caches
     config.add_route(
         name='edit_user_caches',
         pattern=r'/users/{user_id:\d+}/edit/caches',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_caches',
         route_name='edit_user_caches', request_method='GET',
         renderer='rhodecode:templates/admin/users/user_edit.mako')
 
     config.add_route(
         name='edit_user_caches_update',
         pattern=r'/users/{user_id:\d+}/edit/caches/update',
         user_route=True)
     config.add_view(
         UsersView,
         attr='user_caches_update',
         route_name='edit_user_caches_update', request_method='POST')
 
     # user-groups admin
     config.add_route(
         name='user_groups',
         pattern='/user_groups')
     config.add_view(
         AdminUserGroupsView,
         attr='user_groups_list',
         route_name='user_groups', request_method='GET',
         renderer='rhodecode:templates/admin/user_groups/user_groups.mako')
 
     config.add_route(
         name='user_groups_data',
         pattern='/user_groups_data')
     config.add_view(
         AdminUserGroupsView,
         attr='user_groups_list_data',
         route_name='user_groups_data', request_method='GET',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='user_groups_new',
         pattern='/user_groups/new')
     config.add_view(
         AdminUserGroupsView,
         attr='user_groups_new',
         route_name='user_groups_new', request_method='GET',
         renderer='rhodecode:templates/admin/user_groups/user_group_add.mako')
 
     config.add_route(
         name='user_groups_create',
         pattern='/user_groups/create')
     config.add_view(
         AdminUserGroupsView,
         attr='user_groups_create',
         route_name='user_groups_create', request_method='POST',
         renderer='rhodecode:templates/admin/user_groups/user_group_add.mako')
 
     # repos admin
     config.add_route(
         name='repos',
         pattern='/repos')
     config.add_view(
         AdminReposView,
         attr='repository_list',
         route_name='repos', request_method='GET',
         renderer='rhodecode:templates/admin/repos/repos.mako')
 
     config.add_route(
         name='repos_data',
         pattern='/repos_data')
     config.add_view(
         AdminReposView,
         attr='repository_list_data',
         route_name='repos_data', request_method='GET',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='repo_new',
         pattern='/repos/new')
     config.add_view(
         AdminReposView,
         attr='repository_new',
         route_name='repo_new', request_method='GET',
         renderer='rhodecode:templates/admin/repos/repo_add.mako')
 
     config.add_route(
         name='repo_create',
         pattern='/repos/create')
     config.add_view(
         AdminReposView,
         attr='repository_create',
         route_name='repo_create', request_method='POST',
         renderer='rhodecode:templates/admin/repos/repos.mako')
 
     # repo groups admin
     config.add_route(
         name='repo_groups',
         pattern='/repo_groups')
     config.add_view(
         AdminRepoGroupsView,
         attr='repo_group_list',
         route_name='repo_groups', request_method='GET',
         renderer='rhodecode:templates/admin/repo_groups/repo_groups.mako')
 
     config.add_route(
         name='repo_groups_data',
         pattern='/repo_groups_data')
     config.add_view(
         AdminRepoGroupsView,
         attr='repo_group_list_data',
         route_name='repo_groups_data', request_method='GET',
         renderer='json_ext', xhr=True)
 
     config.add_route(
         name='repo_group_new',
         pattern='/repo_group/new')
     config.add_view(
         AdminRepoGroupsView,
         attr='repo_group_new',
         route_name='repo_group_new', request_method='GET',
         renderer='rhodecode:templates/admin/repo_groups/repo_group_add.mako')
 
     config.add_route(
         name='repo_group_create',
         pattern='/repo_group/create')
     config.add_view(
         AdminRepoGroupsView,
         attr='repo_group_create',
         route_name='repo_group_create', request_method='POST',
         renderer='rhodecode:templates/admin/repo_groups/repo_group_add.mako')
 
 
 def includeme(config):
     # Create admin navigation registry and add it to the pyramid registry.
     nav_includeme(config)
 
     # main admin routes
     config.add_route(
         name='admin_home', pattern=ADMIN_PREFIX)
     config.add_view(
         AdminMainView,
         attr='admin_main',
         route_name='admin_home', request_method='GET',
         renderer='rhodecode:templates/admin/main.mako')
 
     # pr global redirect
     config.add_route(
         name='pull_requests_global_0',  # backward compat
         pattern=ADMIN_PREFIX + r'/pull_requests/{pull_request_id:\d+}')
     config.add_view(
         AdminMainView,
         attr='pull_requests',
         route_name='pull_requests_global_0', request_method='GET')
 
     config.add_route(
         name='pull_requests_global_1',  # backward compat
         pattern=ADMIN_PREFIX + r'/pull-requests/{pull_request_id:\d+}')
     config.add_view(
         AdminMainView,
         attr='pull_requests',
         route_name='pull_requests_global_1', request_method='GET')
 
     config.add_route(
         name='pull_requests_global',
         pattern=ADMIN_PREFIX + r'/pull-request/{pull_request_id:\d+}')
     config.add_view(
         AdminMainView,
         attr='pull_requests',
         route_name='pull_requests_global', request_method='GET')
 
     config.include(admin_routes, route_prefix=ADMIN_PREFIX)

rhodecode/apps/admin/views/settings.py

@@ -1,708 +1,715 b''
 # Copyright (C) 2010-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 import logging
 import collections
 
 import datetime
 import formencode
 import formencode.htmlfill
 
 import rhodecode
 
 from pyramid.httpexceptions import HTTPFound, HTTPNotFound
 from pyramid.renderers import render
 from pyramid.response import Response
 
 from rhodecode.apps._base import BaseAppView
 from rhodecode.apps._base.navigation import navigation_list
 from rhodecode.apps.svn_support import config_keys
 from rhodecode.lib import helpers as h
 from rhodecode.lib.auth import (
     LoginRequired, HasPermissionAllDecorator, CSRFRequired)
 from rhodecode.lib.celerylib import tasks, run_task
 from rhodecode.lib.str_utils import safe_str
 from rhodecode.lib.utils import repo2db_mapper, get_rhodecode_repo_store_path
 from rhodecode.lib.utils2 import str2bool, AttributeDict
 from rhodecode.lib.index import searcher_from_config
 
 from rhodecode.model.db import RhodeCodeUi, Repository
 from rhodecode.model.forms import (ApplicationSettingsForm,
     ApplicationUiSettingsForm, ApplicationVisualisationForm,
     LabsSettingsForm, IssueTrackerPatternsForm)
 from rhodecode.model.permission import PermissionModel
 from rhodecode.model.repo_group import RepoGroupModel
 
 from rhodecode.model.scm import ScmModel
 from rhodecode.model.notification import EmailNotificationModel
 from rhodecode.model.meta import Session
 from rhodecode.model.settings import (
     IssueTrackerSettingsModel, VcsSettingsModel, SettingNotFound,
     SettingsModel)
 
 
 log = logging.getLogger(__name__)
 
 
 class AdminSettingsView(BaseAppView):
 
     def load_default_context(self):
         c = self._get_local_tmpl_context()
         c.labs_active = str2bool(
             rhodecode.CONFIG.get('labs_settings_active', 'true'))
         c.navlist = navigation_list(self.request)
         return c
 
     @classmethod
     def _get_ui_settings(cls):
         ret = RhodeCodeUi.query().all()
 
         if not ret:
             raise Exception('Could not get application ui settings !')
-        settings = {}
+        settings = {
+            # legacy param that needs to be kept
+            'web_push_ssl': False
+        }
         for each in ret:
             k = each.ui_key
             v = each.ui_value
+            # skip some options if they are defined
+            if k in ['push_ssl']:
+                continue
+
             if k == '/':
                 k = 'root_path'
 
-            if k in ['push_ssl', 'publish', 'enabled']:
+            if k in ['publish', 'enabled']:
                 v = str2bool(v)
 
             if k.find('.') != -1:
                 k = k.replace('.', '_')
 
             if each.ui_section in ['hooks', 'extensions']:
                 v = each.ui_active
 
             settings[each.ui_section + '_' + k] = v
+
         return settings
 
     @classmethod
     def _form_defaults(cls):
         defaults = SettingsModel().get_all_settings()
         defaults.update(cls._get_ui_settings())
 
         defaults.update({
             'new_svn_branch': '',
             'new_svn_tag': '',
         })
         return defaults
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_vcs(self):
         c = self.load_default_context()
         c.active = 'vcs'
         model = VcsSettingsModel()
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
         c.svn_generate_config = rhodecode.ConfigGet().get_bool(config_keys.generate_config)
         c.svn_config_path = rhodecode.ConfigGet().get_str(config_keys.config_file_path)
         defaults = self._form_defaults()
 
         model.create_largeobjects_dirs_if_needed(defaults['paths_root_path'])
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=defaults,
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_vcs_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'vcs'
 
         model = VcsSettingsModel()
         c.svn_branch_patterns = model.get_global_svn_branch_patterns()
         c.svn_tag_patterns = model.get_global_svn_tag_patterns()
 
         c.svn_generate_config = rhodecode.ConfigGet().get_bool(config_keys.generate_config)
         c.svn_config_path = rhodecode.ConfigGet().get_str(config_keys.config_file_path)
         application_form = ApplicationUiSettingsForm(self.request.translate)()
 
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _("Some form inputs contain invalid data."),
                 category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.unpack_errors() or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         try:
-            model.update_global_ssl_setting(form_result['web_push_ssl'])
             model.update_global_hook_settings(form_result)
 
             model.create_or_update_global_svn_settings(form_result)
             model.create_or_update_global_hg_settings(form_result)
             model.create_or_update_global_git_settings(form_result)
             model.create_or_update_global_pr_settings(form_result)
         except Exception:
             log.exception("Exception while updating settings")
             h.flash(_('Error occurred during updating '
                       'application settings'), category='error')
         else:
             Session().commit()
             h.flash(_('Updated VCS settings'), category='success')
             raise HTTPFound(h.route_path('admin_settings_vcs'))
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_vcs_delete_svn_pattern(self):
         delete_pattern_id = self.request.POST.get('delete_svn_pattern')
         model = VcsSettingsModel()
         try:
             model.delete_global_svn_pattern(delete_pattern_id)
         except SettingNotFound:
             log.exception(
                 'Failed to delete svn_pattern with id %s', delete_pattern_id)
             raise HTTPNotFound()
 
         Session().commit()
         return True
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_mapping(self):
         c = self.load_default_context()
         c.active = 'mapping'
         c.storage_path = get_rhodecode_repo_store_path()
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_mapping_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'mapping'
         rm_obsolete = self.request.POST.get('destroy', False)
         invalidate_cache = self.request.POST.get('invalidate', False)
         log.debug('rescanning repo location with destroy obsolete=%s', rm_obsolete)
 
         if invalidate_cache:
             log.debug('invalidating all repositories cache')
             for repo in Repository.get_all():
                 ScmModel().mark_for_invalidation(repo.repo_name, delete=True)
 
         filesystem_repos = ScmModel().repo_scan()
         added, removed = repo2db_mapper(filesystem_repos, rm_obsolete, force_hooks_rebuild=True)
         PermissionModel().trigger_permission_flush()
 
         def _repr(rm_repo):
             return ', '.join(map(safe_str, rm_repo)) or '-'
 
         h.flash(_('Repositories successfully '
                   'rescanned added: %s ; removed: %s') %
                 (_repr(added), _repr(removed)),
                 category='success')
         raise HTTPFound(h.route_path('admin_settings_mapping'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_global(self):
         c = self.load_default_context()
         c.active = 'global'
         c.personal_repo_group_default_pattern = RepoGroupModel()\
             .get_personal_group_name_pattern()
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_global_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'global'
         c.personal_repo_group_default_pattern = RepoGroupModel()\
             .get_personal_group_name_pattern()
         application_form = ApplicationSettingsForm(self.request.translate)()
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _("Some form inputs contain invalid data."),
                 category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.unpack_errors() or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         settings = [
             ('title', 'rhodecode_title', 'unicode'),
             ('realm', 'rhodecode_realm', 'unicode'),
             ('pre_code', 'rhodecode_pre_code', 'unicode'),
             ('post_code', 'rhodecode_post_code', 'unicode'),
             ('captcha_public_key', 'rhodecode_captcha_public_key', 'unicode'),
             ('captcha_private_key', 'rhodecode_captcha_private_key', 'unicode'),
             ('create_personal_repo_group', 'rhodecode_create_personal_repo_group', 'bool'),
             ('personal_repo_group_pattern', 'rhodecode_personal_repo_group_pattern', 'unicode'),
         ]
 
         try:
             for setting, form_key, type_ in settings:
                 sett = SettingsModel().create_or_update_setting(
                     setting, form_result[form_key], type_)
                 Session().add(sett)
 
             Session().commit()
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated application settings'), category='success')
         except Exception:
             log.exception("Exception while updating application settings")
             h.flash(
                 _('Error occurred during updating application settings'),
                 category='error')
 
         raise HTTPFound(h.route_path('admin_settings_global'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_visual(self):
         c = self.load_default_context()
         c.active = 'visual'
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_visual_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'visual'
         application_form = ApplicationVisualisationForm(self.request.translate)()
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _("Some form inputs contain invalid data."),
                 category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.unpack_errors() or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         try:
             settings = [
                 ('show_public_icon', 'rhodecode_show_public_icon', 'bool'),
                 ('show_private_icon', 'rhodecode_show_private_icon', 'bool'),
                 ('stylify_metatags', 'rhodecode_stylify_metatags', 'bool'),
                 ('repository_fields', 'rhodecode_repository_fields', 'bool'),
                 ('dashboard_items', 'rhodecode_dashboard_items', 'int'),
                 ('admin_grid_items', 'rhodecode_admin_grid_items', 'int'),
                 ('show_version', 'rhodecode_show_version', 'bool'),
                 ('use_gravatar', 'rhodecode_use_gravatar', 'bool'),
                 ('markup_renderer', 'rhodecode_markup_renderer', 'unicode'),
                 ('gravatar_url', 'rhodecode_gravatar_url', 'unicode'),
                 ('clone_uri_tmpl', 'rhodecode_clone_uri_tmpl', 'unicode'),
                 ('clone_uri_id_tmpl', 'rhodecode_clone_uri_id_tmpl', 'unicode'),
                 ('clone_uri_ssh_tmpl', 'rhodecode_clone_uri_ssh_tmpl', 'unicode'),
                 ('support_url', 'rhodecode_support_url', 'unicode'),
                 ('show_revision_number', 'rhodecode_show_revision_number', 'bool'),
                 ('show_sha_length', 'rhodecode_show_sha_length', 'int'),
             ]
             for setting, form_key, type_ in settings:
                 sett = SettingsModel().create_or_update_setting(
                     setting, form_result[form_key], type_)
                 Session().add(sett)
 
             Session().commit()
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated visualisation settings'), category='success')
         except Exception:
             log.exception("Exception updating visualization settings")
             h.flash(_('Error occurred during updating '
                       'visualisation settings'),
                     category='error')
 
         raise HTTPFound(h.route_path('admin_settings_visual'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_issuetracker(self):
         c = self.load_default_context()
         c.active = 'issuetracker'
         defaults = c.rc_config
 
         entry_key = 'rhodecode_issuetracker_pat_'
 
         c.issuetracker_entries = {}
         for k, v in defaults.items():
             if k.startswith(entry_key):
                 uid = k[len(entry_key):]
                 c.issuetracker_entries[uid] = None
 
         for uid in c.issuetracker_entries:
             c.issuetracker_entries[uid] = AttributeDict({
                 'pat': defaults.get('rhodecode_issuetracker_pat_' + uid),
                 'url': defaults.get('rhodecode_issuetracker_url_' + uid),
                 'pref': defaults.get('rhodecode_issuetracker_pref_' + uid),
                 'desc': defaults.get('rhodecode_issuetracker_desc_' + uid),
             })
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_issuetracker_test(self):
         error_container = []
 
         urlified_commit = h.urlify_commit_message(
             self.request.POST.get('test_text', ''),
             'repo_group/test_repo1', error_container=error_container)
         if error_container:
             def converter(inp):
                 return h.html_escape(inp)
 
             return 'ERRORS: ' + '\n'.join(map(converter, error_container))
 
         return urlified_commit
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_issuetracker_update(self):
         _ = self.request.translate
         self.load_default_context()
         settings_model = IssueTrackerSettingsModel()
 
         try:
             form = IssueTrackerPatternsForm(self.request.translate)()
             data = form.to_python(self.request.POST)
         except formencode.Invalid as errors:
             log.exception('Failed to add new pattern')
             error = errors
             h.flash(_(f'Invalid issue tracker pattern: {error}'),
                     category='error')
             raise HTTPFound(h.route_path('admin_settings_issuetracker'))
 
         if data:
             for uid in data.get('delete_patterns', []):
                 settings_model.delete_entries(uid)
 
             for pattern in data.get('patterns', []):
                 for setting, value, type_ in pattern:
                     sett = settings_model.create_or_update_setting(
                         setting, value, type_)
                     Session().add(sett)
 
                 Session().commit()
 
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated issue tracker entries'), category='success')
         raise HTTPFound(h.route_path('admin_settings_issuetracker'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_issuetracker_delete(self):
         _ = self.request.translate
         self.load_default_context()
         uid = self.request.POST.get('uid')
         try:
             IssueTrackerSettingsModel().delete_entries(uid)
         except Exception:
             log.exception('Failed to delete issue tracker setting %s', uid)
             raise HTTPNotFound()
 
         SettingsModel().invalidate_settings_cache()
         h.flash(_('Removed issue tracker entry.'), category='success')
 
         return {'deleted': uid}
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_email(self):
         c = self.load_default_context()
         c.active = 'email'
         c.rhodecode_ini = rhodecode.CONFIG
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_email_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'email'
 
         test_email = self.request.POST.get('test_email')
 
         if not test_email:
             h.flash(_('Please enter email address'), category='error')
             raise HTTPFound(h.route_path('admin_settings_email'))
 
         email_kwargs = {
             'date': datetime.datetime.now(),
             'user': self._rhodecode_db_user
         }
 
         (subject, email_body, email_body_plaintext) = EmailNotificationModel().render_email(
             EmailNotificationModel.TYPE_EMAIL_TEST, **email_kwargs)
 
         recipients = [test_email] if test_email else None
 
         run_task(tasks.send_email, recipients, subject,
                  email_body_plaintext, email_body)
 
         h.flash(_('Send email task created'), category='success')
         raise HTTPFound(h.route_path('admin_settings_email'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_hooks(self):
         c = self.load_default_context()
         c.active = 'hooks'
 
         model = SettingsModel()
         c.hooks = model.get_builtin_hooks()
         c.custom_hooks = model.get_custom_hooks()
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_hooks_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'hooks'
         if c.visual.allow_custom_hooks_settings:
             ui_key = self.request.POST.get('new_hook_ui_key')
             ui_value = self.request.POST.get('new_hook_ui_value')
 
             hook_id = self.request.POST.get('hook_id')
             new_hook = False
 
             model = SettingsModel()
             try:
                 if ui_value and ui_key:
                     model.create_or_update_hook(ui_key, ui_value)
                     h.flash(_('Added new hook'), category='success')
                     new_hook = True
                 elif hook_id:
                     RhodeCodeUi.delete(hook_id)
                     Session().commit()
 
                 # check for edits
                 update = False
                 _d = self.request.POST.dict_of_lists()
                 for k, v in zip(_d.get('hook_ui_key', []),
                                 _d.get('hook_ui_value_new', [])):
                     model.create_or_update_hook(k, v)
                     update = True
 
                 if update and not new_hook:
                     h.flash(_('Updated hooks'), category='success')
                 Session().commit()
             except Exception:
                 log.exception("Exception during hook creation")
                 h.flash(_('Error occurred during hook creation'),
                         category='error')
 
         raise HTTPFound(h.route_path('admin_settings_hooks'))
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_search(self):
         c = self.load_default_context()
         c.active = 'search'
 
         c.searcher = searcher_from_config(self.request.registry.settings)
         c.statistics = c.searcher.statistics(self.request.translate)
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_labs(self):
         c = self.load_default_context()
         if not c.labs_active:
             raise HTTPFound(h.route_path('admin_settings'))
 
         c.active = 'labs'
         c.lab_settings = _LAB_SETTINGS
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     @CSRFRequired()
     def settings_labs_update(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'labs'
 
         application_form = LabsSettingsForm(self.request.translate)()
         try:
             form_result = application_form.to_python(dict(self.request.POST))
         except formencode.Invalid as errors:
             h.flash(
                 _("Some form inputs contain invalid data."),
                 category='error')
             data = render('rhodecode:templates/admin/settings/settings.mako',
                           self._get_template_context(c), self.request)
             html = formencode.htmlfill.render(
                 data,
                 defaults=errors.value,
                 errors=errors.unpack_errors() or {},
                 prefix_error=False,
                 encoding="UTF-8",
                 force_defaults=False
             )
             return Response(html)
 
         try:
             session = Session()
             for setting in _LAB_SETTINGS:
                 setting_name = setting.key[len('rhodecode_'):]
                 sett = SettingsModel().create_or_update_setting(
                     setting_name, form_result[setting.key], setting.type)
                 session.add(sett)
 
         except Exception:
             log.exception('Exception while updating lab settings')
             h.flash(_('Error occurred during updating labs settings'),
                     category='error')
         else:
             Session().commit()
             SettingsModel().invalidate_settings_cache()
             h.flash(_('Updated Labs settings'), category='success')
             raise HTTPFound(h.route_path('admin_settings_labs'))
 
         data = render('rhodecode:templates/admin/settings/settings.mako',
                       self._get_template_context(c), self.request)
         html = formencode.htmlfill.render(
             data,
             defaults=self._form_defaults(),
             encoding="UTF-8",
             force_defaults=False
         )
         return Response(html)
 
 
 # :param key: name of the setting including the 'rhodecode_' prefix
 # :param type: the RhodeCodeSetting type to use.
 # :param group: the i18ned group in which we should dispaly this setting
 # :param label: the i18ned label we should display for this setting
 # :param help: the i18ned help we should dispaly for this setting
 LabSetting = collections.namedtuple(
     'LabSetting', ('key', 'type', 'group', 'label', 'help'))
 
 
 # This list has to be kept in sync with the form
 # rhodecode.model.forms.LabsSettingsForm.
 _LAB_SETTINGS = [
 
 ]

rhodecode/apps/admin/views/system_info.py

@@ -1,243 +1,249 b''
 
 
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 import urllib.request
 import urllib.error
 import urllib.parse
 import os
 
 import rhodecode
 from rhodecode.apps._base import BaseAppView
 from rhodecode.apps._base.navigation import navigation_list
 from rhodecode.lib import helpers as h
 from rhodecode.lib.auth import (LoginRequired, HasPermissionAllDecorator)
 from rhodecode.lib.utils2 import str2bool
 from rhodecode.lib import system_info
 from rhodecode.model.update import UpdateModel
 
 log = logging.getLogger(__name__)
 
 
 class AdminSystemInfoSettingsView(BaseAppView):
     def load_default_context(self):
         c = self._get_local_tmpl_context()
         return c
 
     def get_env_data(self):
         black_list = [
             'NIX_LDFLAGS',
             'NIX_CFLAGS_COMPILE',
             'propagatedBuildInputs',
             'propagatedNativeBuildInputs',
             'postInstall',
             'buildInputs',
             'buildPhase',
             'preShellHook',
             'preShellHook',
             'preCheck',
             'preBuild',
             'postShellHook',
             'postFixup',
             'postCheck',
             'nativeBuildInputs',
             'installPhase',
             'installCheckPhase',
             'checkPhase',
             'configurePhase',
             'shellHook'
         ]
         secret_list = [
             'RHODECODE_USER_PASS'
         ]
 
         for k, v in sorted(os.environ.items()):
             if k in black_list:
                 continue
             if k in secret_list:
                 v = '*****'
             yield k, v
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_system_info(self):
         _ = self.request.translate
         c = self.load_default_context()
 
         c.active = 'system'
         c.navlist = navigation_list(self.request)
 
         # TODO(marcink), figure out how to allow only selected users to do this
         c.allowed_to_snapshot = self._rhodecode_user.admin
 
         snapshot = str2bool(self.request.params.get('snapshot'))
 
         c.rhodecode_update_url = UpdateModel().get_update_url()
         c.env_data = self.get_env_data()
         server_info = system_info.get_system_info(self.request.environ)
 
         for key, val in server_info.items():
             setattr(c, key, val)
 
         def val(name, subkey='human_value'):
             return server_info[name][subkey]
 
         def state(name):
             return server_info[name]['state']
 
         def val2(name):
             val = server_info[name]['human_value']
             state = server_info[name]['state']
             return val, state
 
         update_info_msg = _('Note: please make sure this server can '
                             'access `${url}` for the update link to work',
                             mapping=dict(url=c.rhodecode_update_url))
         version = UpdateModel().get_stored_version()
         is_outdated = UpdateModel().is_outdated(
             rhodecode.__version__, version)
         update_state = {
             'type': 'warning',
             'message': 'New version available: {}'.format(version)
             } \
             if is_outdated else {}
         c.data_items = [
             # update info
             (_('Update info'), h.literal(
                 '<span class="link" id="check_for_update" >%s.</span>' % (
                 _('Check for updates')) +
                 '<br/> <span >%s.</span>' % (update_info_msg)
             ), ''),
 
             # RhodeCode specific
             (_('RhodeCode Version'), val('rhodecode_app')['text'], state('rhodecode_app')),
             (_('Latest version'), version, update_state),
             (_('RhodeCode Base URL'), val('rhodecode_config')['config'].get('app.base_url'), state('rhodecode_config')),
             (_('RhodeCode Server IP'), val('server')['server_ip'], state('server')),
             (_('RhodeCode Server ID'), val('server')['server_id'], state('server')),
             (_('RhodeCode Configuration'), val('rhodecode_config')['path'], state('rhodecode_config')),
             (_('RhodeCode Certificate'), val('rhodecode_config')['cert_path'], state('rhodecode_config')),
             (_('Workers'), val('rhodecode_config')['config']['server:main'].get('workers', '?'), state('rhodecode_config')),
             (_('Worker Type'), val('rhodecode_config')['config']['server:main'].get('worker_class', 'sync'), state('rhodecode_config')),
             ('', '', ''),  # spacer
 
             # Database
             (_('Database'), val('database')['url'], state('database')),
             (_('Database version'), val('database')['version'], state('database')),
             ('', '', ''),  # spacer
 
             # Platform/Python
             (_('Platform'), val('platform')['name'], state('platform')),
             (_('Platform UUID'), val('platform')['uuid'], state('platform')),
             (_('Lang'), val('locale'), state('locale')),
             (_('Python version'), val('python')['version'], state('python')),
             (_('Python path'), val('python')['executable'], state('python')),
             ('', '', ''),  # spacer
 
             # Systems stats
             (_('CPU'), val('cpu')['text'], state('cpu')),
             (_('Load'), val('load')['text'], state('load')),
             (_('Memory'), val('memory')['text'], state('memory')),
             (_('Uptime'), val('uptime')['text'], state('uptime')),
             ('', '', ''),  # spacer
 
             # ulimit
             (_('Ulimit'), val('ulimit')['text'], state('ulimit')),
 
             # Repo storage
             (_('Storage location'), val('storage')['path'], state('storage')),
             (_('Storage info'), val('storage')['text'], state('storage')),
             (_('Storage inodes'), val('storage_inodes')['text'], state('storage_inodes')),
             ('', '', ''),  # spacer
 
             (_('Gist storage location'), val('storage_gist')['path'], state('storage_gist')),
             (_('Gist storage info'), val('storage_gist')['text'], state('storage_gist')),
             ('', '', ''),  # spacer
 
-            (_('Archive cache storage type'), val('storage_archive')['type'], state('storage_archive')),
+            (_('Artifacts storage backend'), val('storage_artifacts')['type'], state('storage_artifacts')),
+            (_('Artifacts storage location'), val('storage_artifacts')['path'], state('storage_artifacts')),
+            (_('Artifacts info'), val('storage_artifacts')['text'], state('storage_artifacts')),
+            ('', '', ''),  # spacer
+
+            (_('Archive cache storage backend'), val('storage_archive')['type'], state('storage_archive')),
             (_('Archive cache storage location'), val('storage_archive')['path'], state('storage_archive')),
             (_('Archive cache info'), val('storage_archive')['text'], state('storage_archive')),
             ('', '', ''),  # spacer
 
+
             (_('Temp storage location'), val('storage_temp')['path'], state('storage_temp')),
             (_('Temp storage info'), val('storage_temp')['text'], state('storage_temp')),
             ('', '', ''),  # spacer
 
             (_('Search info'), val('search')['text'], state('search')),
             (_('Search location'), val('search')['location'], state('search')),
             ('', '', ''),  # spacer
 
             # VCS specific
             (_('VCS Backends'), val('vcs_backends'), state('vcs_backends')),
             (_('VCS Server'), val('vcs_server')['text'], state('vcs_server')),
             (_('GIT'), val('git'), state('git')),
             (_('HG'), val('hg'), state('hg')),
             (_('SVN'), val('svn'), state('svn')),
 
         ]
 
         c.vcsserver_data_items = [
             (k, v) for k, v in (val('vcs_server_config') or {}).items()
         ]
 
         if snapshot:
             if c.allowed_to_snapshot:
                 c.data_items.pop(0)  # remove server info
                 self.request.override_renderer = 'admin/settings/settings_system_snapshot.mako'
             else:
                 h.flash('You are not allowed to do this', category='warning')
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasPermissionAllDecorator('hg.admin')
     def settings_system_info_check_update(self):
         _ = self.request.translate
         c = self.load_default_context()
 
         update_url = UpdateModel().get_update_url()
 
         def _err(s):
             return f'<div style="color:#ff8888; padding:4px 0px">{s}</div>'
 
         try:
             data = UpdateModel().get_update_data(update_url)
         except urllib.error.URLError as e:
             log.exception("Exception contacting upgrade server")
             self.request.override_renderer = 'string'
             return _err('Failed to contact upgrade server: %r' % e)
         except ValueError as e:
             log.exception("Bad data sent from update server")
             self.request.override_renderer = 'string'
             return _err('Bad data sent from update server')
 
         latest = data['versions'][0]
 
         c.update_url = update_url
         c.latest_data = latest
         c.latest_ver = (latest['version'] or '').strip()
         c.cur_ver = self.request.GET.get('ver') or rhodecode.__version__
         c.should_upgrade = False
 
         is_outdated = UpdateModel().is_outdated(c.cur_ver, c.latest_ver)
         if is_outdated:
             c.should_upgrade = True
         c.important_notices = latest['general']
         UpdateModel().store_version(latest['version'])
         return self._get_template_context(c)

rhodecode/apps/file_store/__init__.py

@@ -1,66 +1,97 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 import os
-from rhodecode.apps.file_store import config_keys
+
+
 from rhodecode.config.settings_maker import SettingsMaker
 
 
 def _sanitize_settings_and_apply_defaults(settings):
     """
     Set defaults, convert to python types and validate settings.
     """
+    from rhodecode.apps.file_store import config_keys
+
+    # translate "legacy" params into new config
+    settings.pop(config_keys.deprecated_enabled, True)
+    if config_keys.deprecated_backend in settings:
+        # if legacy backend key is detected we use "legacy" backward compat setting
+        settings.pop(config_keys.deprecated_backend)
+        settings[config_keys.backend_type] = config_keys.backend_legacy_filesystem
+
+    if config_keys.deprecated_store_path in settings:
+        store_path = settings.pop(config_keys.deprecated_store_path)
+        settings[config_keys.legacy_filesystem_storage_path] = store_path
+
     settings_maker = SettingsMaker(settings)
 
-    settings_maker.make_setting(config_keys.enabled, True, parser='bool')
-    settings_maker.make_setting(config_keys.backend, 'local')
+    default_cache_dir = settings['cache_dir']
+    default_store_dir = os.path.join(default_cache_dir, 'artifacts_filestore')
+
+    # set default backend
+    settings_maker.make_setting(config_keys.backend_type, config_keys.backend_legacy_filesystem)
+
+    # legacy filesystem defaults
+    settings_maker.make_setting(config_keys.legacy_filesystem_storage_path, default_store_dir, default_when_empty=True, )
 
-    default_store = os.path.join(os.path.dirname(settings['__file__']), 'upload_store')
-    settings_maker.make_setting(config_keys.store_path, default_store)
+    # filesystem defaults
+    settings_maker.make_setting(config_keys.filesystem_storage_path, default_store_dir, default_when_empty=True,)
+    settings_maker.make_setting(config_keys.filesystem_shards, 8, parser='int')
+
+    # objectstore defaults
+    settings_maker.make_setting(config_keys.objectstore_url, 'http://s3-minio:9000')
+    settings_maker.make_setting(config_keys.objectstore_bucket, 'rhodecode-artifacts-filestore')
+    settings_maker.make_setting(config_keys.objectstore_bucket_shards, 8, parser='int')
+
+    settings_maker.make_setting(config_keys.objectstore_region, '')
+    settings_maker.make_setting(config_keys.objectstore_key, '')
+    settings_maker.make_setting(config_keys.objectstore_secret, '')
 
     settings_maker.env_expand()
 
 
 def includeme(config):
+
     from rhodecode.apps.file_store.views import FileStoreView
 
     settings = config.registry.settings
     _sanitize_settings_and_apply_defaults(settings)
 
     config.add_route(
         name='upload_file',
         pattern='/_file_store/upload')
     config.add_view(
         FileStoreView,
         attr='upload_file',
         route_name='upload_file', request_method='POST', renderer='json_ext')
 
     config.add_route(
         name='download_file',
         pattern='/_file_store/download/{fid:.*}')
     config.add_view(
         FileStoreView,
         attr='download_file',
         route_name='download_file')
 
     config.add_route(
         name='download_file_by_token',
         pattern='/_file_store/token-download/{_auth_token}/{fid:.*}')
     config.add_view(
         FileStoreView,
         attr='download_file_by_token',
         route_name='download_file_by_token')

rhodecode/apps/file_store/config_keys.py

@@ -1,25 +1,57 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 # Definition of setting keys used to configure this module. Defined here to
 # avoid repetition of keys throughout the module.
 
-enabled = 'file_store.enabled'
-backend = 'file_store.backend'
-store_path = 'file_store.storage_path'
+# OLD and deprecated keys not used anymore
+deprecated_enabled = 'file_store.enabled'
+deprecated_backend = 'file_store.backend'
+deprecated_store_path = 'file_store.storage_path'
+
+
+backend_type = 'file_store.backend.type'
+
+backend_legacy_filesystem = 'filesystem_v1'
+backend_filesystem = 'filesystem_v2'
+backend_objectstore = 'objectstore'
+
+backend_types = [
+    backend_legacy_filesystem,
+    backend_filesystem,
+    backend_objectstore,
+]
+
+# filesystem_v1 legacy
+legacy_filesystem_storage_path = 'file_store.filesystem_v1.storage_path'
+
+
+# filesystem_v2 new option
+filesystem_storage_path = 'file_store.filesystem_v2.storage_path'
+filesystem_shards = 'file_store.filesystem_v2.shards'
+
+# objectstore
+objectstore_url = 'file_store.objectstore.url'
+objectstore_bucket = 'file_store.objectstore.bucket'
+objectstore_bucket_shards = 'file_store.objectstore.bucket_shards'
+
+objectstore_region = 'file_store.objectstore.region'
+objectstore_key = 'file_store.objectstore.key'
+objectstore_secret = 'file_store.objectstore.secret'
+

rhodecode/apps/file_store/tests/__init__.py

@@ -1,18 +1,57 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
+import os
+import random
+import tempfile
+import string
+
+import pytest
+
+from rhodecode.apps.file_store import utils as store_utils
+
+
[e-mail]()
+def file_store_instance(ini_settings):
+    config = ini_settings
+    f_store = store_utils.get_filestore_backend(config=config, always_init=True)
+    return f_store
+
+
[e-mail]
+def random_binary_file():
+    # Generate random binary data
+    data = bytearray(random.getrandbits(8) for _ in range(1024 * 512))  # 512 KB of random data
+
+    # Create a temporary file
+    temp_file = tempfile.NamedTemporaryFile(delete=False)
+    filename = temp_file.name
+
+    try:
+        # Write the random binary data to the file
+        temp_file.write(data)
+        temp_file.seek(0)  # Rewind the file pointer to the beginning
+        yield filename, temp_file
+    finally:
+        # Close and delete the temporary file after the test
+        temp_file.close()
+        os.remove(filename)
+
+
+def generate_random_filename(length=10):
+    return ''.join(random.choices(string.ascii_letters + string.digits, k=length))
  No newline at end of file

rhodecode/apps/file_store/tests/test_upload_file.py

@@ -1,246 +1,253 b''
 # Copyright (C) 2010-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
+
 import os
+
 import pytest
 
 from rhodecode.lib.ext_json import json
 from rhodecode.model.auth_token import AuthTokenModel
 from rhodecode.model.db import Session, FileStore, Repository, User
-from rhodecode.apps.file_store import utils, config_keys
+from rhodecode.apps.file_store import utils as store_utils
+from rhodecode.apps.file_store import config_keys
 
 from rhodecode.tests import TestController
 from rhodecode.tests.routes import route_path
 
 
 class TestFileStoreViews(TestController):
 
+    @pytest.fixture()
+    def create_artifact_factory(self, tmpdir, ini_settings):
+
+        def factory(user_id, content, f_name='example.txt'):
+
+            config = ini_settings
+            config[config_keys.backend_type] = config_keys.backend_legacy_filesystem
+
+            f_store = store_utils.get_filestore_backend(config)
+
+            filesystem_file = os.path.join(str(tmpdir), f_name)
+            with open(filesystem_file, 'wt') as f:
+                f.write(content)
+
+            with open(filesystem_file, 'rb') as f:
+                store_uid, metadata = f_store.store(f_name, f, metadata={'filename': f_name})
+            os.remove(filesystem_file)
+
+            entry = FileStore.create(
+                file_uid=store_uid, filename=metadata["filename"],
+                file_hash=metadata["sha256"], file_size=metadata["size"],
+                file_display_name='file_display_name',
+                file_description='repo artifact `{}`'.format(metadata["filename"]),
+                check_acl=True, user_id=user_id,
+            )
+            Session().add(entry)
+            Session().commit()
+            return entry
+        return factory
+
     @pytest.mark.parametrize("fid, content, exists", [
         ('abcde-0.jpg', "xxxxx", True),
         ('abcde-0.exe', "1234567", True),
         ('abcde-0.jpg', "xxxxx", False),
     ])
-    def test_get_files_from_store(self, fid, content, exists, tmpdir, user_util):
+    def test_get_files_from_store(self, fid, content, exists, tmpdir, user_util, ini_settings):
         user = self.log_user()
         user_id = user['user_id']
         repo_id = user_util.create_repo().repo_id
-        store_path = self.app._pyramid_settings[config_keys.store_path]
+
+        config = ini_settings
+        config[config_keys.backend_type] = config_keys.backend_legacy_filesystem
+
         store_uid = fid
 
         if exists:
             status = 200
-            store = utils.get_file_storage({config_keys.store_path: store_path})
+            f_store = store_utils.get_filestore_backend(config)
             filesystem_file = os.path.join(str(tmpdir), fid)
             with open(filesystem_file, 'wt') as f:
                 f.write(content)
 
             with open(filesystem_file, 'rb') as f:
-                store_uid, metadata = store.save_file(f, fid, extra_metadata={'filename': fid})
+                store_uid, metadata = f_store.store(fid, f, metadata={'filename': fid})
+            os.remove(filesystem_file)
 
             entry = FileStore.create(
                 file_uid=store_uid, filename=metadata["filename"],
                 file_hash=metadata["sha256"], file_size=metadata["size"],
                 file_display_name='file_display_name',
                 file_description='repo artifact `{}`'.format(metadata["filename"]),
                 check_acl=True, user_id=user_id,
                 scope_repo_id=repo_id
             )
             Session().add(entry)
             Session().commit()
 
         else:
             status = 404
 
         response = self.app.get(route_path('download_file', fid=store_uid), status=status)
 
         if exists:
             assert response.text == content
-            file_store_path = os.path.dirname(store.resolve_name(store_uid, store_path)[1])
-            metadata_file = os.path.join(file_store_path, store_uid + '.meta')
-            assert os.path.exists(metadata_file)
-            with open(metadata_file, 'rb') as f:
-                json_data = json.loads(f.read())
 
-            assert json_data
-            assert 'size' in json_data
+            metadata = f_store.get_metadata(store_uid)
+
+            assert 'size' in metadata
 
     def test_upload_files_without_content_to_store(self):
         self.log_user()
         response = self.app.post(
             route_path('upload_file'),
             params={'csrf_token': self.csrf_token},
             status=200)
 
         assert response.json == {
             'error': 'store_file data field is missing',
             'access_path': None,
             'store_fid': None}
 
     def test_upload_files_bogus_content_to_store(self):
         self.log_user()
         response = self.app.post(
             route_path('upload_file'),
             params={'csrf_token': self.csrf_token, 'store_file': 'bogus'},
             status=200)
 
         assert response.json == {
             'error': 'filename cannot be read from the data field',
             'access_path': None,
             'store_fid': None}
 
     def test_upload_content_to_store(self):
         self.log_user()
         response = self.app.post(
             route_path('upload_file'),
             upload_files=[('store_file', b'myfile.txt', b'SOME CONTENT')],
             params={'csrf_token': self.csrf_token},
             status=200)
 
         assert response.json['store_fid']
 
-    @pytest.fixture()
-    def create_artifact_factory(self, tmpdir):
-        def factory(user_id, content):
-            store_path = self.app._pyramid_settings[config_keys.store_path]
-            store = utils.get_file_storage({config_keys.store_path: store_path})
-            fid = 'example.txt'
-
-            filesystem_file = os.path.join(str(tmpdir), fid)
-            with open(filesystem_file, 'wt') as f:
-                f.write(content)
-
-            with open(filesystem_file, 'rb') as f:
-                store_uid, metadata = store.save_file(f, fid, extra_metadata={'filename': fid})
-
-            entry = FileStore.create(
-                file_uid=store_uid, filename=metadata["filename"],
-                file_hash=metadata["sha256"], file_size=metadata["size"],
-                file_display_name='file_display_name',
-                file_description='repo artifact `{}`'.format(metadata["filename"]),
-                check_acl=True, user_id=user_id,
-            )
-            Session().add(entry)
-            Session().commit()
-            return entry
-        return factory
-
     def test_download_file_non_scoped(self, user_util, create_artifact_factory):
         user = self.log_user()
         user_id = user['user_id']
         content = 'HELLO MY NAME IS ARTIFACT !'
 
         artifact = create_artifact_factory(user_id, content)
         file_uid = artifact.file_uid
         response = self.app.get(route_path('download_file', fid=file_uid), status=200)
         assert response.text == content
 
         # log-in to new user and test download again
         user = user_util.create_user(password='qweqwe')
         self.log_user(user.username, 'qweqwe')
         response = self.app.get(route_path('download_file', fid=file_uid), status=200)
         assert response.text == content
 
     def test_download_file_scoped_to_repo(self, user_util, create_artifact_factory):
         user = self.log_user()
         user_id = user['user_id']
         content = 'HELLO MY NAME IS ARTIFACT !'
 
         artifact = create_artifact_factory(user_id, content)
         # bind to repo
         repo = user_util.create_repo()
         repo_id = repo.repo_id
         artifact.scope_repo_id = repo_id
         Session().add(artifact)
         Session().commit()
 
         file_uid = artifact.file_uid
         response = self.app.get(route_path('download_file', fid=file_uid), status=200)
         assert response.text == content
 
         # log-in to new user and test download again
         user = user_util.create_user(password='qweqwe')
         self.log_user(user.username, 'qweqwe')
         response = self.app.get(route_path('download_file', fid=file_uid), status=200)
         assert response.text == content
 
         # forbid user the rights to repo
         repo = Repository.get(repo_id)
         user_util.grant_user_permission_to_repo(repo, user, 'repository.none')
         self.app.get(route_path('download_file', fid=file_uid), status=404)
 
     def test_download_file_scoped_to_user(self, user_util, create_artifact_factory):
         user = self.log_user()
         user_id = user['user_id']
         content = 'HELLO MY NAME IS ARTIFACT !'
 
         artifact = create_artifact_factory(user_id, content)
         # bind to user
         user = user_util.create_user(password='qweqwe')
 
         artifact.scope_user_id = user.user_id
         Session().add(artifact)
         Session().commit()
 
         # artifact creator doesn't have access since it's bind to another user
         file_uid = artifact.file_uid
         self.app.get(route_path('download_file', fid=file_uid), status=404)
 
         # log-in to new user and test download again, should be ok since we're bind to this artifact
         self.log_user(user.username, 'qweqwe')
         response = self.app.get(route_path('download_file', fid=file_uid), status=200)
         assert response.text == content
 
     def test_download_file_scoped_to_repo_with_bad_token(self, user_util, create_artifact_factory):
         user_id = User.get_first_super_admin().user_id
         content = 'HELLO MY NAME IS ARTIFACT !'
 
         artifact = create_artifact_factory(user_id, content)
         # bind to repo
         repo = user_util.create_repo()
         repo_id = repo.repo_id
         artifact.scope_repo_id = repo_id
         Session().add(artifact)
         Session().commit()
 
         file_uid = artifact.file_uid
         self.app.get(route_path('download_file_by_token',
                                 _auth_token='bogus', fid=file_uid), status=302)
 
     def test_download_file_scoped_to_repo_with_token(self, user_util, create_artifact_factory):
         user = User.get_first_super_admin()
         AuthTokenModel().create(user, 'test artifact token',
                                 role=AuthTokenModel.cls.ROLE_ARTIFACT_DOWNLOAD)
 
         user = User.get_first_super_admin()
         artifact_token = user.artifact_token
 
         user_id = User.get_first_super_admin().user_id
         content = 'HELLO MY NAME IS ARTIFACT !'
 
         artifact = create_artifact_factory(user_id, content)
         # bind to repo
         repo = user_util.create_repo()
         repo_id = repo.repo_id
         artifact.scope_repo_id = repo_id
         Session().add(artifact)
         Session().commit()
 
         file_uid = artifact.file_uid
         response = self.app.get(
             route_path('download_file_by_token',
                        _auth_token=artifact_token, fid=file_uid), status=200)
         assert response.text == content

rhodecode/apps/file_store/utils.py

@@ -1,55 +1,145 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import io
 import uuid
 import pathlib
+import s3fs
+
+from rhodecode.lib.hash_utils import sha256_safe
+from rhodecode.apps.file_store import config_keys
+
+
+file_store_meta = None
+
+
+def get_filestore_config(config) -> dict:
+
+    final_config = {}
+
+    for k, v in config.items():
+        if k.startswith('file_store'):
+            final_config[k] = v
+
+    return final_config
 
 
-def get_file_storage(settings):
-    from rhodecode.apps.file_store.backends.local_store import LocalFileStorage
-    from rhodecode.apps.file_store import config_keys
-    store_path = settings.get(config_keys.store_path)
-    return LocalFileStorage(base_path=store_path)
+def get_filestore_backend(config, always_init=False):
+    """
+
+    usage::
+        from rhodecode.apps.file_store import get_filestore_backend
+        f_store = get_filestore_backend(config=CONFIG)
+
+    :param config:
+    :param always_init:
+    :return:
+    """
+
+    global file_store_meta
+    if file_store_meta is not None and not always_init:
+        return file_store_meta
+
+    config = get_filestore_config(config)
+    backend = config[config_keys.backend_type]
+
+    match backend:
+        case config_keys.backend_legacy_filesystem:
+            # Legacy backward compatible storage
+            from rhodecode.apps.file_store.backends.filesystem_legacy import LegacyFileSystemBackend
+            d_cache = LegacyFileSystemBackend(
+                settings=config
+            )
+        case config_keys.backend_filesystem:
+            from rhodecode.apps.file_store.backends.filesystem import FileSystemBackend
+            d_cache = FileSystemBackend(
+                settings=config
+            )
+        case config_keys.backend_objectstore:
+            from rhodecode.apps.file_store.backends.objectstore import ObjectStoreBackend
+            d_cache = ObjectStoreBackend(
+                settings=config
+            )
+        case _:
+            raise ValueError(
+                f'file_store.backend.type only supports "{config_keys.backend_types}" got {backend}'
+            )
+
+    cache_meta = d_cache
+    return cache_meta
 
 
 def splitext(filename):
-    ext = ''.join(pathlib.Path(filename).suffixes)
+    final_ext = []
+    for suffix in pathlib.Path(filename).suffixes:
+        if not suffix.isascii():
+            continue
+
+        suffix = " ".join(suffix.split()).replace(" ", "")
+        final_ext.append(suffix)
+    ext = ''.join(final_ext)
     return filename, ext
 
 
-def uid_filename(filename, randomized=True):
+def get_uid_filename(filename, randomized=True):
     """
     Generates a randomized or stable (uuid) filename,
     preserving the original extension.
 
     :param filename: the original filename
     :param randomized: define if filename should be stable (sha1 based) or randomized
     """
 
     _, ext = splitext(filename)
     if randomized:
         uid = uuid.uuid4()
     else:
-        hash_key = '{}.{}'.format(filename, 'store')
+        store_suffix = "store"
+        hash_key = f'{filename}.{store_suffix}'
         uid = uuid.uuid5(uuid.NAMESPACE_URL, hash_key)
     return str(uid) + ext.lower()
 
 
 def bytes_to_file_obj(bytes_data):
-    return io.StringIO(bytes_data)
+    return io.BytesIO(bytes_data)
+
+
+class ShardFileReader:
+
+    def __init__(self, file_like_reader):
+        self._file_like_reader = file_like_reader
+
+    def __getattr__(self, item):
+        if isinstance(self._file_like_reader, s3fs.core.S3File):
+            match item:
+                case 'name':
+                    # S3 FileWrapper doesn't support name attribute, and we use it
+                    return self._file_like_reader.full_name
+                case _:
+                    return getattr(self._file_like_reader, item)
+        else:
+            return getattr(self._file_like_reader, item)
+
+
+def archive_iterator(_reader, block_size: int = 4096 * 512):
+    # 4096 * 64 = 64KB
+    while 1:
+        data = _reader.read(block_size)
+        if not data:
+            break
+        yield data

rhodecode/apps/file_store/views.py

@@ -1,200 +1,197 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 import logging
 
-
-from pyramid.response import FileResponse
+from pyramid.response import Response
 from pyramid.httpexceptions import HTTPFound, HTTPNotFound
 
 from rhodecode.apps._base import BaseAppView
-from rhodecode.apps.file_store import utils
+from rhodecode.apps.file_store import utils as store_utils
 from rhodecode.apps.file_store.exceptions import (
     FileNotAllowedException, FileOverSizeException)
 
 from rhodecode.lib import helpers as h
 from rhodecode.lib import audit_logger
 from rhodecode.lib.auth import (
     CSRFRequired, NotAnonymous, HasRepoPermissionAny, HasRepoGroupPermissionAny,
     LoginRequired)
+from rhodecode.lib.str_utils import header_safe_str
 from rhodecode.lib.vcs.conf.mtypes import get_mimetypes_db
 from rhodecode.model.db import Session, FileStore, UserApiKeys
 
 log = logging.getLogger(__name__)
 
 
 class FileStoreView(BaseAppView):
     upload_key = 'store_file'
 
     def load_default_context(self):
         c = self._get_local_tmpl_context()
-        self.storage = utils.get_file_storage(self.request.registry.settings)
+        self.f_store = store_utils.get_filestore_backend(self.request.registry.settings)
         return c
 
     def _guess_type(self, file_name):
         """
         Our own type guesser for mimetypes using the rich DB
         """
         if not hasattr(self, 'db'):
             self.db = get_mimetypes_db()
         _content_type, _encoding = self.db.guess_type(file_name, strict=False)
         return _content_type, _encoding
 
     def _serve_file(self, file_uid):
-        if not self.storage.exists(file_uid):
-            store_path = self.storage.store_path(file_uid)
-            log.debug('File with FID:%s not found in the store under `%s`',
+        if not self.f_store.filename_exists(file_uid):
+            store_path = self.f_store.store_path(file_uid)
+            log.warning('File with FID:%s not found in the store under `%s`',
                       file_uid, store_path)
             raise HTTPNotFound()
 
         db_obj = FileStore.get_by_store_uid(file_uid, safe=True)
         if not db_obj:
             raise HTTPNotFound()
 
         # private upload for user
         if db_obj.check_acl and db_obj.scope_user_id:
             log.debug('Artifact: checking scope access for bound artifact user: `%s`',
                       db_obj.scope_user_id)
             user = db_obj.user
             if self._rhodecode_db_user.user_id != user.user_id:
                 log.warning('Access to file store object forbidden')
                 raise HTTPNotFound()
 
         # scoped to repository permissions
         if db_obj.check_acl and db_obj.scope_repo_id:
             log.debug('Artifact: checking scope access for bound artifact repo: `%s`',
                       db_obj.scope_repo_id)
             repo = db_obj.repo
             perm_set = ['repository.read', 'repository.write', 'repository.admin']
             has_perm = HasRepoPermissionAny(*perm_set)(repo.repo_name, 'FileStore check')
             if not has_perm:
                 log.warning('Access to file store object `%s` forbidden', file_uid)
                 raise HTTPNotFound()
 
         # scoped to repository group permissions
         if db_obj.check_acl and db_obj.scope_repo_group_id:
             log.debug('Artifact: checking scope access for bound artifact repo group: `%s`',
                       db_obj.scope_repo_group_id)
             repo_group = db_obj.repo_group
             perm_set = ['group.read', 'group.write', 'group.admin']
             has_perm = HasRepoGroupPermissionAny(*perm_set)(repo_group.group_name, 'FileStore check')
             if not has_perm:
                 log.warning('Access to file store object `%s` forbidden', file_uid)
                 raise HTTPNotFound()
 
         FileStore.bump_access_counter(file_uid)
 
-        file_path = self.storage.store_path(file_uid)
+        file_name = db_obj.file_display_name
         content_type = 'application/octet-stream'
-        content_encoding = None
 
-        _content_type, _encoding = self._guess_type(file_path)
+        _content_type, _encoding = self._guess_type(file_name)
         if _content_type:
             content_type = _content_type
 
         # For file store we don't submit any session data, this logic tells the
         # Session lib to skip it
         setattr(self.request, '_file_response', True)
-        response = FileResponse(
-            file_path, request=self.request,
-            content_type=content_type, content_encoding=content_encoding)
+        reader, _meta = self.f_store.fetch(file_uid)
 
-        file_name = db_obj.file_display_name
+        response = Response(app_iter=store_utils.archive_iterator(reader))
 
-        response.headers["Content-Disposition"] = (
-            f'attachment; filename="{str(file_name)}"'
-        )
+        response.content_type = str(content_type)
+        response.content_disposition = f'attachment; filename="{header_safe_str(file_name)}"'
+
         response.headers["X-RC-Artifact-Id"] = str(db_obj.file_store_id)
-        response.headers["X-RC-Artifact-Desc"] = str(db_obj.file_description)
+        response.headers["X-RC-Artifact-Desc"] = header_safe_str(db_obj.file_description)
         response.headers["X-RC-Artifact-Sha256"] = str(db_obj.file_hash)
         return response
 
     @LoginRequired()
     @NotAnonymous()
     @CSRFRequired()
     def upload_file(self):
         self.load_default_context()
         file_obj = self.request.POST.get(self.upload_key)
 
         if file_obj is None:
             return {'store_fid': None,
                     'access_path': None,
                     'error': f'{self.upload_key} data field is missing'}
 
         if not hasattr(file_obj, 'filename'):
             return {'store_fid': None,
                     'access_path': None,
                     'error': 'filename cannot be read from the data field'}
 
         filename = file_obj.filename
 
         metadata = {
             'user_uploaded': {'username': self._rhodecode_user.username,
                               'user_id': self._rhodecode_user.user_id,
                               'ip': self._rhodecode_user.ip_addr}}
         try:
-            store_uid, metadata = self.storage.save_file(
-                file_obj.file, filename, extra_metadata=metadata)
+            store_uid, metadata = self.f_store.store(
+                filename, file_obj.file, extra_metadata=metadata)
         except FileNotAllowedException:
             return {'store_fid': None,
                     'access_path': None,
                     'error': f'File {filename} is not allowed.'}
 
         except FileOverSizeException:
             return {'store_fid': None,
                     'access_path': None,
                     'error': f'File {filename} is exceeding allowed limit.'}
 
         try:
             entry = FileStore.create(
                 file_uid=store_uid, filename=metadata["filename"],
                 file_hash=metadata["sha256"], file_size=metadata["size"],
                 file_description='upload attachment',
                 check_acl=False, user_id=self._rhodecode_user.user_id
             )
             Session().add(entry)
             Session().commit()
             log.debug('Stored upload in DB as %s', entry)
         except Exception:
             log.exception('Failed to store file %s', filename)
             return {'store_fid': None,
                     'access_path': None,
                     'error': f'File {filename} failed to store in DB.'}
 
         return {'store_fid': store_uid,
                 'access_path': h.route_path('download_file', fid=store_uid)}
 
     # ACL is checked by scopes, if no scope the file is accessible to all
     def download_file(self):
         self.load_default_context()
         file_uid = self.request.matchdict['fid']
-        log.debug('Requesting FID:%s from store %s', file_uid, self.storage)
+        log.debug('Requesting FID:%s from store %s', file_uid, self.f_store)
         return self._serve_file(file_uid)
 
     # in addition to @LoginRequired ACL is checked by scopes
     @LoginRequired(auth_token_access=[UserApiKeys.ROLE_ARTIFACT_DOWNLOAD])
     @NotAnonymous()
     def download_file_by_token(self):
         """
         Special view that allows to access the download file by special URL that
         is stored inside the URL.
 
         http://example.com/_file_store/token-download/TOKEN/FILE_UID
         """
         self.load_default_context()
         file_uid = self.request.matchdict['fid']
         return self._serve_file(file_uid)

rhodecode/apps/repository/views/repo_commits.py

@@ -1,830 +1,830 b''
 # Copyright (C) 2010-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 import collections
 
 from pyramid.httpexceptions import (
     HTTPNotFound, HTTPBadRequest, HTTPFound, HTTPForbidden, HTTPConflict)
 from pyramid.renderers import render
 from pyramid.response import Response
 
 from rhodecode.apps._base import RepoAppView
 from rhodecode.apps.file_store import utils as store_utils
 from rhodecode.apps.file_store.exceptions import FileNotAllowedException, FileOverSizeException
 
 from rhodecode.lib import diffs, codeblocks, channelstream
 from rhodecode.lib.auth import (
     LoginRequired, HasRepoPermissionAnyDecorator, NotAnonymous, CSRFRequired)
 from rhodecode.lib import ext_json
 from collections import OrderedDict
 from rhodecode.lib.diffs import (
     cache_diff, load_cached_diff, diff_cache_exist, get_diff_context,
     get_diff_whitespace_flag)
 from rhodecode.lib.exceptions import StatusChangeOnClosedPullRequestError, CommentVersionMismatch
 import rhodecode.lib.helpers as h
 from rhodecode.lib.utils2 import str2bool, StrictAttributeDict, safe_str
 from rhodecode.lib.vcs.backends.base import EmptyCommit
 from rhodecode.lib.vcs.exceptions import (
     RepositoryError, CommitDoesNotExistError)
 from rhodecode.model.db import ChangesetComment, ChangesetStatus, FileStore, \
     ChangesetCommentHistory
 from rhodecode.model.changeset_status import ChangesetStatusModel
 from rhodecode.model.comment import CommentsModel
 from rhodecode.model.meta import Session
 from rhodecode.model.settings import VcsSettingsModel
 
 log = logging.getLogger(__name__)
 
 
 def _update_with_GET(params, request):
     for k in ['diff1', 'diff2', 'diff']:
         params[k] += request.GET.getall(k)
 
 
 class RepoCommitsView(RepoAppView):
     def load_default_context(self):
         c = self._get_local_tmpl_context(include_app_defaults=True)
         c.rhodecode_repo = self.rhodecode_vcs_repo
 
         return c
 
     def _is_diff_cache_enabled(self, target_repo):
         caching_enabled = self._get_general_setting(
             target_repo, 'rhodecode_diff_cache')
         log.debug('Diff caching enabled: %s', caching_enabled)
         return caching_enabled
 
     def _commit(self, commit_id_range, method):
         _ = self.request.translate
         c = self.load_default_context()
         c.fulldiff = self.request.GET.get('fulldiff')
         redirect_to_combined = str2bool(self.request.GET.get('redirect_combined'))
 
         # fetch global flags of ignore ws or context lines
         diff_context = get_diff_context(self.request)
         hide_whitespace_changes = get_diff_whitespace_flag(self.request)
 
         # diff_limit will cut off the whole diff if the limit is applied
         # otherwise it will just hide the big files from the front-end
         diff_limit = c.visual.cut_off_limit_diff
         file_limit = c.visual.cut_off_limit_file
 
         # get ranges of commit ids if preset
         commit_range = commit_id_range.split('...')[:2]
 
         try:
             pre_load = ['affected_files', 'author', 'branch', 'date',
                         'message', 'parents']
             if self.rhodecode_vcs_repo.alias == 'hg':
                 pre_load += ['hidden', 'obsolete', 'phase']
 
             if len(commit_range) == 2:
                 commits = self.rhodecode_vcs_repo.get_commits(
                     start_id=commit_range[0], end_id=commit_range[1],
                     pre_load=pre_load, translate_tags=False)
                 commits = list(commits)
             else:
                 commits = [self.rhodecode_vcs_repo.get_commit(
                     commit_id=commit_id_range, pre_load=pre_load)]
 
             c.commit_ranges = commits
             if not c.commit_ranges:
                 raise RepositoryError('The commit range returned an empty result')
         except CommitDoesNotExistError as e:
             msg = _('No such commit exists. Org exception: `{}`').format(safe_str(e))
             h.flash(msg, category='error')
             raise HTTPNotFound()
         except Exception:
             log.exception("General failure")
             raise HTTPNotFound()
         single_commit = len(c.commit_ranges) == 1
 
         if redirect_to_combined and not single_commit:
             source_ref = getattr(c.commit_ranges[0].parents[0]
                                  if c.commit_ranges[0].parents else h.EmptyCommit(), 'raw_id')
             target_ref = c.commit_ranges[-1].raw_id
             next_url = h.route_path(
                 'repo_compare',
                 repo_name=c.repo_name,
                 source_ref_type='rev',
                 source_ref=source_ref,
                 target_ref_type='rev',
                 target_ref=target_ref)
             raise HTTPFound(next_url)
 
         c.changes = OrderedDict()
         c.lines_added = 0
         c.lines_deleted = 0
 
         # auto collapse if we have more than limit
         collapse_limit = diffs.DiffProcessor._collapse_commits_over
         c.collapse_all_commits = len(c.commit_ranges) > collapse_limit
 
         c.commit_statuses = ChangesetStatus.STATUSES
         c.inline_comments = []
         c.files = []
 
         c.comments = []
         c.unresolved_comments = []
         c.resolved_comments = []
 
         # Single commit
         if single_commit:
             commit = c.commit_ranges[0]
             c.comments = CommentsModel().get_comments(
                 self.db_repo.repo_id,
                 revision=commit.raw_id)
 
             # comments from PR
             statuses = ChangesetStatusModel().get_statuses(
                 self.db_repo.repo_id, commit.raw_id,
                 with_revisions=True)
 
             prs = set()
             reviewers = list()
             reviewers_duplicates = set()  # to not have duplicates from multiple votes
             for c_status in statuses:
 
                 # extract associated pull-requests from votes
                 if c_status.pull_request:
                     prs.add(c_status.pull_request)
 
                 # extract reviewers
                 _user_id = c_status.author.user_id
                 if _user_id not in reviewers_duplicates:
                     reviewers.append(
                         StrictAttributeDict({
                             'user': c_status.author,
 
                             # fake attributed for commit, page that we don't have
                             # but we share the display with PR page
                             'mandatory': False,
                             'reasons': [],
                             'rule_user_group_data': lambda: None
                         })
                     )
                     reviewers_duplicates.add(_user_id)
 
             c.reviewers_count = len(reviewers)
             c.observers_count = 0
 
             # from associated statuses, check the pull requests, and
             # show comments from them
             for pr in prs:
                 c.comments.extend(pr.comments)
 
             c.unresolved_comments = CommentsModel()\
                 .get_commit_unresolved_todos(commit.raw_id)
             c.resolved_comments = CommentsModel()\
                 .get_commit_resolved_todos(commit.raw_id)
 
             c.inline_comments_flat = CommentsModel()\
                 .get_commit_inline_comments(commit.raw_id)
 
             review_statuses = ChangesetStatusModel().aggregate_votes_by_user(
                 statuses, reviewers)
 
             c.commit_review_status = ChangesetStatus.STATUS_NOT_REVIEWED
 
             c.commit_set_reviewers_data_json = collections.OrderedDict({'reviewers': []})
 
             for review_obj, member, reasons, mandatory, status in review_statuses:
                 member_reviewer = h.reviewer_as_json(
                     member, reasons=reasons, mandatory=mandatory, role=None,
                     user_group=None
                 )
 
                 current_review_status = status[0][1].status if status else ChangesetStatus.STATUS_NOT_REVIEWED
                 member_reviewer['review_status'] = current_review_status
                 member_reviewer['review_status_label'] = h.commit_status_lbl(current_review_status)
                 member_reviewer['allowed_to_update'] = False
                 c.commit_set_reviewers_data_json['reviewers'].append(member_reviewer)
 
             c.commit_set_reviewers_data_json = ext_json.str_json(c.commit_set_reviewers_data_json)
 
             # NOTE(marcink): this uses the same voting logic as in pull-requests
             c.commit_review_status = ChangesetStatusModel().calculate_status(review_statuses)
             c.commit_broadcast_channel = channelstream.comment_channel(c.repo_name, commit_obj=commit)
 
         diff = None
         # Iterate over ranges (default commit view is always one commit)
         for commit in c.commit_ranges:
             c.changes[commit.raw_id] = []
 
             commit2 = commit
             commit1 = commit.first_parent
 
             if method == 'show':
                 inline_comments = CommentsModel().get_inline_comments(
                     self.db_repo.repo_id, revision=commit.raw_id)
                 c.inline_cnt = len(CommentsModel().get_inline_comments_as_list(
                     inline_comments))
                 c.inline_comments = inline_comments
 
                 cache_path = self.rhodecode_vcs_repo.get_create_shadow_cache_pr_path(
                     self.db_repo)
                 cache_file_path = diff_cache_exist(
                     cache_path, 'diff', commit.raw_id,
                     hide_whitespace_changes, diff_context, c.fulldiff)
 
                 caching_enabled = self._is_diff_cache_enabled(self.db_repo)
                 force_recache = str2bool(self.request.GET.get('force_recache'))
 
                 cached_diff = None
                 if caching_enabled:
                     cached_diff = load_cached_diff(cache_file_path)
 
                 has_proper_diff_cache = cached_diff and cached_diff.get('diff')
                 if not force_recache and has_proper_diff_cache:
                     diffset = cached_diff['diff']
                 else:
                     vcs_diff = self.rhodecode_vcs_repo.get_diff(
                         commit1, commit2,
                         ignore_whitespace=hide_whitespace_changes,
                         context=diff_context)
 
                     diff_processor = diffs.DiffProcessor(vcs_diff, diff_format='newdiff',
                                                          diff_limit=diff_limit,
                                                          file_limit=file_limit,
                                                          show_full_diff=c.fulldiff)
 
                     _parsed = diff_processor.prepare()
 
                     diffset = codeblocks.DiffSet(
                         repo_name=self.db_repo_name,
                         source_node_getter=codeblocks.diffset_node_getter(commit1),
                         target_node_getter=codeblocks.diffset_node_getter(commit2))
 
                     diffset = self.path_filter.render_patchset_filtered(
                         diffset, _parsed, commit1.raw_id, commit2.raw_id)
 
                     # save cached diff
                     if caching_enabled:
                         cache_diff(cache_file_path, diffset, None)
 
                 c.limited_diff = diffset.limited_diff
                 c.changes[commit.raw_id] = diffset
             else:
                 # TODO(marcink): no cache usage here...
                 _diff = self.rhodecode_vcs_repo.get_diff(
                     commit1, commit2,
                     ignore_whitespace=hide_whitespace_changes, context=diff_context)
                 diff_processor = diffs.DiffProcessor(_diff, diff_format='newdiff',
                                                      diff_limit=diff_limit,
                                                      file_limit=file_limit, show_full_diff=c.fulldiff)
                 # downloads/raw we only need RAW diff nothing else
                 diff = self.path_filter.get_raw_patch(diff_processor)
                 c.changes[commit.raw_id] = [None, None, None, None, diff, None, None]
 
         # sort comments by how they were generated
         c.comments = sorted(c.comments, key=lambda x: x.comment_id)
         c.at_version_num = None
 
         if len(c.commit_ranges) == 1:
             c.commit = c.commit_ranges[0]
             c.parent_tmpl = ''.join(
                 '# Parent  %s\n' % x.raw_id for x in c.commit.parents)
 
         if method == 'download':
             response = Response(diff)
             response.content_type = 'text/plain'
             response.content_disposition = (
                 'attachment; filename=%s.diff' % commit_id_range[:12])
             return response
         elif method == 'patch':
 
             c.diff = safe_str(diff)
             patch = render(
                 'rhodecode:templates/changeset/patch_changeset.mako',
                 self._get_template_context(c), self.request)
             response = Response(patch)
             response.content_type = 'text/plain'
             return response
         elif method == 'raw':
             response = Response(diff)
             response.content_type = 'text/plain'
             return response
         elif method == 'show':
             if len(c.commit_ranges) == 1:
                 html = render(
                     'rhodecode:templates/changeset/changeset.mako',
                     self._get_template_context(c), self.request)
                 return Response(html)
             else:
                 c.ancestor = None
                 c.target_repo = self.db_repo
                 html = render(
                     'rhodecode:templates/changeset/changeset_range.mako',
                     self._get_template_context(c), self.request)
                 return Response(html)
 
         raise HTTPBadRequest()
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_commit_show(self):
         commit_id = self.request.matchdict['commit_id']
         return self._commit(commit_id, method='show')
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_commit_raw(self):
         commit_id = self.request.matchdict['commit_id']
         return self._commit(commit_id, method='raw')
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_commit_patch(self):
         commit_id = self.request.matchdict['commit_id']
         return self._commit(commit_id, method='patch')
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_commit_download(self):
         commit_id = self.request.matchdict['commit_id']
         return self._commit(commit_id, method='download')
 
     def _commit_comments_create(self, commit_id, comments):
         _ = self.request.translate
         data = {}
         if not comments:
             return
 
         commit = self.db_repo.get_commit(commit_id)
 
         all_drafts = len([x for x in comments if str2bool(x['is_draft'])]) == len(comments)
         for entry in comments:
             c = self.load_default_context()
             comment_type = entry['comment_type']
             text = entry['text']
             status = entry['status']
             is_draft = str2bool(entry['is_draft'])
             resolves_comment_id = entry['resolves_comment_id']
             f_path = entry['f_path']
             line_no = entry['line']
             target_elem_id = f'file-{h.safeid(h.safe_str(f_path))}'
 
             if status:
                 text = text or (_('Status change %(transition_icon)s %(status)s')
                                 % {'transition_icon': '>',
                                    'status': ChangesetStatus.get_status_lbl(status)})
 
             comment = CommentsModel().create(
                 text=text,
                 repo=self.db_repo.repo_id,
                 user=self._rhodecode_db_user.user_id,
                 commit_id=commit_id,
                 f_path=f_path,
                 line_no=line_no,
                 status_change=(ChangesetStatus.get_status_lbl(status)
                                if status else None),
                 status_change_type=status,
                 comment_type=comment_type,
                 is_draft=is_draft,
                 resolves_comment_id=resolves_comment_id,
                 auth_user=self._rhodecode_user,
                 send_email=not is_draft,  # skip notification for draft comments
             )
             is_inline = comment.is_inline
 
             # get status if set !
             if status:
                 # `dont_allow_on_closed_pull_request = True` means
                 # if latest status was from pull request and it's closed
                 # disallow changing status !
 
                 try:
                     ChangesetStatusModel().set_status(
                         self.db_repo.repo_id,
                         status,
                         self._rhodecode_db_user.user_id,
                         comment,
                         revision=commit_id,
                         dont_allow_on_closed_pull_request=True
                     )
                 except StatusChangeOnClosedPullRequestError:
                     msg = _('Changing the status of a commit associated with '
                             'a closed pull request is not allowed')
                     log.exception(msg)
                     h.flash(msg, category='warning')
                     raise HTTPFound(h.route_path(
                         'repo_commit', repo_name=self.db_repo_name,
                         commit_id=commit_id))
 
             Session().flush()
             # this is somehow required to get access to some relationship
             # loaded on comment
             Session().refresh(comment)
 
             # skip notifications for drafts
             if not is_draft:
                 CommentsModel().trigger_commit_comment_hook(
                     self.db_repo, self._rhodecode_user, 'create',
                     data={'comment': comment, 'commit': commit})
 
             comment_id = comment.comment_id
             data[comment_id] = {
                 'target_id': target_elem_id
             }
             Session().flush()
 
             c.co = comment
             c.at_version_num = 0
             c.is_new = True
             rendered_comment = render(
                 'rhodecode:templates/changeset/changeset_comment_block.mako',
                 self._get_template_context(c), self.request)
 
             data[comment_id].update(comment.get_dict())
             data[comment_id].update({'rendered_text': rendered_comment})
 
         # finalize, commit and redirect
         Session().commit()
 
         # skip channelstream for draft comments
         if not all_drafts:
             comment_broadcast_channel = channelstream.comment_channel(
                 self.db_repo_name, commit_obj=commit)
 
             comment_data = data
             posted_comment_type = 'inline' if is_inline else 'general'
             if len(data) == 1:
                 msg = _('posted {} new {} comment').format(len(data), posted_comment_type)
             else:
                 msg = _('posted {} new {} comments').format(len(data), posted_comment_type)
 
             channelstream.comment_channelstream_push(
                 self.request, comment_broadcast_channel, self._rhodecode_user, msg,
                 comment_data=comment_data)
 
         return data
 
     @LoginRequired()
     @NotAnonymous()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_commit_comment_create(self):
         _ = self.request.translate
         commit_id = self.request.matchdict['commit_id']
 
         multi_commit_ids = []
         for _commit_id in self.request.POST.get('commit_ids', '').split(','):
             if _commit_id not in ['', None, EmptyCommit.raw_id]:
                 if _commit_id not in multi_commit_ids:
                     multi_commit_ids.append(_commit_id)
 
         commit_ids = multi_commit_ids or [commit_id]
 
         data = []
         # Multiple comments for each passed commit id
         for current_id in filter(None, commit_ids):
             comment_data = {
                 'comment_type': self.request.POST.get('comment_type'),
                 'text': self.request.POST.get('text'),
                 'status': self.request.POST.get('changeset_status', None),
                 'is_draft': self.request.POST.get('draft'),
                 'resolves_comment_id': self.request.POST.get('resolves_comment_id', None),
                 'close_pull_request': self.request.POST.get('close_pull_request'),
                 'f_path': self.request.POST.get('f_path'),
                 'line': self.request.POST.get('line'),
             }
             comment = self._commit_comments_create(commit_id=current_id, comments=[comment_data])
             data.append(comment)
 
         return data if len(data) > 1 else data[0]
 
     @LoginRequired()
     @NotAnonymous()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_commit_comment_preview(self):
         # Technically a CSRF token is not needed as no state changes with this
         # call. However, as this is a POST is better to have it, so automated
         # tools don't flag it as potential CSRF.
         # Post is required because the payload could be bigger than the maximum
         # allowed by GET.
 
         text = self.request.POST.get('text')
         renderer = self.request.POST.get('renderer') or 'rst'
         if text:
             return h.render(text, renderer=renderer, mentions=True,
                             repo_name=self.db_repo_name)
         return ''
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_commit_comment_history_view(self):
         c = self.load_default_context()
         comment_id = self.request.matchdict['comment_id']
         comment_history_id = self.request.matchdict['comment_history_id']
 
         comment = ChangesetComment.get_or_404(comment_id)
         comment_owner = (comment.author.user_id == self._rhodecode_db_user.user_id)
         if comment.draft and not comment_owner:
             # if we see draft comments history, we only allow this for owner
             raise HTTPNotFound()
 
         comment_history = ChangesetCommentHistory.get_or_404(comment_history_id)
         is_repo_comment = comment_history.comment.repo.repo_id == self.db_repo.repo_id
 
         if is_repo_comment:
             c.comment_history = comment_history
 
             rendered_comment = render(
                 'rhodecode:templates/changeset/comment_history.mako',
                 self._get_template_context(c), self.request)
             return rendered_comment
         else:
             log.warning('No permissions for user %s to show comment_history_id: %s',
                         self._rhodecode_db_user, comment_history_id)
             raise HTTPNotFound()
 
     @LoginRequired()
     @NotAnonymous()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_commit_comment_attachment_upload(self):
         c = self.load_default_context()
         upload_key = 'attachment'
 
         file_obj = self.request.POST.get(upload_key)
 
         if file_obj is None:
             self.request.response.status = 400
             return {'store_fid': None,
                     'access_path': None,
                     'error': f'{upload_key} data field is missing'}
 
         if not hasattr(file_obj, 'filename'):
             self.request.response.status = 400
             return {'store_fid': None,
                     'access_path': None,
                     'error': 'filename cannot be read from the data field'}
 
         filename = file_obj.filename
         file_display_name = filename
 
         metadata = {
             'user_uploaded': {'username': self._rhodecode_user.username,
                               'user_id': self._rhodecode_user.user_id,
                               'ip': self._rhodecode_user.ip_addr}}
 
         # TODO(marcink): allow .ini configuration for allowed_extensions, and file-size
         allowed_extensions = [
             'gif', '.jpeg', '.jpg', '.png', '.docx', '.gz', '.log', '.pdf',
             '.pptx', '.txt', '.xlsx', '.zip']
         max_file_size = 10 * 1024 * 1024  # 10MB, also validated via dropzone.js
 
         try:
-            storage = store_utils.get_file_storage(self.request.registry.settings)
-            store_uid, metadata = storage.save_file(
-                file_obj.file, filename, extra_metadata=metadata,
+            f_store = store_utils.get_filestore_backend(self.request.registry.settings)
+            store_uid, metadata = f_store.store(
+                filename, file_obj.file, metadata=metadata,
                 extensions=allowed_extensions, max_filesize=max_file_size)
         except FileNotAllowedException:
             self.request.response.status = 400
             permitted_extensions = ', '.join(allowed_extensions)
-            error_msg = 'File `{}` is not allowed. ' \
-                        'Only following extensions are permitted: {}'.format(
-                            filename, permitted_extensions)
+            error_msg = f'File `{filename}` is not allowed. ' \
+                        f'Only following extensions are permitted: {permitted_extensions}'
+
             return {'store_fid': None,
                     'access_path': None,
                     'error': error_msg}
         except FileOverSizeException:
             self.request.response.status = 400
             limit_mb = h.format_byte_size_binary(max_file_size)
+            error_msg = f'File {filename} is exceeding allowed limit of {limit_mb}.'
             return {'store_fid': None,
                     'access_path': None,
-                    'error': 'File {} is exceeding allowed limit of {}.'.format(
-                        filename, limit_mb)}
+                    'error': error_msg}
 
         try:
             entry = FileStore.create(
                 file_uid=store_uid, filename=metadata["filename"],
                 file_hash=metadata["sha256"], file_size=metadata["size"],
                 file_display_name=file_display_name,
                 file_description=f'comment attachment `{safe_str(filename)}`',
                 hidden=True, check_acl=True, user_id=self._rhodecode_user.user_id,
                 scope_repo_id=self.db_repo.repo_id
             )
             Session().add(entry)
             Session().commit()
             log.debug('Stored upload in DB as %s', entry)
         except Exception:
             log.exception('Failed to store file %s', filename)
             self.request.response.status = 400
             return {'store_fid': None,
                     'access_path': None,
                     'error': f'File {filename} failed to store in DB.'}
 
         Session().commit()
 
         data = {
             'store_fid': store_uid,
             'access_path': h.route_path(
                 'download_file', fid=store_uid),
             'fqn_access_path': h.route_url(
                 'download_file', fid=store_uid),
             # for EE those are replaced by FQN links on repo-only like
             'repo_access_path': h.route_url(
                 'download_file', fid=store_uid),
             'repo_fqn_access_path': h.route_url(
                 'download_file', fid=store_uid),
         }
         # this data is a part of CE/EE additional code
         if c.rhodecode_edition_id == 'EE':
             data.update({
                 'repo_access_path': h.route_path(
                     'repo_artifacts_get', repo_name=self.db_repo_name, uid=store_uid),
                 'repo_fqn_access_path': h.route_url(
                     'repo_artifacts_get', repo_name=self.db_repo_name, uid=store_uid),
             })
 
         return data
 
     @LoginRequired()
     @NotAnonymous()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_commit_comment_delete(self):
         commit_id = self.request.matchdict['commit_id']
         comment_id = self.request.matchdict['comment_id']
 
         comment = ChangesetComment.get_or_404(comment_id)
         if not comment:
             log.debug('Comment with id:%s not found, skipping', comment_id)
             # comment already deleted in another call probably
             return True
 
         if comment.immutable:
             # don't allow deleting comments that are immutable
             raise HTTPForbidden()
 
         is_repo_admin = h.HasRepoPermissionAny('repository.admin')(self.db_repo_name)
         super_admin = h.HasPermissionAny('hg.admin')()
         comment_owner = (comment.author.user_id == self._rhodecode_db_user.user_id)
         is_repo_comment = comment.repo.repo_id == self.db_repo.repo_id
         comment_repo_admin = is_repo_admin and is_repo_comment
 
         if comment.draft and not comment_owner:
             # We never allow to delete draft comments for other than owners
             raise HTTPNotFound()
 
         if super_admin or comment_owner or comment_repo_admin:
             CommentsModel().delete(comment=comment, auth_user=self._rhodecode_user)
             Session().commit()
             return True
         else:
             log.warning('No permissions for user %s to delete comment_id: %s',
                         self._rhodecode_db_user, comment_id)
             raise HTTPNotFound()
 
     @LoginRequired()
     @NotAnonymous()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_commit_comment_edit(self):
         self.load_default_context()
 
         commit_id = self.request.matchdict['commit_id']
         comment_id = self.request.matchdict['comment_id']
         comment = ChangesetComment.get_or_404(comment_id)
 
         if comment.immutable:
             # don't allow deleting comments that are immutable
             raise HTTPForbidden()
 
         is_repo_admin = h.HasRepoPermissionAny('repository.admin')(self.db_repo_name)
         super_admin = h.HasPermissionAny('hg.admin')()
         comment_owner = (comment.author.user_id == self._rhodecode_db_user.user_id)
         is_repo_comment = comment.repo.repo_id == self.db_repo.repo_id
         comment_repo_admin = is_repo_admin and is_repo_comment
 
         if super_admin or comment_owner or comment_repo_admin:
             text = self.request.POST.get('text')
             version = self.request.POST.get('version')
             if text == comment.text:
                 log.warning(
                     'Comment(repo): '
                     'Trying to create new version '
                     'with the same comment body {}'.format(
                         comment_id,
                     )
                 )
                 raise HTTPNotFound()
 
             if version.isdigit():
                 version = int(version)
             else:
                 log.warning(
                     'Comment(repo): Wrong version type {} {} '
                     'for comment {}'.format(
                         version,
                         type(version),
                         comment_id,
                     )
                 )
                 raise HTTPNotFound()
 
             try:
                 comment_history = CommentsModel().edit(
                     comment_id=comment_id,
                     text=text,
                     auth_user=self._rhodecode_user,
                     version=version,
                 )
             except CommentVersionMismatch:
                 raise HTTPConflict()
 
             if not comment_history:
                 raise HTTPNotFound()
 
             if not comment.draft:
                 commit = self.db_repo.get_commit(commit_id)
                 CommentsModel().trigger_commit_comment_hook(
                     self.db_repo, self._rhodecode_user, 'edit',
                     data={'comment': comment, 'commit': commit})
 
             Session().commit()
             return {
                 'comment_history_id': comment_history.comment_history_id,
                 'comment_id': comment.comment_id,
                 'comment_version': comment_history.version,
                 'comment_author_username': comment_history.author.username,
                 'comment_author_gravatar': h.gravatar_url(comment_history.author.email, 16, request=self.request),
                 'comment_created_on': h.age_component(comment_history.created_on,
                                                       time_is_local=True),
             }
         else:
             log.warning('No permissions for user %s to edit comment_id: %s',
                         self._rhodecode_db_user, comment_id)
             raise HTTPNotFound()
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_commit_data(self):
         commit_id = self.request.matchdict['commit_id']
         self.load_default_context()
 
         try:
             return self.rhodecode_vcs_repo.get_commit(commit_id=commit_id)
         except CommitDoesNotExistError as e:
             return EmptyCommit(message=str(e))
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_commit_children(self):
         commit_id = self.request.matchdict['commit_id']
         self.load_default_context()
 
         try:
             commit = self.rhodecode_vcs_repo.get_commit(commit_id=commit_id)
             children = commit.children
         except CommitDoesNotExistError:
             children = []
 
         result = {"results": children}
         return result
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_commit_parents(self):
         commit_id = self.request.matchdict['commit_id']
         self.load_default_context()
 
         try:
             commit = self.rhodecode_vcs_repo.get_commit(commit_id=commit_id)
             parents = commit.parents
         except CommitDoesNotExistError:
             parents = []
         result = {"results": parents}
         return result

rhodecode/apps/repository/views/repo_files.py

@@ -1,1716 +1,1716 b''
 # Copyright (C) 2011-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import itertools
 import logging
 import os
 import collections
 import urllib.request
 import urllib.parse
 import urllib.error
 import pathlib
 import time
 import random
 
 from pyramid.httpexceptions import HTTPNotFound, HTTPBadRequest, HTTPFound
 
 from pyramid.renderers import render
 from pyramid.response import Response
 
 import rhodecode
 from rhodecode.apps._base import RepoAppView
 
 
 from rhodecode.lib import diffs, helpers as h, rc_cache
 from rhodecode.lib import audit_logger
 from rhodecode.lib.hash_utils import sha1_safe
 from rhodecode.lib.archive_cache import (
     get_archival_cache_store, get_archival_config, ArchiveCacheGenerationLock, archive_iterator)
 from rhodecode.lib.str_utils import safe_bytes, convert_special_chars
 from rhodecode.lib.view_utils import parse_path_ref
 from rhodecode.lib.exceptions import NonRelativePathError
 from rhodecode.lib.codeblocks import (
     filenode_as_lines_tokens, filenode_as_annotated_lines_tokens)
 from rhodecode.lib.utils2 import convert_line_endings, detect_mode
 from rhodecode.lib.type_utils import str2bool
-from rhodecode.lib.str_utils import safe_str, safe_int
+from rhodecode.lib.str_utils import safe_str, safe_int, header_safe_str
 from rhodecode.lib.auth import (
     LoginRequired, HasRepoPermissionAnyDecorator, CSRFRequired)
 from rhodecode.lib.vcs import path as vcspath
 from rhodecode.lib.vcs.backends.base import EmptyCommit
 from rhodecode.lib.vcs.conf import settings
 from rhodecode.lib.vcs.nodes import FileNode
 from rhodecode.lib.vcs.exceptions import (
     RepositoryError, CommitDoesNotExistError, EmptyRepositoryError,
     ImproperArchiveTypeError, VCSError, NodeAlreadyExistsError,
     NodeDoesNotExistError, CommitError, NodeError)
 
 from rhodecode.model.scm import ScmModel
 from rhodecode.model.db import Repository
 
 log = logging.getLogger(__name__)
 
 
 def get_archive_name(db_repo_id, db_repo_name, commit_sha, ext, subrepos=False, path_sha='', with_hash=True):
     # original backward compat name of archive
     clean_name = safe_str(convert_special_chars(db_repo_name).replace('/', '_'))
 
     # e.g vcsserver-id-abcd-sub-1-abcfdef-archive-all.zip
     # vcsserver-id-abcd-sub-0-abcfdef-COMMIT_SHA-PATH_SHA.zip
     id_sha = sha1_safe(str(db_repo_id))[:4]
     sub_repo = 'sub-1' if subrepos else 'sub-0'
     commit = commit_sha if with_hash else 'archive'
     path_marker = (path_sha if with_hash else '') or 'all'
     archive_name = f'{clean_name}-id-{id_sha}-{sub_repo}-{commit}-{path_marker}{ext}'
 
     return archive_name
 
 
 def get_path_sha(at_path):
     return safe_str(sha1_safe(at_path)[:8])
 
 
 def _get_archive_spec(fname):
     log.debug('Detecting archive spec for: `%s`', fname)
 
     fileformat = None
     ext = None
     content_type = None
     for a_type, content_type, extension in settings.ARCHIVE_SPECS:
 
         if fname.endswith(extension):
             fileformat = a_type
             log.debug('archive is of type: %s', fileformat)
             ext = extension
             break
 
     if not fileformat:
         raise ValueError()
 
     # left over part of whole fname is the commit
     commit_id = fname[:-len(ext)]
 
     return commit_id, ext, fileformat, content_type
 
 
 class RepoFilesView(RepoAppView):
 
     @staticmethod
     def adjust_file_path_for_svn(f_path, repo):
         """
         Computes the relative path of `f_path`.
 
         This is mainly based on prefix matching of the recognized tags and
         branches in the underlying repository.
         """
         tags_and_branches = itertools.chain(
             repo.branches.keys(),
             repo.tags.keys())
         tags_and_branches = sorted(tags_and_branches, key=len, reverse=True)
 
         for name in tags_and_branches:
             if f_path.startswith(f'{name}/'):
                 f_path = vcspath.relpath(f_path, name)
                 break
         return f_path
 
     def load_default_context(self):
         c = self._get_local_tmpl_context(include_app_defaults=True)
         c.rhodecode_repo = self.rhodecode_vcs_repo
         c.enable_downloads = self.db_repo.enable_downloads
         return c
 
     def _ensure_not_locked(self, commit_id='tip'):
         _ = self.request.translate
 
         repo = self.db_repo
         if repo.enable_locking and repo.locked[0]:
             h.flash(_('This repository has been locked by %s on %s')
                     % (h.person_by_id(repo.locked[0]),
                     h.format_date(h.time_to_datetime(repo.locked[1]))),
                     'warning')
             files_url = h.route_path(
                 'repo_files:default_path',
                 repo_name=self.db_repo_name, commit_id=commit_id)
             raise HTTPFound(files_url)
 
     def forbid_non_head(self, is_head, f_path, commit_id='tip', json_mode=False):
         _ = self.request.translate
 
         if not is_head:
             message = _('Cannot modify file. '
                         'Given commit `{}` is not head of a branch.').format(commit_id)
             h.flash(message, category='warning')
 
             if json_mode:
                 return message
 
             files_url = h.route_path(
                 'repo_files', repo_name=self.db_repo_name, commit_id=commit_id,
                 f_path=f_path)
             raise HTTPFound(files_url)
 
     def check_branch_permission(self, branch_name, commit_id='tip', json_mode=False):
         _ = self.request.translate
 
         rule, branch_perm = self._rhodecode_user.get_rule_and_branch_permission(
             self.db_repo_name, branch_name)
         if branch_perm and branch_perm not in ['branch.push', 'branch.push_force']:
             message = _('Branch `{}` changes forbidden by rule {}.').format(
                 h.escape(branch_name), h.escape(rule))
             h.flash(message, 'warning')
 
             if json_mode:
                 return message
 
             files_url = h.route_path(
                 'repo_files:default_path', repo_name=self.db_repo_name, commit_id=commit_id)
 
             raise HTTPFound(files_url)
 
     def _get_commit_and_path(self):
         default_commit_id = self.db_repo.landing_ref_name
         default_f_path = '/'
 
         commit_id = self.request.matchdict.get(
             'commit_id', default_commit_id)
         f_path = self._get_f_path(self.request.matchdict, default_f_path)
         return commit_id, f_path
 
     def _get_default_encoding(self, c):
         enc_list = getattr(c, 'default_encodings', [])
         return enc_list[0] if enc_list else 'UTF-8'
 
     def _get_commit_or_redirect(self, commit_id, redirect_after=True):
         """
         This is a safe way to get commit. If an error occurs it redirects to
         tip with proper message
 
         :param commit_id: id of commit to fetch
         :param redirect_after: toggle redirection
         """
         _ = self.request.translate
 
         try:
             return self.rhodecode_vcs_repo.get_commit(commit_id)
         except EmptyRepositoryError:
             if not redirect_after:
                 return None
 
             add_new = upload_new = ""
             if h.HasRepoPermissionAny(
                     'repository.write', 'repository.admin')(self.db_repo_name):
                 _url = h.route_path(
                     'repo_files_add_file',
                     repo_name=self.db_repo_name, commit_id=0, f_path='')
                 add_new = h.link_to(
                     _('add a new file'), _url, class_="alert-link")
 
                 _url_upld = h.route_path(
                     'repo_files_upload_file',
                     repo_name=self.db_repo_name, commit_id=0, f_path='')
                 upload_new = h.link_to(
                     _('upload a new file'), _url_upld, class_="alert-link")
 
             h.flash(h.literal(
                 _('There are no files yet. Click here to %s or %s.') % (add_new, upload_new)), category='warning')
             raise HTTPFound(
                 h.route_path('repo_summary', repo_name=self.db_repo_name))
 
         except (CommitDoesNotExistError, LookupError) as e:
             msg = _('No such commit exists for this repository. Commit: {}').format(commit_id)
             h.flash(msg, category='error')
             raise HTTPNotFound()
         except RepositoryError as e:
             h.flash(h.escape(safe_str(e)), category='error')
             raise HTTPNotFound()
 
     def _get_filenode_or_redirect(self, commit_obj, path, pre_load=None):
         """
         Returns file_node, if error occurs or given path is directory,
         it'll redirect to top level path
         """
         _ = self.request.translate
 
         try:
             file_node = commit_obj.get_node(path, pre_load=pre_load)
             if file_node.is_dir():
                 raise RepositoryError('The given path is a directory')
         except CommitDoesNotExistError:
             log.exception('No such commit exists for this repository')
             h.flash(_('No such commit exists for this repository'), category='error')
             raise HTTPNotFound()
         except RepositoryError as e:
             log.warning('Repository error while fetching filenode `%s`. Err:%s', path, e)
             h.flash(h.escape(safe_str(e)), category='error')
             raise HTTPNotFound()
 
         return file_node
 
     def _is_valid_head(self, commit_id, repo, landing_ref):
         branch_name = sha_commit_id = ''
         is_head = False
         log.debug('Checking if commit_id `%s` is a head for %s.', commit_id, repo)
 
         for _branch_name, branch_commit_id in repo.branches.items():
             # simple case we pass in branch name, it's a HEAD
             if commit_id == _branch_name:
                 is_head = True
                 branch_name = _branch_name
                 sha_commit_id = branch_commit_id
                 break
             # case when we pass in full sha commit_id, which is a head
             elif commit_id == branch_commit_id:
                 is_head = True
                 branch_name = _branch_name
                 sha_commit_id = branch_commit_id
                 break
 
         if h.is_svn(repo) and not repo.is_empty():
             # Note: Subversion only has one head.
             if commit_id == repo.get_commit(commit_idx=-1).raw_id:
                 is_head = True
                 return branch_name, sha_commit_id, is_head
 
         # checked branches, means we only need to try to get the branch/commit_sha
         if repo.is_empty():
             is_head = True
             branch_name = landing_ref
             sha_commit_id = EmptyCommit().raw_id
         else:
             commit = repo.get_commit(commit_id=commit_id)
             if commit:
                 branch_name = commit.branch
                 sha_commit_id = commit.raw_id
 
         return branch_name, sha_commit_id, is_head
 
     def _get_tree_at_commit(self, c, commit_id, f_path, full_load=False, at_rev=None):
 
         repo_id = self.db_repo.repo_id
         force_recache = self.get_recache_flag()
 
         cache_seconds = safe_int(
             rhodecode.CONFIG.get('rc_cache.cache_repo.expiration_time'))
         cache_on = not force_recache and cache_seconds > 0
         log.debug(
             'Computing FILE TREE for repo_id %s commit_id `%s` and path `%s`'
             'with caching: %s[TTL: %ss]' % (
                 repo_id, commit_id, f_path, cache_on, cache_seconds or 0))
 
         cache_namespace_uid = f'repo.{rc_cache.FILE_TREE_CACHE_VER}.{repo_id}'
         region = rc_cache.get_or_create_region('cache_repo', cache_namespace_uid)
 
         @region.conditional_cache_on_arguments(namespace=cache_namespace_uid, condition=cache_on)
         def compute_file_tree(_name_hash, _repo_id, _commit_id, _f_path, _full_load, _at_rev):
             log.debug('Generating cached file tree at for repo_id: %s, %s, %s',
                       _repo_id, _commit_id, _f_path)
 
             c.full_load = _full_load
             return render(
                 'rhodecode:templates/files/files_browser_tree.mako',
                 self._get_template_context(c), self.request, _at_rev)
 
         return compute_file_tree(
             self.db_repo.repo_name_hash, self.db_repo.repo_id, commit_id, f_path, full_load, at_rev)
 
     def create_pure_path(self, *parts):
         # Split paths and sanitize them, removing any ../ etc
         sanitized_path = [
             x for x in pathlib.PurePath(*parts).parts
             if x not in ['.', '..']]
 
         pure_path = pathlib.PurePath(*sanitized_path)
         return pure_path
 
     def _is_lf_enabled(self, target_repo):
         lf_enabled = False
 
         lf_key_for_vcs_map = {
             'hg': 'extensions_largefiles',
             'git': 'vcs_git_lfs_enabled'
         }
 
         lf_key_for_vcs = lf_key_for_vcs_map.get(target_repo.repo_type)
 
         if lf_key_for_vcs:
             lf_enabled = self._get_repo_setting(target_repo, lf_key_for_vcs)
 
         return lf_enabled
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_archivefile(self):
         # archive cache config
         from rhodecode import CONFIG
         _ = self.request.translate
         self.load_default_context()
         default_at_path = '/'
         fname = self.request.matchdict['fname']
         subrepos = self.request.GET.get('subrepos') == 'true'
         with_hash = str2bool(self.request.GET.get('with_hash', '1'))
         at_path = self.request.GET.get('at_path') or default_at_path
 
         if not self.db_repo.enable_downloads:
             return Response(_('Downloads disabled'))
 
         try:
             commit_id, ext, fileformat, content_type = \
                 _get_archive_spec(fname)
         except ValueError:
             return Response(_('Unknown archive type for: `{}`').format(
                 h.escape(fname)))
 
         try:
             commit = self.rhodecode_vcs_repo.get_commit(commit_id)
         except CommitDoesNotExistError:
             return Response(_('Unknown commit_id {}').format(
                 h.escape(commit_id)))
         except EmptyRepositoryError:
             return Response(_('Empty repository'))
 
         # we used a ref, or a shorter version, lets redirect client ot use explicit hash
         if commit_id != commit.raw_id:
             fname=f'{commit.raw_id}{ext}'
             raise HTTPFound(self.request.current_route_path(fname=fname))
 
         try:
             at_path = commit.get_node(at_path).path or default_at_path
         except Exception:
             return Response(_('No node at path {} for this repository').format(h.escape(at_path)))
 
         path_sha = get_path_sha(at_path)
 
         # used for cache etc, consistent unique archive name
         archive_name_key = get_archive_name(
             self.db_repo.repo_id, self.db_repo_name, commit_sha=commit.short_id, ext=ext, subrepos=subrepos,
             path_sha=path_sha, with_hash=True)
 
         if not with_hash:
             path_sha = ''
 
         # what end client gets served
         response_archive_name = get_archive_name(
             self.db_repo.repo_id, self.db_repo_name, commit_sha=commit.short_id, ext=ext, subrepos=subrepos,
             path_sha=path_sha, with_hash=with_hash)
 
         # remove extension from our archive directory name
         archive_dir_name = response_archive_name[:-len(ext)]
 
         archive_cache_disable = self.request.GET.get('no_cache')
 
         d_cache = get_archival_cache_store(config=CONFIG)
 
         # NOTE: we get the config to pass to a call to lazy-init the SAME type of cache on vcsserver
         d_cache_conf = get_archival_config(config=CONFIG)
 
         # This is also a cache key, and lock key
         reentrant_lock_key = archive_name_key + '.lock'
 
         use_cached_archive = False
         if not archive_cache_disable and archive_name_key in d_cache:
             reader, metadata = d_cache.fetch(archive_name_key)
 
             use_cached_archive = True
             log.debug('Found cached archive as key=%s tag=%s, serving archive from cache reader=%s',
                       archive_name_key, metadata, reader.name)
         else:
             reader = None
             log.debug('Archive with key=%s is not yet cached, creating one now...', archive_name_key)
 
         if not reader:
             # generate new archive, as previous was not found in the cache
             try:
                 with d_cache.get_lock(reentrant_lock_key):
                     try:
                         commit.archive_repo(archive_name_key, archive_dir_name=archive_dir_name,
                                             kind=fileformat, subrepos=subrepos,
                                             archive_at_path=at_path, cache_config=d_cache_conf)
                     except ImproperArchiveTypeError:
                         return _('Unknown archive type')
 
             except ArchiveCacheGenerationLock:
                 retry_after = round(random.uniform(0.3, 3.0), 1)
                 time.sleep(retry_after)
 
                 location = self.request.url
                 response = Response(
                     f"archive {archive_name_key} generation in progress, Retry-After={retry_after}, Location={location}"
                 )
                 response.headers["Retry-After"] = str(retry_after)
                 response.status_code = 307  # temporary redirect
 
                 response.location = location
                 return response
 
             reader, metadata = d_cache.fetch(archive_name_key, retry=True, retry_attempts=30)
 
         response = Response(app_iter=archive_iterator(reader))
         response.content_disposition = f'attachment; filename={response_archive_name}'
         response.content_type = str(content_type)
 
         try:
             return response
         finally:
             # store download action
             audit_logger.store_web(
                 'repo.archive.download', action_data={
                     'user_agent': self.request.user_agent,
                     'archive_name': archive_name_key,
                     'archive_spec': fname,
                     'archive_cached': use_cached_archive},
                 user=self._rhodecode_user,
                 repo=self.db_repo,
                 commit=True
             )
 
     def _get_file_node(self, commit_id, f_path):
         if commit_id not in ['', None, 'None', '0' * 12, '0' * 40]:
             commit = self.rhodecode_vcs_repo.get_commit(commit_id=commit_id)
             try:
                 node = commit.get_node(f_path)
                 if node.is_dir():
                     raise NodeError(f'{node} path is a {type(node)} not a file')
             except NodeDoesNotExistError:
                 commit = EmptyCommit(
                     commit_id=commit_id,
                     idx=commit.idx,
                     repo=commit.repository,
                     alias=commit.repository.alias,
                     message=commit.message,
                     author=commit.author,
                     date=commit.date)
                 node = FileNode(safe_bytes(f_path), b'', commit=commit)
         else:
             commit = EmptyCommit(
                 repo=self.rhodecode_vcs_repo,
                 alias=self.rhodecode_vcs_repo.alias)
             node = FileNode(safe_bytes(f_path), b'', commit=commit)
         return node
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_files_diff(self):
         c = self.load_default_context()
         f_path = self._get_f_path(self.request.matchdict)
         diff1 = self.request.GET.get('diff1', '')
         diff2 = self.request.GET.get('diff2', '')
 
         path1, diff1 = parse_path_ref(diff1, default_path=f_path)
 
         ignore_whitespace = str2bool(self.request.GET.get('ignorews'))
         line_context = self.request.GET.get('context', 3)
 
         if not any((diff1, diff2)):
             h.flash(
                 'Need query parameter "diff1" or "diff2" to generate a diff.',
                 category='error')
             raise HTTPBadRequest()
 
         c.action = self.request.GET.get('diff')
         if c.action not in ['download', 'raw']:
             compare_url = h.route_path(
                 'repo_compare',
                 repo_name=self.db_repo_name,
                 source_ref_type='rev',
                 source_ref=diff1,
                 target_repo=self.db_repo_name,
                 target_ref_type='rev',
                 target_ref=diff2,
                 _query=dict(f_path=f_path))
             # redirect to new view if we render diff
             raise HTTPFound(compare_url)
 
         try:
             node1 = self._get_file_node(diff1, path1)
             node2 = self._get_file_node(diff2, f_path)
         except (RepositoryError, NodeError):
             log.exception("Exception while trying to get node from repository")
             raise HTTPFound(
                 h.route_path('repo_files', repo_name=self.db_repo_name,
                              commit_id='tip', f_path=f_path))
 
         if all(isinstance(node.commit, EmptyCommit)
                for node in (node1, node2)):
             raise HTTPNotFound()
 
         c.commit_1 = node1.commit
         c.commit_2 = node2.commit
 
         if c.action == 'download':
             _diff = diffs.get_gitdiff(node1, node2,
                                       ignore_whitespace=ignore_whitespace,
                                       context=line_context)
             # NOTE: this was using diff_format='gitdiff'
             diff = diffs.DiffProcessor(_diff, diff_format='newdiff')
 
             response = Response(self.path_filter.get_raw_patch(diff))
             response.content_type = 'text/plain'
             response.content_disposition = (
                 f'attachment; filename={f_path}_{diff1}_vs_{diff2}.diff'
             )
             charset = self._get_default_encoding(c)
             if charset:
                 response.charset = charset
             return response
 
         elif c.action == 'raw':
             _diff = diffs.get_gitdiff(node1, node2,
                                       ignore_whitespace=ignore_whitespace,
                                       context=line_context)
             # NOTE: this was using diff_format='gitdiff'
             diff = diffs.DiffProcessor(_diff, diff_format='newdiff')
 
             response = Response(self.path_filter.get_raw_patch(diff))
             response.content_type = 'text/plain'
             charset = self._get_default_encoding(c)
             if charset:
                 response.charset = charset
             return response
 
         # in case we ever end up here
         raise HTTPNotFound()
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_files_diff_2way_redirect(self):
         """
         Kept only to make OLD links work
         """
         f_path = self._get_f_path_unchecked(self.request.matchdict)
         diff1 = self.request.GET.get('diff1', '')
         diff2 = self.request.GET.get('diff2', '')
 
         if not any((diff1, diff2)):
             h.flash(
                 'Need query parameter "diff1" or "diff2" to generate a diff.',
                 category='error')
             raise HTTPBadRequest()
 
         compare_url = h.route_path(
             'repo_compare',
             repo_name=self.db_repo_name,
             source_ref_type='rev',
             source_ref=diff1,
             target_ref_type='rev',
             target_ref=diff2,
             _query=dict(f_path=f_path, diffmode='sideside',
                         target_repo=self.db_repo_name,))
         raise HTTPFound(compare_url)
 
     @LoginRequired()
     def repo_files_default_commit_redirect(self):
         """
         Special page that redirects to the landing page of files based on the default
         commit for repository
         """
         c = self.load_default_context()
         ref_name = c.rhodecode_db_repo.landing_ref_name
         landing_url = h.repo_files_by_ref_url(
             c.rhodecode_db_repo.repo_name,
             c.rhodecode_db_repo.repo_type,
             f_path='',
             ref_name=ref_name,
             commit_id='tip',
             query=dict(at=ref_name)
         )
 
         raise HTTPFound(landing_url)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_files(self):
         c = self.load_default_context()
 
         view_name = getattr(self.request.matched_route, 'name', None)
 
         c.annotate = view_name == 'repo_files:annotated'
         # default is false, but .rst/.md files later are auto rendered, we can
         # overwrite auto rendering by setting this GET flag
         c.renderer = view_name == 'repo_files:rendered' or not self.request.GET.get('no-render', False)
 
         commit_id, f_path = self._get_commit_and_path()
 
         c.commit = self._get_commit_or_redirect(commit_id)
         c.branch = self.request.GET.get('branch', None)
         c.f_path = f_path
         at_rev = self.request.GET.get('at')
 
         # files or dirs
         try:
             c.file = c.commit.get_node(f_path, pre_load=['is_binary', 'size', 'data'])
 
             c.file_author = True
             c.file_tree = ''
 
             # prev link
             try:
                 prev_commit = c.commit.prev(c.branch)
                 c.prev_commit = prev_commit
                 c.url_prev = h.route_path(
                     'repo_files', repo_name=self.db_repo_name,
                     commit_id=prev_commit.raw_id, f_path=f_path)
                 if c.branch:
                     c.url_prev += '?branch=%s' % c.branch
             except (CommitDoesNotExistError, VCSError):
                 c.url_prev = '#'
                 c.prev_commit = EmptyCommit()
 
             # next link
             try:
                 next_commit = c.commit.next(c.branch)
                 c.next_commit = next_commit
                 c.url_next = h.route_path(
                     'repo_files', repo_name=self.db_repo_name,
                     commit_id=next_commit.raw_id, f_path=f_path)
                 if c.branch:
                     c.url_next += '?branch=%s' % c.branch
             except (CommitDoesNotExistError, VCSError):
                 c.url_next = '#'
                 c.next_commit = EmptyCommit()
 
             # load file content
             if c.file.is_file():
 
                 c.lf_node = {}
 
                 has_lf_enabled = self._is_lf_enabled(self.db_repo)
                 if has_lf_enabled:
                     c.lf_node = c.file.get_largefile_node()
 
                 c.file_source_page = 'true'
                 c.file_last_commit = c.file.last_commit
 
                 c.file_size_too_big = c.file.size > c.visual.cut_off_limit_file
 
                 if not (c.file_size_too_big or c.file.is_binary):
                     if c.annotate:  # annotation has precedence over renderer
                         c.annotated_lines = filenode_as_annotated_lines_tokens(
                             c.file
                         )
                     else:
                         c.renderer = (
                             c.renderer and h.renderer_from_filename(c.file.path)
                         )
                         if not c.renderer:
                             c.lines = filenode_as_lines_tokens(c.file)
 
                 _branch_name, _sha_commit_id, is_head = \
                     self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                         landing_ref=self.db_repo.landing_ref_name)
                 c.on_branch_head = is_head
 
                 branch = c.commit.branch if (
                     c.commit.branch and '/' not in c.commit.branch) else None
                 c.branch_or_raw_id = branch or c.commit.raw_id
                 c.branch_name = c.commit.branch or h.short_id(c.commit.raw_id)
 
                 author = c.file_last_commit.author
                 c.authors = [[
                     h.email(author),
                     h.person(author, 'username_or_name_or_email'),
                     1
                 ]]
 
             else:  # load tree content at path
                 c.file_source_page = 'false'
                 c.authors = []
                 # this loads a simple tree without metadata to speed things up
                 # later via ajax we call repo_nodetree_full and fetch whole
                 c.file_tree = self._get_tree_at_commit(c, c.commit.raw_id, f_path, at_rev=at_rev)
 
                 c.readme_data, c.readme_file = \
                     self._get_readme_data(self.db_repo, c.visual.default_renderer,
                                           c.commit.raw_id, f_path)
 
         except RepositoryError as e:
             h.flash(h.escape(safe_str(e)), category='error')
             raise HTTPNotFound()
 
         if self.request.environ.get('HTTP_X_PJAX'):
             html = render('rhodecode:templates/files/files_pjax.mako',
                           self._get_template_context(c), self.request)
         else:
             html = render('rhodecode:templates/files/files.mako',
                           self._get_template_context(c), self.request)
         return Response(html)
 
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_files_annotated_previous(self):
         self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         commit = self._get_commit_or_redirect(commit_id)
         prev_commit_id = commit.raw_id
         line_anchor = self.request.GET.get('line_anchor')
         is_file = False
         try:
             _file = commit.get_node(f_path)
             is_file = _file.is_file()
         except (NodeDoesNotExistError, CommitDoesNotExistError, VCSError):
             pass
 
         if is_file:
             history = commit.get_path_history(f_path)
             prev_commit_id = history[1].raw_id \
                 if len(history) > 1 else prev_commit_id
         prev_url = h.route_path(
             'repo_files:annotated', repo_name=self.db_repo_name,
             commit_id=prev_commit_id, f_path=f_path,
             _anchor=f'L{line_anchor}')
 
         raise HTTPFound(prev_url)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_nodetree_full(self):
         """
         Returns rendered html of file tree that contains commit date,
         author, commit_id for the specified combination of
         repo, commit_id and file path
         """
         c = self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         commit = self._get_commit_or_redirect(commit_id)
         try:
             dir_node = commit.get_node(f_path)
         except RepositoryError as e:
             return Response(f'error: {h.escape(safe_str(e))}')
 
         if dir_node.is_file():
             return Response('')
 
         c.file = dir_node
         c.commit = commit
         at_rev = self.request.GET.get('at')
 
         html = self._get_tree_at_commit(
             c, commit.raw_id, dir_node.path, full_load=True, at_rev=at_rev)
 
         return Response(html)
 
     def _get_attachement_headers(self, f_path):
         f_name = safe_str(f_path.split(Repository.NAME_SEP)[-1])
         safe_path = f_name.replace('"', '\\"')
         encoded_path = urllib.parse.quote(f_name)
 
         headers = "attachment; " \
                   "filename=\"{}\"; " \
                   "filename*=UTF-8\'\'{}".format(safe_path, encoded_path)
 
-        return safe_bytes(headers).decode('latin-1', errors='replace')
+        return header_safe_str(headers)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_file_raw(self):
         """
         Action for show as raw, some mimetypes are "rendered",
         those include images, icons.
         """
         c = self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         commit = self._get_commit_or_redirect(commit_id)
         file_node = self._get_filenode_or_redirect(commit, f_path)
 
         raw_mimetype_mapping = {
             # map original mimetype to a mimetype used for "show as raw"
             # you can also provide a content-disposition to override the
             # default "attachment" disposition.
             # orig_type: (new_type, new_dispo)
 
             # show images inline:
             # Do not re-add SVG: it is unsafe and permits XSS attacks. One can
             # for example render an SVG with javascript inside or even render
             # HTML.
             'image/x-icon': ('image/x-icon', 'inline'),
             'image/png': ('image/png', 'inline'),
             'image/gif': ('image/gif', 'inline'),
             'image/jpeg': ('image/jpeg', 'inline'),
             'application/pdf': ('application/pdf', 'inline'),
         }
 
         mimetype = file_node.mimetype
         try:
             mimetype, disposition = raw_mimetype_mapping[mimetype]
         except KeyError:
             # we don't know anything special about this, handle it safely
             if file_node.is_binary:
                 # do same as download raw for binary files
                 mimetype, disposition = 'application/octet-stream', 'attachment'
             else:
                 # do not just use the original mimetype, but force text/plain,
                 # otherwise it would serve text/html and that might be unsafe.
                 # Note: underlying vcs library fakes text/plain mimetype if the
                 # mimetype can not be determined and it thinks it is not
                 # binary.This might lead to erroneous text display in some
                 # cases, but helps in other cases, like with text files
                 # without extension.
                 mimetype, disposition = 'text/plain', 'inline'
 
         if disposition == 'attachment':
             disposition = self._get_attachement_headers(f_path)
 
         stream_content = file_node.stream_bytes()
 
         response = Response(app_iter=stream_content)
         response.content_disposition = disposition
         response.content_type = mimetype
 
         charset = self._get_default_encoding(c)
         if charset:
             response.charset = charset
 
         return response
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_file_download(self):
         c = self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         commit = self._get_commit_or_redirect(commit_id)
         file_node = self._get_filenode_or_redirect(commit, f_path)
 
         if self.request.GET.get('lf'):
             # only if lf get flag is passed, we download this file
             # as LFS/Largefile
             lf_node = file_node.get_largefile_node()
             if lf_node:
                 # overwrite our pointer with the REAL large-file
                 file_node = lf_node
 
         disposition = self._get_attachement_headers(f_path)
 
         stream_content = file_node.stream_bytes()
 
         response = Response(app_iter=stream_content)
         response.content_disposition = disposition
         response.content_type = file_node.mimetype
 
         charset = self._get_default_encoding(c)
         if charset:
             response.charset = charset
 
         return response
 
     def _get_nodelist_at_commit(self, repo_name, repo_id, commit_id, f_path):
 
         cache_seconds = safe_int(
             rhodecode.CONFIG.get('rc_cache.cache_repo.expiration_time'))
         cache_on = cache_seconds > 0
         log.debug(
             'Computing FILE SEARCH for repo_id %s commit_id `%s` and path `%s`'
             'with caching: %s[TTL: %ss]' % (
                 repo_id, commit_id, f_path, cache_on, cache_seconds or 0))
 
         cache_namespace_uid = f'repo.{repo_id}'
         region = rc_cache.get_or_create_region('cache_repo', cache_namespace_uid)
 
         @region.conditional_cache_on_arguments(namespace=cache_namespace_uid, condition=cache_on)
         def compute_file_search(_name_hash, _repo_id, _commit_id, _f_path):
             log.debug('Generating cached nodelist for repo_id:%s, %s, %s',
                       _repo_id, commit_id, f_path)
             try:
                 _d, _f = ScmModel().get_quick_filter_nodes(repo_name, _commit_id, _f_path)
             except (RepositoryError, CommitDoesNotExistError, Exception) as e:
                 log.exception(safe_str(e))
                 h.flash(h.escape(safe_str(e)), category='error')
                 raise HTTPFound(h.route_path(
                     'repo_files', repo_name=self.db_repo_name,
                     commit_id='tip', f_path='/'))
 
             return _d + _f
 
         result = compute_file_search(self.db_repo.repo_name_hash, self.db_repo.repo_id,
                                      commit_id, f_path)
         return filter(lambda n: self.path_filter.path_access_allowed(n['name']), result)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_nodelist(self):
         self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         commit = self._get_commit_or_redirect(commit_id)
 
         metadata = self._get_nodelist_at_commit(
             self.db_repo_name, self.db_repo.repo_id, commit.raw_id, f_path)
         return {'nodes': [x for x in metadata]}
 
     def _create_references(self, branches_or_tags, symbolic_reference, f_path, ref_type):
         items = []
         for name, commit_id in branches_or_tags.items():
             sym_ref = symbolic_reference(commit_id, name, f_path, ref_type)
             items.append((sym_ref, name, ref_type))
         return items
 
     def _symbolic_reference(self, commit_id, name, f_path, ref_type):
         return commit_id
 
     def _symbolic_reference_svn(self, commit_id, name, f_path, ref_type):
         return commit_id
 
         # NOTE(dan): old code we used in "diff" mode compare
         new_f_path = vcspath.join(name, f_path)
         return f'{new_f_path}@{commit_id}'
 
     def _get_node_history(self, commit_obj, f_path, commits=None):
         """
         get commit history for given node
 
         :param commit_obj: commit to calculate history
         :param f_path: path for node to calculate history for
         :param commits: if passed don't calculate history and take
             commits defined in this list
         """
         _ = self.request.translate
 
         # calculate history based on tip
         tip = self.rhodecode_vcs_repo.get_commit()
         if commits is None:
             pre_load = ["author", "branch"]
             try:
                 commits = tip.get_path_history(f_path, pre_load=pre_load)
             except (NodeDoesNotExistError, CommitError):
                 # this node is not present at tip!
                 commits = commit_obj.get_path_history(f_path, pre_load=pre_load)
 
         history = []
         commits_group = ([], _("Changesets"))
         for commit in commits:
             branch = ' (%s)' % commit.branch if commit.branch else ''
             n_desc = f'r{commit.idx}:{commit.short_id}{branch}'
             commits_group[0].append((commit.raw_id, n_desc, 'sha'))
         history.append(commits_group)
 
         symbolic_reference = self._symbolic_reference
 
         if self.rhodecode_vcs_repo.alias == 'svn':
             adjusted_f_path = RepoFilesView.adjust_file_path_for_svn(
                 f_path, self.rhodecode_vcs_repo)
             if adjusted_f_path != f_path:
                 log.debug(
                     'Recognized svn tag or branch in file "%s", using svn '
                     'specific symbolic references', f_path)
                 f_path = adjusted_f_path
                 symbolic_reference = self._symbolic_reference_svn
 
         branches = self._create_references(
             self.rhodecode_vcs_repo.branches, symbolic_reference, f_path, 'branch')
         branches_group = (branches, _("Branches"))
 
         tags = self._create_references(
             self.rhodecode_vcs_repo.tags, symbolic_reference, f_path, 'tag')
         tags_group = (tags, _("Tags"))
 
         history.append(branches_group)
         history.append(tags_group)
 
         return history, commits
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_file_history(self):
         self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         commit = self._get_commit_or_redirect(commit_id)
         file_node = self._get_filenode_or_redirect(commit, f_path)
 
         if file_node.is_file():
             file_history, _hist = self._get_node_history(commit, f_path)
 
             res = []
             for section_items, section in file_history:
                 items = []
                 for obj_id, obj_text, obj_type in section_items:
                     at_rev = ''
                     if obj_type in ['branch', 'bookmark', 'tag']:
                         at_rev = obj_text
                     entry = {
                         'id': obj_id,
                         'text': obj_text,
                         'type': obj_type,
                         'at_rev': at_rev
                     }
 
                     items.append(entry)
 
                 res.append({
                     'text': section,
                     'children': items
                 })
 
             data = {
                 'more': False,
                 'results': res
             }
             return data
 
         log.warning('Cannot fetch history for directory')
         raise HTTPBadRequest()
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator(
         'repository.read', 'repository.write', 'repository.admin')
     def repo_file_authors(self):
         c = self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         commit = self._get_commit_or_redirect(commit_id)
         file_node = self._get_filenode_or_redirect(commit, f_path)
 
         if not file_node.is_file():
             raise HTTPBadRequest()
 
         c.file_last_commit = file_node.last_commit
         if self.request.GET.get('annotate') == '1':
             # use _hist from annotation if annotation mode is on
             commit_ids = {x[1] for x in file_node.annotate}
             _hist = (
                 self.rhodecode_vcs_repo.get_commit(commit_id)
                 for commit_id in commit_ids)
         else:
             _f_history, _hist = self._get_node_history(commit, f_path)
         c.file_author = False
 
         unique = collections.OrderedDict()
         for commit in _hist:
             author = commit.author
             if author not in unique:
                 unique[commit.author] = [
                     h.email(author),
                     h.person(author, 'username_or_name_or_email'),
                     1  # counter
                 ]
 
             else:
                 # increase counter
                 unique[commit.author][2] += 1
 
         c.authors = [val for val in unique.values()]
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     def repo_files_check_head(self):
         self.load_default_context()
 
         commit_id, f_path = self._get_commit_and_path()
         _branch_name, _sha_commit_id, is_head = \
             self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                 landing_ref=self.db_repo.landing_ref_name)
 
         new_path = self.request.POST.get('path')
         operation = self.request.POST.get('operation')
         path_exist = ''
 
         if new_path and operation in ['create', 'upload']:
             new_f_path = os.path.join(f_path.lstrip('/'), new_path)
             try:
                 commit_obj = self.rhodecode_vcs_repo.get_commit(commit_id)
                 # NOTE(dan): construct whole path without leading /
                 file_node = commit_obj.get_node(new_f_path)
                 if file_node is not None:
                     path_exist = new_f_path
             except EmptyRepositoryError:
                 pass
             except Exception:
                 pass
 
         return {
             'branch': _branch_name,
             'sha': _sha_commit_id,
             'is_head': is_head,
             'path_exists': path_exist
         }
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     def repo_files_remove_file(self):
         _ = self.request.translate
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
         _branch_name, _sha_commit_id, is_head = \
             self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                 landing_ref=self.db_repo.landing_ref_name)
 
         self.forbid_non_head(is_head, f_path)
         self.check_branch_permission(_branch_name)
 
         c.commit = self._get_commit_or_redirect(commit_id)
         c.file = self._get_filenode_or_redirect(c.commit, f_path)
 
         c.default_message = _(
             'Deleted file {} via RhodeCode Enterprise').format(f_path)
         c.f_path = f_path
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_files_delete_file(self):
         _ = self.request.translate
 
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
         _branch_name, _sha_commit_id, is_head = \
             self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                 landing_ref=self.db_repo.landing_ref_name)
 
         self.forbid_non_head(is_head, f_path)
         self.check_branch_permission(_branch_name)
 
         c.commit = self._get_commit_or_redirect(commit_id)
         c.file = self._get_filenode_or_redirect(c.commit, f_path)
 
         c.default_message = _(
             'Deleted file {} via RhodeCode Enterprise').format(f_path)
         c.f_path = f_path
         node_path = f_path
         author = self._rhodecode_db_user.full_contact
         message = self.request.POST.get('message') or c.default_message
         try:
             nodes = {
                 safe_bytes(node_path): {
                     'content': b''
                 }
             }
             ScmModel().delete_nodes(
                 user=self._rhodecode_db_user.user_id, repo=self.db_repo,
                 message=message,
                 nodes=nodes,
                 parent_commit=c.commit,
                 author=author,
             )
 
             h.flash(
                 _('Successfully deleted file `{}`').format(
                     h.escape(f_path)), category='success')
         except Exception:
             log.exception('Error during commit operation')
             h.flash(_('Error occurred during commit'), category='error')
         raise HTTPFound(
             h.route_path('repo_commit', repo_name=self.db_repo_name,
                          commit_id='tip'))
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     def repo_files_edit_file(self):
         _ = self.request.translate
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
         _branch_name, _sha_commit_id, is_head = \
             self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                 landing_ref=self.db_repo.landing_ref_name)
 
         self.forbid_non_head(is_head, f_path, commit_id=commit_id)
         self.check_branch_permission(_branch_name, commit_id=commit_id)
 
         c.commit = self._get_commit_or_redirect(commit_id)
         c.file = self._get_filenode_or_redirect(c.commit, f_path)
 
         if c.file.is_binary:
             files_url = h.route_path(
                 'repo_files',
                 repo_name=self.db_repo_name,
                 commit_id=c.commit.raw_id, f_path=f_path)
             raise HTTPFound(files_url)
 
         c.default_message = _('Edited file {} via RhodeCode Enterprise').format(f_path)
         c.f_path = f_path
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_files_update_file(self):
         _ = self.request.translate
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
 
         c.commit = self._get_commit_or_redirect(commit_id)
         c.file = self._get_filenode_or_redirect(c.commit, f_path)
 
         if c.file.is_binary:
             raise HTTPFound(h.route_path('repo_files', repo_name=self.db_repo_name,
                                          commit_id=c.commit.raw_id, f_path=f_path))
 
         _branch_name, _sha_commit_id, is_head = \
             self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                 landing_ref=self.db_repo.landing_ref_name)
 
         self.forbid_non_head(is_head, f_path, commit_id=commit_id)
         self.check_branch_permission(_branch_name, commit_id=commit_id)
 
         c.default_message = _('Edited file {} via RhodeCode Enterprise').format(f_path)
         c.f_path = f_path
 
         old_content = c.file.str_content
         sl = old_content.splitlines(1)
         first_line = sl[0] if sl else ''
 
         r_post = self.request.POST
         # line endings:  0 - Unix, 1 - Mac, 2 - DOS
         line_ending_mode = detect_mode(first_line, 0)
         content = convert_line_endings(r_post.get('content', ''), line_ending_mode)
 
         message = r_post.get('message') or c.default_message
 
         org_node_path = c.file.str_path
         filename = r_post['filename']
 
         root_path = c.file.dir_path
         pure_path = self.create_pure_path(root_path, filename)
         node_path = pure_path.as_posix()
 
         default_redirect_url = h.route_path('repo_commit', repo_name=self.db_repo_name,
                                             commit_id=commit_id)
         if content == old_content and node_path == org_node_path:
             h.flash(_('No changes detected on {}').format(h.escape(org_node_path)),
                     category='warning')
             raise HTTPFound(default_redirect_url)
 
         try:
             mapping = {
                 c.file.bytes_path: {
                     'org_filename': org_node_path,
                     'filename': safe_bytes(node_path),
                     'content': safe_bytes(content),
                     'lexer': '',
                     'op': 'mod',
                     'mode': c.file.mode
                 }
             }
 
             commit = ScmModel().update_nodes(
                 user=self._rhodecode_db_user.user_id,
                 repo=self.db_repo,
                 message=message,
                 nodes=mapping,
                 parent_commit=c.commit,
             )
 
             h.flash(_('Successfully committed changes to file `{}`').format(
                     h.escape(f_path)), category='success')
             default_redirect_url = h.route_path(
                 'repo_commit', repo_name=self.db_repo_name, commit_id=commit.raw_id)
 
         except Exception:
             log.exception('Error occurred during commit')
             h.flash(_('Error occurred during commit'), category='error')
 
         raise HTTPFound(default_redirect_url)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     def repo_files_add_file(self):
         _ = self.request.translate
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
 
         # Check if we need to use this page to upload binary
         upload_binary = str2bool(self.request.params.get('upload_binary', False))
 
         c.commit = self._get_commit_or_redirect(commit_id, redirect_after=False)
         if c.commit is None:
             c.commit = EmptyCommit(alias=self.rhodecode_vcs_repo.alias)
 
         if self.rhodecode_vcs_repo.is_empty():
             # for empty repository we cannot check for current branch, we rely on
             # c.commit.branch instead
             _branch_name, _sha_commit_id, is_head = c.commit.branch, '', True
         else:
             _branch_name, _sha_commit_id, is_head = \
                 self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                     landing_ref=self.db_repo.landing_ref_name)
 
         self.forbid_non_head(is_head, f_path, commit_id=commit_id)
         self.check_branch_permission(_branch_name, commit_id=commit_id)
 
         c.default_message = (_('Added file via RhodeCode Enterprise')) \
             if not upload_binary else (_('Edited file {} via RhodeCode Enterprise').format(f_path))
         c.f_path = f_path.lstrip('/')  # ensure not relative path
         c.replace_binary = upload_binary
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_files_create_file(self):
         _ = self.request.translate
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
 
         c.commit = self._get_commit_or_redirect(commit_id, redirect_after=False)
         if c.commit is None:
             c.commit = EmptyCommit(alias=self.rhodecode_vcs_repo.alias)
 
         # calculate redirect URL
         if self.rhodecode_vcs_repo.is_empty():
             default_redirect_url = h.route_path(
                 'repo_summary', repo_name=self.db_repo_name)
         else:
             default_redirect_url = h.route_path(
                 'repo_commit', repo_name=self.db_repo_name, commit_id='tip')
 
         if self.rhodecode_vcs_repo.is_empty():
             # for empty repository we cannot check for current branch, we rely on
             # c.commit.branch instead
             _branch_name, _sha_commit_id, is_head = c.commit.branch, '', True
         else:
             _branch_name, _sha_commit_id, is_head = \
                 self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                     landing_ref=self.db_repo.landing_ref_name)
 
         self.forbid_non_head(is_head, f_path, commit_id=commit_id)
         self.check_branch_permission(_branch_name, commit_id=commit_id)
 
         c.default_message = (_('Added file via RhodeCode Enterprise'))
         c.f_path = f_path
 
         r_post = self.request.POST
         message = r_post.get('message') or c.default_message
         filename = r_post.get('filename')
         unix_mode = 0
 
         if not filename:
             # If there's no commit, redirect to repo summary
             if type(c.commit) is EmptyCommit:
                 redirect_url = h.route_path(
                     'repo_summary', repo_name=self.db_repo_name)
             else:
                 redirect_url = default_redirect_url
             h.flash(_('No filename specified'), category='warning')
             raise HTTPFound(redirect_url)
 
         root_path = f_path
         pure_path = self.create_pure_path(root_path, filename)
         node_path = pure_path.as_posix().lstrip('/')
 
         author = self._rhodecode_db_user.full_contact
         content = convert_line_endings(r_post.get('content', ''), unix_mode)
         nodes = {
             safe_bytes(node_path): {
                 'content': safe_bytes(content)
             }
         }
 
         try:
 
             commit = ScmModel().create_nodes(
                 user=self._rhodecode_db_user.user_id,
                 repo=self.db_repo,
                 message=message,
                 nodes=nodes,
                 parent_commit=c.commit,
                 author=author,
             )
 
             h.flash(_('Successfully committed new file `{}`').format(
                     h.escape(node_path)), category='success')
 
             default_redirect_url = h.route_path(
                 'repo_commit', repo_name=self.db_repo_name, commit_id=commit.raw_id)
 
         except NonRelativePathError:
             log.exception('Non Relative path found')
             h.flash(_('The location specified must be a relative path and must not '
                       'contain .. in the path'), category='warning')
             raise HTTPFound(default_redirect_url)
         except (NodeError, NodeAlreadyExistsError) as e:
             h.flash(h.escape(safe_str(e)), category='error')
         except Exception:
             log.exception('Error occurred during commit')
             h.flash(_('Error occurred during commit'), category='error')
 
         raise HTTPFound(default_redirect_url)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_files_upload_file(self):
         _ = self.request.translate
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
 
         c.commit = self._get_commit_or_redirect(commit_id, redirect_after=False)
         if c.commit is None:
             c.commit = EmptyCommit(alias=self.rhodecode_vcs_repo.alias)
 
         # calculate redirect URL
         if self.rhodecode_vcs_repo.is_empty():
             default_redirect_url = h.route_path(
                 'repo_summary', repo_name=self.db_repo_name)
         else:
             default_redirect_url = h.route_path(
                 'repo_commit', repo_name=self.db_repo_name, commit_id='tip')
 
         if self.rhodecode_vcs_repo.is_empty():
             # for empty repository we cannot check for current branch, we rely on
             # c.commit.branch instead
             _branch_name, _sha_commit_id, is_head = c.commit.branch, '', True
         else:
             _branch_name, _sha_commit_id, is_head = \
                 self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                     landing_ref=self.db_repo.landing_ref_name)
 
         error = self.forbid_non_head(is_head, f_path, json_mode=True)
         if error:
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
         error = self.check_branch_permission(_branch_name, json_mode=True)
         if error:
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
 
         c.default_message = (_('Added file via RhodeCode Enterprise'))
         c.f_path = f_path
 
         r_post = self.request.POST
 
         message = c.default_message
         user_message = r_post.getall('message')
         if isinstance(user_message, list) and user_message:
             # we take the first from duplicated results if it's not empty
             message = user_message[0] if user_message[0] else message
 
         nodes = {}
 
         for file_obj in r_post.getall('files_upload') or []:
             content = file_obj.file
             filename = file_obj.filename
 
             root_path = f_path
             pure_path = self.create_pure_path(root_path, filename)
             node_path = pure_path.as_posix().lstrip('/')
 
             nodes[safe_bytes(node_path)] = {
                 'content': content
             }
 
         if not nodes:
             error = 'missing files'
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
 
         author = self._rhodecode_db_user.full_contact
 
         try:
             commit = ScmModel().create_nodes(
                 user=self._rhodecode_db_user.user_id,
                 repo=self.db_repo,
                 message=message,
                 nodes=nodes,
                 parent_commit=c.commit,
                 author=author,
             )
             if len(nodes) == 1:
                 flash_message = _('Successfully committed {} new files').format(len(nodes))
             else:
                 flash_message = _('Successfully committed 1 new file')
 
             h.flash(flash_message, category='success')
 
             default_redirect_url = h.route_path(
                 'repo_commit', repo_name=self.db_repo_name, commit_id=commit.raw_id)
 
         except NonRelativePathError:
             log.exception('Non Relative path found')
             error = _('The location specified must be a relative path and must not '
                       'contain .. in the path')
             h.flash(error, category='warning')
 
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
         except (NodeError, NodeAlreadyExistsError) as e:
             error = h.escape(e)
             h.flash(error, category='error')
 
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
         except Exception:
             log.exception('Error occurred during commit')
             error = _('Error occurred during commit')
             h.flash(error, category='error')
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
 
         return {
             'error': None,
             'redirect_url': default_redirect_url
         }
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.write', 'repository.admin')
     @CSRFRequired()
     def repo_files_replace_file(self):
         _ = self.request.translate
         c = self.load_default_context()
         commit_id, f_path = self._get_commit_and_path()
 
         self._ensure_not_locked()
 
         c.commit = self._get_commit_or_redirect(commit_id, redirect_after=False)
         if c.commit is None:
             c.commit = EmptyCommit(alias=self.rhodecode_vcs_repo.alias)
 
         if self.rhodecode_vcs_repo.is_empty():
             default_redirect_url = h.route_path(
                 'repo_summary', repo_name=self.db_repo_name)
         else:
             default_redirect_url = h.route_path(
                 'repo_commit', repo_name=self.db_repo_name, commit_id='tip')
 
         if self.rhodecode_vcs_repo.is_empty():
             # for empty repository we cannot check for current branch, we rely on
             # c.commit.branch instead
             _branch_name, _sha_commit_id, is_head = c.commit.branch, '', True
         else:
             _branch_name, _sha_commit_id, is_head = \
                 self._is_valid_head(commit_id, self.rhodecode_vcs_repo,
                                     landing_ref=self.db_repo.landing_ref_name)
 
         error = self.forbid_non_head(is_head, f_path, json_mode=True)
         if error:
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
         error = self.check_branch_permission(_branch_name, json_mode=True)
         if error:
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
 
         c.default_message = (_('Edited file {} via RhodeCode Enterprise').format(f_path))
         c.f_path = f_path
 
         r_post = self.request.POST
 
         message = c.default_message
         user_message = r_post.getall('message')
         if isinstance(user_message, list) and user_message:
             # we take the first from duplicated results if it's not empty
             message = user_message[0] if user_message[0] else message
 
         data_for_replacement = r_post.getall('files_upload') or []
         if (objects_count := len(data_for_replacement)) > 1:
             return {
                 'error': 'too many files for replacement',
                 'redirect_url': default_redirect_url
             }
         elif not objects_count:
             return {
                 'error': 'missing files',
                 'redirect_url': default_redirect_url
             }
 
         content = data_for_replacement[0].file
         retrieved_filename = data_for_replacement[0].filename
 
         if retrieved_filename.split('.')[-1] != f_path.split('.')[-1]:
             return {
                 'error': 'file extension of uploaded file doesn\'t match an original file\'s extension',
                 'redirect_url': default_redirect_url
             }
 
         author = self._rhodecode_db_user.full_contact
 
         try:
             commit = ScmModel().update_binary_node(
                 user=self._rhodecode_db_user.user_id,
                 repo=self.db_repo,
                 message=message,
                 node={
                     'content': content,
                     'file_path': f_path.encode(),
                 },
                 parent_commit=c.commit,
                 author=author,
             )
 
             h.flash(_('Successfully committed 1 new file'), category='success')
 
             default_redirect_url = h.route_path(
                 'repo_commit', repo_name=self.db_repo_name, commit_id=commit.raw_id)
 
         except (NodeError, NodeAlreadyExistsError) as e:
             error = h.escape(e)
             h.flash(error, category='error')
 
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
         except Exception:
             log.exception('Error occurred during commit')
             error = _('Error occurred during commit')
             h.flash(error, category='error')
             return {
                 'error': error,
                 'redirect_url': default_redirect_url
             }
 
         return {
             'error': None,
             'redirect_url': default_redirect_url
         }

rhodecode/apps/repository/views/repo_settings_advanced.py

@@ -1,302 +1,309 b''
 # Copyright (C) 2011-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 
 
 from pyramid.httpexceptions import HTTPFound
 from packaging.version import Version
 
 from rhodecode import events
 from rhodecode.apps._base import RepoAppView
 from rhodecode.lib import helpers as h
 from rhodecode.lib import audit_logger
 from rhodecode.lib.auth import (
     LoginRequired, HasRepoPermissionAnyDecorator, CSRFRequired,
     HasRepoPermissionAny)
-from rhodecode.lib.exceptions import AttachedForksError, AttachedPullRequestsError
+from rhodecode.lib.exceptions import AttachedForksError, AttachedPullRequestsError, AttachedArtifactsError
 from rhodecode.lib.utils2 import safe_int
 from rhodecode.lib.vcs import RepositoryError
 from rhodecode.model.db import Session, UserFollowing, User, Repository
 from rhodecode.model.permission import PermissionModel
 from rhodecode.model.repo import RepoModel
 from rhodecode.model.scm import ScmModel
 
 log = logging.getLogger(__name__)
 
 
 class RepoSettingsAdvancedView(RepoAppView):
 
     def load_default_context(self):
         c = self._get_local_tmpl_context()
         return c
 
     def _get_users_with_permissions(self):
         user_permissions = {}
         for perm in self.db_repo.permissions():
             user_permissions[perm.user_id] = perm
 
         return user_permissions
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.admin')
     def edit_advanced(self):
         _ = self.request.translate
         c = self.load_default_context()
         c.active = 'advanced'
 
         c.default_user_id = User.get_default_user_id()
         c.in_public_journal = UserFollowing.query() \
             .filter(UserFollowing.user_id == c.default_user_id) \
             .filter(UserFollowing.follows_repository == self.db_repo).scalar()
 
         c.ver_info_dict = self.rhodecode_vcs_repo.get_hooks_info()
         c.hooks_outdated = False
 
         try:
             if Version(c.ver_info_dict['pre_version']) < Version(c.rhodecode_version):
                 c.hooks_outdated = True
         except Exception:
             pass
 
         # update commit cache if GET flag is present
         if self.request.GET.get('update_commit_cache'):
             self.db_repo.update_commit_cache()
             h.flash(_('updated commit cache'), category='success')
 
         return self._get_template_context(c)
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.admin')
     @CSRFRequired()
     def edit_advanced_archive(self):
         """
         Archives the repository. It will become read-only, and not visible in search
         or other queries. But still visible for super-admins.
         """
 
         _ = self.request.translate
 
         try:
             old_data = self.db_repo.get_api_data()
             RepoModel().archive(self.db_repo)
 
             repo = audit_logger.RepoWrap(repo_id=None, repo_name=self.db_repo.repo_name)
             audit_logger.store_web(
                 'repo.archive', action_data={'old_data': old_data},
                 user=self._rhodecode_user, repo=repo)
 
             ScmModel().mark_for_invalidation(self.db_repo_name, delete=True)
             h.flash(
                 _('Archived repository `%s`') % self.db_repo_name,
                 category='success')
             Session().commit()
         except Exception:
             log.exception("Exception during archiving of repository")
             h.flash(_('An error occurred during archiving of `%s`')
                     % self.db_repo_name, category='error')
             # redirect to advanced for more deletion options
             raise HTTPFound(
                 h.route_path('edit_repo_advanced', repo_name=self.db_repo_name,
                              _anchor='advanced-archive'))
 
         # flush permissions for all users defined in permissions
         affected_user_ids = self._get_users_with_permissions().keys()
         PermissionModel().trigger_permission_flush(affected_user_ids)
 
         raise HTTPFound(h.route_path('home'))
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.admin')
     @CSRFRequired()
     def edit_advanced_delete(self):
         """
         Deletes the repository, or shows warnings if deletion is not possible
         because of attached forks or other errors.
         """
         _ = self.request.translate
         handle_forks = self.request.POST.get('forks', None)
         if handle_forks == 'detach_forks':
             handle_forks = 'detach'
         elif handle_forks == 'delete_forks':
             handle_forks = 'delete'
 
+        repo_advanced_url = h.route_path(
+            'edit_repo_advanced', repo_name=self.db_repo_name,
+            _anchor='advanced-delete')
         try:
             old_data = self.db_repo.get_api_data()
             RepoModel().delete(self.db_repo, forks=handle_forks)
 
             _forks = self.db_repo.forks.count()
             if _forks and handle_forks:
                 if handle_forks == 'detach_forks':
                     h.flash(_('Detached %s forks') % _forks, category='success')
                 elif handle_forks == 'delete_forks':
                     h.flash(_('Deleted %s forks') % _forks, category='success')
 
             repo = audit_logger.RepoWrap(repo_id=None, repo_name=self.db_repo.repo_name)
             audit_logger.store_web(
                 'repo.delete', action_data={'old_data': old_data},
                 user=self._rhodecode_user, repo=repo)
 
             ScmModel().mark_for_invalidation(self.db_repo_name, delete=True)
             h.flash(
                 _('Deleted repository `%s`') % self.db_repo_name,
                 category='success')
             Session().commit()
         except AttachedForksError:
-            repo_advanced_url = h.route_path(
-                'edit_repo_advanced', repo_name=self.db_repo_name,
-                _anchor='advanced-delete')
             delete_anchor = h.link_to(_('detach or delete'), repo_advanced_url)
             h.flash(_('Cannot delete `{repo}` it still contains attached forks. '
                       'Try using {delete_or_detach} option.')
                     .format(repo=self.db_repo_name, delete_or_detach=delete_anchor),
                     category='warning')
 
             # redirect to advanced for forks handle action ?
             raise HTTPFound(repo_advanced_url)
 
         except AttachedPullRequestsError:
-            repo_advanced_url = h.route_path(
-                'edit_repo_advanced', repo_name=self.db_repo_name,
-                _anchor='advanced-delete')
             attached_prs = len(self.db_repo.pull_requests_source +
                                self.db_repo.pull_requests_target)
             h.flash(
                 _('Cannot delete `{repo}` it still contains {num} attached pull requests. '
                   'Consider archiving the repository instead.').format(
                     repo=self.db_repo_name, num=attached_prs), category='warning')
 
             # redirect to advanced for forks handle action ?
             raise HTTPFound(repo_advanced_url)
 
+        except AttachedArtifactsError:
+
+            attached_artifacts = len(self.db_repo.artifacts)
+            h.flash(
+                _('Cannot delete `{repo}` it still contains {num} attached artifacts. '
+                  'Consider archiving the repository instead.').format(
+                    repo=self.db_repo_name, num=attached_artifacts), category='warning')
+
+            # redirect to advanced for forks handle action ?
+            raise HTTPFound(repo_advanced_url)
         except Exception:
             log.exception("Exception during deletion of repository")
             h.flash(_('An error occurred during deletion of `%s`')
                     % self.db_repo_name, category='error')
             # redirect to advanced for more deletion options
             raise HTTPFound(
                 h.route_path('edit_repo_advanced', repo_name=self.db_repo_name,
                              _anchor='advanced-delete'))
 
         raise HTTPFound(h.route_path('home'))
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.admin')
     @CSRFRequired()
     def edit_advanced_journal(self):
         """
         Set's this repository to be visible in public journal,
         in other words making default user to follow this repo
         """
         _ = self.request.translate
 
         try:
             user_id = User.get_default_user_id()
             ScmModel().toggle_following_repo(self.db_repo.repo_id, user_id)
             h.flash(_('Updated repository visibility in public journal'),
                     category='success')
             Session().commit()
         except Exception:
             h.flash(_('An error occurred during setting this '
                       'repository in public journal'),
                     category='error')
 
         raise HTTPFound(
             h.route_path('edit_repo_advanced', repo_name=self.db_repo_name))
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.admin')
     @CSRFRequired()
     def edit_advanced_fork(self):
         """
         Mark given repository as a fork of another
         """
         _ = self.request.translate
 
         new_fork_id = safe_int(self.request.POST.get('id_fork_of'))
 
         # valid repo, re-check permissions
         if new_fork_id:
             repo = Repository.get(new_fork_id)
             # ensure we have at least read access to the repo we mark
             perm_check = HasRepoPermissionAny(
                 'repository.read', 'repository.write', 'repository.admin')
 
             if repo and perm_check(repo_name=repo.repo_name):
                 new_fork_id = repo.repo_id
             else:
                 new_fork_id = None
 
         try:
             repo = ScmModel().mark_as_fork(
                 self.db_repo_name, new_fork_id, self._rhodecode_user.user_id)
             fork = repo.fork.repo_name if repo.fork else _('Nothing')
             Session().commit()
             h.flash(
                 _('Marked repo %s as fork of %s') % (self.db_repo_name, fork),
                 category='success')
         except RepositoryError as e:
             log.exception("Repository Error occurred")
             h.flash(str(e), category='error')
         except Exception:
             log.exception("Exception while editing fork")
             h.flash(_('An error occurred during this operation'),
                     category='error')
 
         raise HTTPFound(
             h.route_path('edit_repo_advanced', repo_name=self.db_repo_name))
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.admin')
     @CSRFRequired()
     def edit_advanced_toggle_locking(self):
         """
         Toggle locking of repository
         """
         _ = self.request.translate
         set_lock = self.request.POST.get('set_lock')
         set_unlock = self.request.POST.get('set_unlock')
 
         try:
             if set_lock:
                 Repository.lock(self.db_repo, self._rhodecode_user.user_id,
                                 lock_reason=Repository.LOCK_WEB)
                 h.flash(_('Locked repository'), category='success')
             elif set_unlock:
                 Repository.unlock(self.db_repo)
                 h.flash(_('Unlocked repository'), category='success')
         except Exception as e:
             log.exception("Exception during unlocking")
             h.flash(_('An error occurred during unlocking'), category='error')
 
         raise HTTPFound(
             h.route_path('edit_repo_advanced', repo_name=self.db_repo_name))
 
     @LoginRequired()
     @HasRepoPermissionAnyDecorator('repository.admin')
     def edit_advanced_install_hooks(self):
         """
         Install Hooks for repository
         """
         _ = self.request.translate
         self.load_default_context()
         self.rhodecode_vcs_repo.install_hooks(force=True)
         h.flash(_('installed updated hooks into this repository'),
                 category='success')
         raise HTTPFound(
             h.route_path('edit_repo_advanced', repo_name=self.db_repo_name))

rhodecode/apps/ssh_support/__init__.py

@@ -1,60 +1,60 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 import logging
 
 from . import config_keys
 
 from rhodecode.config.settings_maker import SettingsMaker
 
 log = logging.getLogger(__name__)
 
 
 def _sanitize_settings_and_apply_defaults(settings):
     """
     Set defaults, convert to python types and validate settings.
     """
     settings_maker = SettingsMaker(settings)
 
     settings_maker.make_setting(config_keys.generate_authorized_keyfile, False, parser='bool')
     settings_maker.make_setting(config_keys.wrapper_allow_shell, False, parser='bool')
     settings_maker.make_setting(config_keys.enable_debug_logging, False, parser='bool')
     settings_maker.make_setting(config_keys.ssh_key_generator_enabled, True, parser='bool')
 
     settings_maker.make_setting(config_keys.authorized_keys_file_path, '~/.ssh/authorized_keys_rhodecode')
-    settings_maker.make_setting(config_keys.wrapper_cmd, '')
+    settings_maker.make_setting(config_keys.wrapper_cmd, '/usr/local/bin/rhodecode_bin/bin/rc-ssh-wrapper-v2')
     settings_maker.make_setting(config_keys.authorized_keys_line_ssh_opts, '')
 
     settings_maker.make_setting(config_keys.ssh_hg_bin, '/usr/local/bin/rhodecode_bin/vcs_bin/hg')
     settings_maker.make_setting(config_keys.ssh_git_bin, '/usr/local/bin/rhodecode_bin/vcs_bin/git')
     settings_maker.make_setting(config_keys.ssh_svn_bin, '/usr/local/bin/rhodecode_bin/vcs_bin/svnserve')
 
     settings_maker.env_expand()
 
 
 def includeme(config):
     settings = config.registry.settings
     _sanitize_settings_and_apply_defaults(settings)
 
     # if we have enable generation of file, subscribe to event
     if settings[config_keys.generate_authorized_keyfile]:
         # lazy import here for faster code reading... via sshwrapper-v2 mode
         from .subscribers import generate_ssh_authorized_keys_file_subscriber
         from .events import SshKeyFileChangeEvent
         config.add_subscriber(
             generate_ssh_authorized_keys_file_subscriber, SshKeyFileChangeEvent)

rhodecode/apps/ssh_support/config_keys.py

@@ -1,32 +1,32 b''
 # Copyright (C) 2016-2023 RhodeCode GmbH
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License, version 3
 # (only), as published by the Free Software Foundation.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 # This program is dual-licensed. If you wish to learn more about the
 # RhodeCode Enterprise Edition, including its added features, Support services,
 # and proprietary license terms, please see https://rhodecode.com/licenses/
 
 
 # Definition of setting keys used to configure this module. Defined here to
 # avoid repetition of keys throughout the module.
 generate_authorized_keyfile = 'ssh.generate_authorized_keyfile'
 authorized_keys_file_path = 'ssh.authorized_keys_file_path'
 authorized_keys_line_ssh_opts = 'ssh.authorized_keys_ssh_opts'
 ssh_key_generator_enabled = 'ssh.enable_ui_key_generator'
-wrapper_cmd = 'ssh.wrapper_cmd'
+wrapper_cmd = 'ssh.wrapper_cmd.v2'
 wrapper_allow_shell = 'ssh.wrapper_cmd_allow_shell'
 enable_debug_logging = 'ssh.enable_debug_logging'
 
 ssh_hg_bin = 'ssh.executable.hg'
 ssh_git_bin = 'ssh.executable.git'
 ssh_svn_bin = 'ssh.executable.svn'