release: Merge default into stable for release preparation

.babelrc

@@ -0,0 +1,10
+{
+  "presets": [
+    ["env", {
+      "targets": {
+        "browsers": ["last 2 versions"]
+      }
+    }]
+  ],
+  "plugins": ["transform-object-rest-spread"]
+}

docs/integrations/integrations-rcextensions.rst

@@ -0,0 +1,44
+.. _integrations-rcextensions:
+
+
+rcextensions integrations
+=========================
+
+
+Since RhodeCode 4.14 release rcextensions aren't part of rhodecode-tools, and instead
+they are shipped with the new or upgraded installations.
+
+The rcextensions template `rcextensions.tmpl` is created in the `etc/` directory
+of enterprise or community installation. It's always re-created and updated on upgrades.
+
+
+Activating rcextensions
++++++++++++++++++++++++
+
+To activate rcextensions simply copy or rename the created template rcextensions
+into the path where the rhodecode.ini file is located::
+
+    pushd ~/rccontrol/enterprise-1/
+    or
+    pushd ~/rccontrol/community-1/
+
+    mv etc/rcextensions.tmpl rcextensions
+
+
+rcextensions are loaded when |RCE| starts. So a restart is required after activation or
+change of code in rcextensions.
+
+Simply restart only the enterprise/community instance::
+
+    rccontrol restart enterprise-1
+    or
+    rccontrol restart community-1
+
+
+Example usage
++++++++++++++
+
+
+To see examples of usage please check the examples directory under:
+
+https://code.rhodecode.com/rhodecode-enterprise-ce/files/stable/rhodecode/config/rcextensions/examples

docs/release-notes/release-notes-4.14.0.rst

@@ -0,0 +1,114
+|RCE| 4.14.0 |RNS|
+------------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2018-11-02
+
+
+New Features
+^^^^^^^^^^^^
+
+- Diffs: expose range diff inside the PR view. It's now possible to show
+  commit-per-commit view of changes in pull request.
+- Diffs: new sticky context bar.
+  When browsing diffs we show file path of the current diff so users are aware all the time
+  what file they are reviewing.
+- Diffs: added quick file selector in diffs views. Now it's possible to select a file
+  in large diffs from the sticky header for quicker access to certain interesting files
+  in diffs.
+- Diffs: introducing diff menu for whitespace toggle and context changes.
+  It's now possible to show/hide whitespace changes and toggle the file context in
+  all diff places including pull requests.
+- Comments: allow commenting on empty files without content.
+- Repositories: added option to archive repositories instead of deleting them.
+  Archived repositories are useful for future auditing, but they are read-only.
+- rcextensions: new rcextensions. We're introducing new `rcextensions` that will be base
+  for future low-level integrations. It's now possible to expose nice messages back
+  to the users when using `rcextensions`.
+- Summary page: slightly re-organize summary page for better user experience.
+
+
+General
+^^^^^^^
+
+- Mailing: switched from custom library to pyramid_mailer with python3 compatibility.
+- Frontend: Switched to Polymer 3.0.
+- Frontend: fixed problems with IE11 and brought back support for that browser.
+- Git: use a fetch_sync based creation of remote repositories.
+  This fixes problems with importing from Bitbucket.
+- Comments: update comments email templates.
+- Packaging: only wrap external dependency scripts. This makes execution of scripts
+  roughly 5x faster due to much smaller PATH tree.
+- HTTP: use application wide detection of invalid bytes sent via URL/GET/POST data.
+- Fonts/UI: use consistent fonts across the whole application.
+  Few places had non-standard custom fonts.
+- Google: updated google auth plugin with latest API changes.
+- Core: handle edge case requesting matched routes but with hg/svn/git or api context.
+- Dependencies: bumped rhodecode-tools to 1.0.0 release using Apache2 license.
+- Dependencies: atomicwrites==1.2.1
+- Dependencies: attrs==18.2.0
+- Dependencies: dogpile.cache==0.6.7
+- Dependencies: psutil==5.4.7
+- Dependencies: pathlib2==2.3.2
+- Dependencies: subprocess32==3.5.2
+- Dependencies: gevent==1.3.6
+- Dependencies: greenlet==0.4.15
+- Dependencies: pytest==3.8.2
+- Dependencies: py==1.6.0
+- Dependencies: pytest-cov==2.6.0
+- Dependencies: pytest-timeout==1.3.2
+- Dependencies: coverage==4.5.1
+- Dependencies: psycopg2==2.7.5
+
+
+Security
+^^^^^^^^
+
+- RST: improve Javascript RST sandbox.
+- Jupyter: sanitize markdown cells similar as we do for our own markdown cleanup.
+
+
+Performance
+^^^^^^^^^^^
+
+- SSH: improved SSH wrapper execution speed by using optimized binary script wrapping.
+- Core: reduced font and JavaScript load times.
+
+
+Fixes
+^^^^^
+
+- Comments: ensure we always display unmatched comments.
+- Branch Permissions: fixed changing rule order for branch permissions.
+- Users: ensure get_first_superadmin actually gets the 1st created super-admin.
+- Users: when deleting users ensure we also clear personal flag.
+  This fixes some problems with multiple personal groups.
+- Diffs: disable the error border on highlight errors.
+- Integrations: implement retry to HTTP[S] calls for integrations.
+  Web parts will do a 3x retry call in case service is not reachable or
+  responds with 5XX codes.
+- Git: fixed pull-request updates in case branch names are the same as the file names.
+- Supervisor: add patch for older kernel support.
+- Compare: fixed file after/before links in compare view for cross repo compare.
+- Emails: improve fonts and rendering of email HTML.
+- Permissions: flush members of user groups permissions to clear caches.
+- Repository: add check preventing of removal of repo with attached pull requests. Users
+  should use the new archive repo function instead.
+
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+- In this release, we're shipping a new `rcextensions`. The changes made are
+  backward incompatible. An update of `rcextensions` is required
+  prior to using them again. Please check the new `rcextensions.tmpl` directory
+  located in `etc/rcextensions.tmpl` in your instance installation path.
+  Old code should be 100% portable by just copy&paste to the right function.
+
+- Mailing: We introduced a new mailing library. The older options should be compatible and
+  generally, old configuration doesn't need any changes in order to send emails.
+  We, however, encourage users to re-check mailing setup in case of some more
+  sophisticated email setups.
+  There's a possibility to send a test email from admin > settings > email section.

pkgs/overlays.nix

@@ -0,0 +1,3
+self: super: {
+
+}

pkgs/patches/supervisor/patch-rlimits-old-kernel.diff

@@ -0,0 +1,16
+diff -rup supervisor-3.3.4-orig/supervisor/options.py supervisor-3.3.4/supervisor/options.py
+--- supervisor-3.3.4-orig/supervisor/options.py	1970-01-01 01:00:01.000000000 +0100
++++ supervisor-3.3.4/supervisor-new/options.py	2018-10-24 10:53:19.368503735 +0200
+@@ -1395,7 +1395,11 @@ class ServerOptions(Options):
+             name = limit['name']
+             name = name # name is used below by locals()
+
+-            soft, hard = resource.getrlimit(res)
++            try:
++                soft, hard = resource.getrlimit(res)
++            except Exception:
++                # handle old kernel problems, this is not critical to execute
++                soft, hard = -1, -1
+
+             if (soft < min) and (soft != -1): # -1 means unlimited
+                 if (hard < min) and (hard != -1):

requirements_debug.txt

@@ -0,0 +1,3
+## special libraries we could extend the requirements.txt file with to add some
+## custom libraries usefull for debug and memory tracing
+objgraph==3.1.1

.bumpversion.cfg

@@ -1,5 +1,5
 [bumpversion]
-current_version = 4.13.3
+current_version = 4.14.0
 message = release: Bump version {current_version} to {new_version}
 
 [bumpversion:file:rhodecode/VERSION]

.hgignore

@@ -24,7 +24,6 syntax: regexp
 ^.dev
 ^._dev
 ^build/
-^bower_components/
 ^coverage\.xml$
 ^data$
 ^\.eggs/
@@ -36,14 +35,15 syntax: regexp
 ^htmlcov
 ^junit\.xml$
 ^node_modules/
+^node_binaries/
 ^pylint.log$
 ^rcextensions/
 ^result$
 ^rhodecode/public/css/style.css$
 ^rhodecode/public/css/style-polymer.css$
 ^rhodecode/public/js/rhodecode-components.html$
+^rhodecode/public/js/rhodecode-components.js$
 ^rhodecode/public/js/scripts.js$
-^rhodecode/public/js/rhodecode-components.js$
 ^rhodecode/public/js/src/components/root-styles.gen.html$
 ^rhodecode/public/js/vendors/webcomponentsjs/
 ^rhodecode\.db$

.release.cfg

@@ -5,25 +5,20 done = false
 done = true
 
 [task:rc_tools_pinned]
-done = true
 
 [task:fixes_on_stable]
-done = true
 
 [task:pip2nix_generated]
-done = true
 
 [task:changelog_updated]
-done = true
 
 [task:generate_api_docs]
-done = true
+
+[task:updated_translation]
 
 [release]
-state = prepared
-version = 4.13.3
-
-[task:updated_translation]
+state = in_progress
+version = 4.14.0
 
 [task:generate_js_routes]
 

Gruntfile.js

@@ -1,4 +1,12
 var gruntConfig = require('./grunt_config.json');
+var webpackConfig = require('./webpack.config');
+gruntConfig["webpack"] = {
+    options: {
+        stats: !process.env.NODE_ENV || process.env.NODE_ENV === 'development'
+    },
+    prod: webpackConfig,
+    dev: Object.assign({ watch: false }, webpackConfig)
+};
 
 module.exports = function(grunt) {
   grunt.initConfig(gruntConfig);
@@ -7,9 +15,7 module.exports = function(grunt) {
   grunt.loadNpmTasks('grunt-contrib-concat');
   grunt.loadNpmTasks('grunt-contrib-watch');
   grunt.loadNpmTasks('grunt-contrib-jshint');
-  grunt.loadNpmTasks('grunt-vulcanize');
-  grunt.loadNpmTasks('grunt-crisper');
   grunt.loadNpmTasks('grunt-contrib-copy');
-
-  grunt.registerTask('default', ['less:production', 'less:components', 'concat:polymercss', 'copy', 'concat:dist', 'vulcanize', 'crisper']);
+  grunt.loadNpmTasks('grunt-webpack');
+  grunt.registerTask('default', ['less:production', 'less:components', 'copy', 'webpack', 'concat:dist']);
 };

MANIFEST.in

@@ -39,7 +39,6 include rhodecode/public/sounds/*.mp3
 include rhodecode/public/sounds/*.wav
 
 # fonts
-recursive-include rhodecode/public/fonts/ProximaNova *
 recursive-include rhodecode/public/fonts/RCIcons *
 
 # js

Makefile

@@ -1,9 +1,9
 
 .PHONY: clean docs docs-clean docs-cleanup test test-clean test-only test-only-postgres test-only-mysql web-build
 
-WEBPACK=./node_modules/webpack/bin/webpack.js
-GRUNT=grunt
 NODE_PATH=./node_modules
+WEBPACK=./node_binaries/webpack
+GRUNT=./node_binaries/grunt
 
 
 clean:

configs/development.ini

@@ -26,8 +26,6 debug = true
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
-## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
-#smtp_auth =
 
 [server:main]
 ## COMMON ##
@@ -281,8 +279,10 supervisor.group_id = dev
 ## Display extended labs settings
 labs_settings_active = true
 
-## custom exception store path, defaults to TMPDIR
-exception_tracker.store_path =
+## Custom exception store path, defaults to TMPDIR
+## This is used to store exception from RhodeCode in shared directory
+#exception_tracker.store_path =
+
 
 ####################################
 ###        CELERY CONFIG        ####
@@ -506,7 +506,7 debug_style = true
 ###########################################
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
-#sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
+#sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode?charset=utf8
 # pymysql is an alternative driver for MySQL, use in case of problems with default one
 #sqlalchemy.db1.url = mysql+pymysql://root:qweqwe@localhost/rhodecode
 

configs/production.ini

@@ -26,8 +26,6 debug = true
 #smtp_port =
 #smtp_use_tls = false
 #smtp_use_ssl = true
-## Specify available auth parameters here (e.g. LOGIN PLAIN CRAM-MD5, etc.)
-#smtp_auth =
 
 [server:main]
 ## COMMON ##
@@ -256,8 +254,9 supervisor.group_id = prod
 ## Display extended labs settings
 labs_settings_active = true
 
-## custom exception store path, defaults to TMPDIR
-exception_tracker.store_path =
+## Custom exception store path, defaults to TMPDIR
+## This is used to store exception from RhodeCode in shared directory
+#exception_tracker.store_path =
 
 
 ####################################
@@ -480,7 +479,7 set debug = false
 ###########################################
 #sqlalchemy.db1.url = sqlite:///%(here)s/rhodecode.db?timeout=30
 #sqlalchemy.db1.url = postgresql://postgres:qweqwe@localhost/rhodecode
-#sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode
+#sqlalchemy.db1.url = mysql://root:qweqwe@localhost/rhodecode?charset=utf8
 # pymysql is an alternative driver for MySQL, use in case of problems with default one
 #sqlalchemy.db1.url = mysql+pymysql://root:qweqwe@localhost/rhodecode
 

default.nix

@@ -9,7 +9,7
 #  {
 #    # Thoughts on how to configure the dev environment
 #    rc = {
-#      codeInternalUrl = "https://usr:[e-mail]";
+#      codeInternalUrl = "https://usr:[e-mail]/internal";
 #      sources = {
 #        rhodecode-vcsserver = "/home/user/work/rhodecode-vcsserver";
 #        rhodecode-enterprise-ce = "/home/user/work/rhodecode-enterprise-ce";
@@ -26,10 +26,18 args@
 }:
 
 let
-  # Use nixpkgs from args or import them. We use this indirect approach
-  # through args to be able to use the name `pkgs` for our customized packages.
-  # Otherwise we will end up with an infinite recursion.
-  pkgs = args.pkgs or (import <nixpkgs> { });
+  pkgs_ = (import <nixpkgs> {});
+in
+
+let
+  pkgs = import <nixpkgs> {
+    overlays = [
+      (import ./pkgs/overlays.nix)
+    ];
+    inherit
+      (pkgs_)
+      system;
+  };
 
   # Works with the new python-packages, still can fallback to the old
   # variant.
@@ -46,7 +54,7 let
     in
       !builtins.elem (basename path) [
         ".git" ".hg" "__pycache__" ".eggs" ".idea" ".dev"
-        "bower_components" "node_modules"
+        "node_modules" "node_binaries"
         "build" "data" "result" "tmp"] &&
       !builtins.elem ext ["egg-info" "pyc"] &&
       # TODO: johbo: This check is wrong, since "path" contains an absolute path,
@@ -55,7 +63,11 let
 
   sources =
     let
-      inherit (pkgs.lib) all isString attrValues;
+      inherit
+        (pkgs.lib)
+        all
+        isString
+        attrValues;
       sourcesConfig = pkgs.config.rc.sources or {};
     in
       # Ensure that sources are configured as strings. Using a path
@@ -66,18 +78,12 let
   version = builtins.readFile "${rhodecode-enterprise-ce-src}/rhodecode/VERSION";
   rhodecode-enterprise-ce-src = builtins.filterSource src-filter ./.;
 
-  buildBowerComponents = pkgs.buildBowerComponents;
   nodeEnv = import ./pkgs/node-default.nix {
-    inherit pkgs;
+    inherit
+      pkgs;
   };
   nodeDependencies = nodeEnv.shell.nodeDependencies;
 
-  bowerComponents = buildBowerComponents {
-    name = "enterprise-ce-${version}";
-    generated = ./pkgs/bower-packages.nix;
-    src = rhodecode-enterprise-ce-src;
-  };
-
   rhodecode-testdata-src = sources.rhodecode-testdata or (
     pkgs.fetchhg {
       url = "https://code.rhodecode.com/upstream/rc_testdata";
@@ -95,23 +101,22 let
   pythonLocalOverrides = self: super: {
     rhodecode-enterprise-ce =
       let
-        linkNodeAndBowerPackages = ''
+        linkNodePackages = ''
           export RHODECODE_CE_PATH=${rhodecode-enterprise-ce-src}
 
-          echo "[BEGIN]: Link node packages"
-          rm -fr node_modules
-          mkdir node_modules
+          echo "[BEGIN]: Link node packages and binaries"
           # johbo: Linking individual packages allows us to run "npm install"
           # inside of a shell to try things out. Re-entering the shell will
           # restore a clean environment.
+          rm -fr node_modules
+          mkdir node_modules
           ln -s ${nodeDependencies}/lib/node_modules/* node_modules/
-          echo "[DONE]: Link node packages"
+          export NODE_PATH=./node_modules
 
-          echo "[BEGIN]: Link bower packages"
-          rm -fr bower_components
-          mkdir bower_components
-          ln -s ${bowerComponents}/bower_components/* bower_components/
-          echo "[DONE]: Link bower packages"
+          rm -fr node_binaries
+          mkdir node_binaries
+          ln -s ${nodeDependencies}/bin/* node_binaries/
+          echo "[DONE ]: Link node packages and binaries"
         '';
 
         releaseName = "RhodeCodeEnterpriseCE-${version}";
@@ -129,8 +134,7 let
       passthru = {
         inherit
           rhodecode-testdata
-          bowerComponents
-          linkNodeAndBowerPackages
+          linkNodePackages
           myPythonPackagesUnfix
           pythonLocalOverrides
           pythonCommunityOverrides;
@@ -141,8 +145,6 let
       buildInputs =
         attrs.buildInputs or [] ++ [
           rhodecode-testdata
-          pkgs.nodePackages.bower
-          pkgs.nodePackages.grunt-cli
         ];
 
       #NOTE: option to inject additional propagatedBuildInputs
@@ -175,14 +177,28 let
       '';
 
       preBuild = ''
-
-        echo "Building frontend assets"
-        ${linkNodeAndBowerPackages}
-        grunt
+        echo "[BEGIN]: Building frontend assets"
+        ${linkNodePackages}
+        make web-build
         rm -fr node_modules
+        rm -fr node_binaries
+        echo "[DONE ]: Building frontend assets"
       '';
 
       postInstall = ''
+        # check required files
+        STATIC_CHECK="/robots.txt /502.html
+                      /js/scripts.js /js/rhodecode-components.js
+                      /css/style.css /css/style-polymer.css"
+
+        for file in $STATIC_CHECK;
+        do
+            if [ ! -f rhodecode/public/$file ]; then
+              echo "Missing $file"
+              exit 1
+            fi
+        done
+
         echo "Writing enterprise-ce meta information for rccontrol to nix-support/rccontrol"
         mkdir -p $out/nix-support/rccontrol
         cp -v rhodecode/VERSION $out/nix-support/rccontrol/version
@@ -192,22 +208,35 let
         cp configs/production.ini $out/etc
         echo "[DONE]: saved enterprise-ce production.ini into $out/etc"
 
+        cp -Rf rhodecode/config/rcextensions $out/etc/rcextensions.tmpl
+        echo "[DONE ]: saved enterprise-ce rcextensions into $out/etc/rcextensions.tmpl"
+
         # python based programs need to be wrapped
         mkdir -p $out/bin
-        # rhodecode-tools
-        ln -s ${self.rhodecode-tools}/bin/rhodecode-* $out/bin/
 
         # required binaries from dependencies
-        #ln -s ${self.python}/bin/python $out/bin
-        ln -s ${self.pyramid}/bin/* $out/bin/
-        ln -s ${self.gunicorn}/bin/gunicorn $out/bin/
-        ln -s ${self.supervisor}/bin/supervisor* $out/bin/
+        ln -s ${self.supervisor}/bin/supervisorctl $out/bin/
+        ln -s ${self.supervisor}/bin/supervisord $out/bin/
         ln -s ${self.pastescript}/bin/paster $out/bin/
         ln -s ${self.channelstream}/bin/channelstream $out/bin/
         ln -s ${self.celery}/bin/celery $out/bin/
-        echo "[DONE]: created symlinks into $out/bin"
+        ln -s ${self.gunicorn}/bin/gunicorn $out/bin/
+        ln -s ${self.pyramid}/bin/prequest $out/bin/
+        ln -s ${self.pyramid}/bin/pserve $out/bin/
 
-        for file in $out/bin/*;
+        echo "[DONE ]: created symlinks into $out/bin"
+        DEPS="$out/bin/supervisorctl \
+              $out/bin/supervisord \
+              $out/bin/paster \
+              $out/bin/channelstream \
+              $out/bin/celery \
+              $out/bin/gunicorn \
+              $out/bin/prequest \
+              $out/bin/pserve"
+
+        # wrap only dependency scripts, they require to have full PYTHONPATH set
+        # to be able to import all packages
+        for file in $DEPS;
         do
           wrapProgram $file \
             --prefix PATH : $PATH \
@@ -217,14 +246,16 let
 
         echo "[DONE]: enterprise-ce binary wrapping"
 
-        if [ ! -f rhodecode/public/js/scripts.js ]; then
-          echo "Missing scripts.js"
-          exit 1
-        fi
-        if [ ! -f rhodecode/public/css/style.css ]; then
-          echo "Missing style.css"
-          exit 1
-        fi
+        # rhodecode-tools don't need wrapping
+        ln -s ${self.rhodecode-tools}/bin/rhodecode-* $out/bin/
+
+        # expose sources of CE
+        ln -s $out $out/etc/rhodecode_enterprise_ce_source
+
+        # expose static files folder
+        cp -Rf $out/lib/${self.python.libPrefix}/site-packages/rhodecode/public/ $out/etc/static
+        chmod 755 -R $out/etc/static
+
       '';
     });
 
@@ -237,8 +268,13 let
       getAttr pythonPackages pkgs;
 
   pythonGeneratedPackages = import ./pkgs/python-packages.nix {
-    inherit pkgs;
-    inherit (pkgs) fetchurl fetchgit fetchhg;
+    inherit
+      pkgs;
+    inherit
+      (pkgs)
+      fetchurl
+      fetchgit
+      fetchhg;
   };
 
   pythonCommunityOverrides = import ./pkgs/python-packages-overrides.nix {

docs/admin/admin-tricks.rst

@@ -162,7 +162,8 To add custom hooks to your instance, us
 
 1. Open :menuselection:`Admin --> Settings --> Hooks`
 2. Add your custom hook details, you can use a file path to specify custom
-   hook scripts, for example: ``python:/path/to/custom_hook.py``
+   hook scripts, for example:
+   ``pretxnchangegroup.example`` with value ``python:/path/to/custom_hook.py:my_func_name``
 3. Select :guilabel:`Save`
 
 Also, see the |RC| Extensions section of the :ref:`rc-tools` guide. |RC|

docs/admin/svn-http.rst

@@ -18,8 +18,8 Prerequisites
 .. tip::
 
    We recommend using Wandisco repositories which provide latest SVN versions
-   for most platforms. Below is example how to add the wandisco repositories
-   for Ubuntu.
+   for most platforms.
+   Here is an example how to add the Wandisco repositories for Ubuntu.
 
     .. code-block:: bash
 
@@ -27,15 +27,28 Prerequisites
         $ sudo wget -q http://opensource.wandisco.com/wandisco-debian.gpg -O- | sudo apt-key add -
         $ sudo apt-get update