Show More
@@ -1,125 +1,125 b'' | |||||
1 | .. _performance: |
|
1 | .. _performance: | |
2 |
|
2 | |||
3 | ================================ |
|
3 | ================================ | |
4 | Optimizing Kallithea performance |
|
4 | Optimizing Kallithea performance | |
5 | ================================ |
|
5 | ================================ | |
6 |
|
6 | |||
7 | When serving a large amount of big repositories, Kallithea can start performing |
|
7 | When serving a large amount of big repositories, Kallithea can start performing | |
8 | slower than expected. Because of the demanding nature of handling large amounts |
|
8 | slower than expected. Because of the demanding nature of handling large amounts | |
9 | of data from version control systems, here are some tips on how to get the best |
|
9 | of data from version control systems, here are some tips on how to get the best | |
10 | performance. |
|
10 | performance. | |
11 |
|
11 | |||
12 |
|
12 | |||
13 | Fast storage |
|
13 | Fast storage | |
14 | ------------ |
|
14 | ------------ | |
15 |
|
15 | |||
16 | Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM |
|
16 | Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM | |
17 | is usually more important than a fast CPU. |
|
17 | is usually more important than a fast CPU. | |
18 |
|
18 | |||
19 |
|
19 | |||
20 | Caching |
|
20 | Caching | |
21 | ------- |
|
21 | ------- | |
22 |
|
22 | |||
23 | Tweak beaker cache settings in the ini file. The actual effect of that is |
|
23 | Tweak beaker cache settings in the ini file. The actual effect of that is | |
24 | questionable. |
|
24 | questionable. | |
25 |
|
25 | |||
26 | .. note:: |
|
26 | .. note:: | |
27 |
|
27 | |||
28 | Beaker has no upper bound on cache size and will never drop any caches. For |
|
28 | Beaker has no upper bound on cache size and will never drop any caches. For | |
29 | memory cache, the only option is to regularly restart the worker process. |
|
29 | memory cache, the only option is to regularly restart the worker process. | |
30 | For file cache, it must be cleaned manually, as described in the `Beaker |
|
30 | For file cache, it must be cleaned manually, as described in the `Beaker | |
31 | documentation <https://beaker.readthedocs.io/en/latest/sessions.html#removing-expired-old-sessions>`_:: |
|
31 | documentation <https://beaker.readthedocs.io/en/latest/sessions.html#removing-expired-old-sessions>`_:: | |
32 |
|
32 | |||
33 | find data/cache -type f -mtime +30 -print -exec rm {} \; |
|
33 | find data/cache -type f -mtime +30 -print -exec rm {} \; | |
34 |
|
34 | |||
35 |
|
35 | |||
36 | Database |
|
36 | Database | |
37 | -------- |
|
37 | -------- | |
38 |
|
38 | |||
39 | SQLite is a good option when having a small load on the system. But due to |
|
39 | SQLite is a good option when having a small load on the system. But due to | |
40 | locking issues with SQLite, it is not recommended to use it for larger |
|
40 | locking issues with SQLite, it is not recommended to use it for larger | |
41 | deployments. |
|
41 | deployments. | |
42 |
|
42 | |||
43 | Switching to PostgreSQL or MariaDB/MySQL will result in an immediate performance |
|
43 | Switching to PostgreSQL or MariaDB/MySQL will result in an immediate performance | |
44 | increase. A tool like SQLAlchemyGrate_ can be used for migrating to another |
|
44 | increase. A tool like SQLAlchemyGrate_ can be used for migrating to another | |
45 | database platform. |
|
45 | database platform. | |
46 |
|
46 | |||
47 |
|
47 | |||
48 | Horizontal scaling |
|
48 | Horizontal scaling | |
49 | ------------------ |
|
49 | ------------------ | |
50 |
|
50 | |||
51 | Scaling horizontally means running several Kallithea instances (also known as |
|
51 | Scaling horizontally means running several Kallithea instances (also known as | |
52 | worker processes) and let them share the load. That is essential to serve other |
|
52 | worker processes) and let them share the load. That is essential to serve other | |
53 | users while processing a long-running request from a user. Usually, the |
|
53 | users while processing a long-running request from a user. Usually, the | |
54 | bottleneck on a Kallithea server is not CPU but I/O speed - especially network |
|
54 | bottleneck on a Kallithea server is not CPU but I/O speed - especially network | |
55 | speed. It is thus a good idea to run multiple worker processes on one server. |
|
55 | speed. It is thus a good idea to run multiple worker processes on one server. | |
56 |
|
56 | |||
57 | .. note:: |
|
57 | .. note:: | |
58 |
|
58 | |||
59 | Kallithea and the embedded Mercurial backend are not thread-safe. Each |
|
59 | Kallithea and the embedded Mercurial backend are not thread-safe. Each | |
60 | worker process must thus be single-threaded. |
|
60 | worker process must thus be single-threaded. | |
61 |
|
61 | |||
62 | Web servers can usually launch multiple worker processes - for example ``mod_wsgi`` with the |
|
62 | Web servers can usually launch multiple worker processes - for example ``mod_wsgi`` with the | |
63 | ``WSGIDaemonProcess`` ``processes`` parameter or ``uWSGI`` or ``gunicorn`` with |
|
63 | ``WSGIDaemonProcess`` ``processes`` parameter or ``uWSGI`` or ``gunicorn`` with | |
64 | their ``workers`` setting. |
|
64 | their ``workers`` setting. | |
65 |
|
65 | |||
66 | Kallithea can also be scaled horizontally across multiple machines. |
|
66 | Kallithea can also be scaled horizontally across multiple machines. | |
67 | In order to scale horizontally on multiple machines, you need to do the |
|
67 | In order to scale horizontally on multiple machines, you need to do the | |
68 | following: |
|
68 | following: | |
69 |
|
69 | |||
70 |
|
|
70 | - Each instance's ``data`` storage needs to be configured to be stored on a | |
71 |
|
|
71 | shared disk storage, preferably together with repositories. This ``data`` | |
72 |
|
|
72 | dir contains template caches, sessions, whoosh index and is used for | |
73 |
|
|
73 | task locking (so it is safe across multiple instances). Set the | |
74 |
|
|
74 | ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` | |
75 |
|
|
75 | variables in each .ini file to a shared location across Kallithea instances | |
76 |
|
|
76 | - If using several Celery instances, | |
77 |
|
|
77 | the message broker should be common to all of them (e.g., one | |
78 |
|
|
78 | shared RabbitMQ server) | |
79 |
|
|
79 | - Load balance using round robin or IP hash, recommended is writing LB rules | |
80 |
|
|
80 | that will separate regular user traffic from automated processes like CI | |
81 |
|
|
81 | servers or build bots. | |
82 |
|
82 | |||
83 |
|
83 | |||
84 | Serve static files directly from the web server |
|
84 | Serve static files directly from the web server | |
85 | ----------------------------------------------- |
|
85 | ----------------------------------------------- | |
86 |
|
86 | |||
87 | With the default ``static_files`` ini setting, the Kallithea WSGI application |
|
87 | With the default ``static_files`` ini setting, the Kallithea WSGI application | |
88 | will take care of serving the static files from ``kallithea/public/`` at the |
|
88 | will take care of serving the static files from ``kallithea/public/`` at the | |
89 | root of the application URL. |
|
89 | root of the application URL. | |
90 |
|
90 | |||
91 | The actual serving of the static files is very fast and unlikely to be a |
|
91 | The actual serving of the static files is very fast and unlikely to be a | |
92 | problem in a Kallithea setup - the responses generated by Kallithea from |
|
92 | problem in a Kallithea setup - the responses generated by Kallithea from | |
93 | database and repository content will take significantly more time and |
|
93 | database and repository content will take significantly more time and | |
94 | resources. |
|
94 | resources. | |
95 |
|
95 | |||
96 | To serve static files from the web server, use something like this Apache config |
|
96 | To serve static files from the web server, use something like this Apache config | |
97 | snippet:: |
|
97 | snippet:: | |
98 |
|
98 | |||
99 | Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/ |
|
99 | Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/ | |
100 | Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/ |
|
100 | Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/ | |
101 | Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/ |
|
101 | Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/ | |
102 | Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/ |
|
102 | Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/ | |
103 | Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/ |
|
103 | Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/ | |
104 |
|
104 | |||
105 | Then disable serving of static files in the ``.ini`` ``app:main`` section:: |
|
105 | Then disable serving of static files in the ``.ini`` ``app:main`` section:: | |
106 |
|
106 | |||
107 | static_files = false |
|
107 | static_files = false | |
108 |
|
108 | |||
109 | If using Kallithea installed as a package, you should be able to find the files |
|
109 | If using Kallithea installed as a package, you should be able to find the files | |
110 | under ``site-packages/kallithea``, either in your Python installation or in your |
|
110 | under ``site-packages/kallithea``, either in your Python installation or in your | |
111 | virtualenv. When upgrading, make sure to update the web server configuration |
|
111 | virtualenv. When upgrading, make sure to update the web server configuration | |
112 | too if necessary. |
|
112 | too if necessary. | |
113 |
|
113 | |||
114 | It might also be possible to improve performance by configuring the web server |
|
114 | It might also be possible to improve performance by configuring the web server | |
115 | to compress responses (served from static files or generated by Kallithea) when |
|
115 | to compress responses (served from static files or generated by Kallithea) when | |
116 | serving them. That might also imply buffering of responses - that is more |
|
116 | serving them. That might also imply buffering of responses - that is more | |
117 | likely to be a problem; large responses (clones or pulls) will have to be fully |
|
117 | likely to be a problem; large responses (clones or pulls) will have to be fully | |
118 | processed and spooled to disk or memory before the client will see any |
|
118 | processed and spooled to disk or memory before the client will see any | |
119 | response. See the documentation for your web server. |
|
119 | response. See the documentation for your web server. | |
120 |
|
120 | |||
121 |
|
121 | |||
122 | .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate |
|
122 | .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate | |
123 | .. _mod_wsgi: https://modwsgi.readthedocs.io/ |
|
123 | .. _mod_wsgi: https://modwsgi.readthedocs.io/ | |
124 | .. _uWSGI: https://uwsgi-docs.readthedocs.io/ |
|
124 | .. _uWSGI: https://uwsgi-docs.readthedocs.io/ | |
125 | .. _gunicorn: http://pypi.python.org/pypi/gunicorn |
|
125 | .. _gunicorn: http://pypi.python.org/pypi/gunicorn |
General Comments 0
You need to be logged in to leave comments.
Login now