Show More
@@ -1,99 +1,112 b'' | |||||
1 | .. _performance: |
|
1 | .. _performance: | |
2 |
|
2 | |||
3 | ================================ |
|
3 | ================================ | |
4 | Optimizing Kallithea performance |
|
4 | Optimizing Kallithea performance | |
5 | ================================ |
|
5 | ================================ | |
6 |
|
6 | |||
7 | When serving a large amount of big repositories, Kallithea can start |
|
7 | When serving a large amount of big repositories, Kallithea can start performing | |
8 |
|
|
8 | slower than expected. Because of the demanding nature of handling large amounts | |
9 |
|
|
9 | of data from version control systems, here are some tips on how to get the best | |
10 |
|
|
10 | performance. | |
|
11 | ||||
11 |
|
12 | |||
12 | Follow these few steps to improve performance of Kallithea system. |
|
13 | Fast storage | |
|
14 | ------------ | |||
13 |
|
15 | |||
14 |
|
|
16 | Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM | |
15 |
|
|
17 | is usually more important than a fast CPU. | |
|
18 | ||||
16 |
|
19 | |||
17 | 2. Increase cache |
|
20 | Caching | |
|
21 | ------- | |||
18 |
|
22 | |||
19 |
|
|
23 | Tweak beaker cache settings in the ini file. The actual effect of that is | |
20 |
|
|
24 | questionable. | |
|
25 | ||||
21 |
|
26 | |||
22 | 3. Switch from SQLite to PostgreSQL or MySQL |
|
27 | Database | |
|
28 | -------- | |||
23 |
|
29 | |||
24 |
|
|
30 | SQLite is a good option when having a small load on the system. But due to | |
25 |
|
|
31 | locking issues with SQLite, it is not recommended to use it for larger | |
26 | deployments. Switching to MySQL or PostgreSQL will result in an immediate |
|
32 | deployments. | |
27 | performance increase. A tool like SQLAlchemyGrate_ can be used for |
|
|||
28 | migrating to another database platform. |
|
|||
29 |
|
33 | |||
30 | 4. Scale Kallithea horizontally |
|
34 | Switching to MySQL or PostgreSQL will result in an immediate performance | |
|
35 | increase. A tool like SQLAlchemyGrate_ can be used for migrating to another | |||
|
36 | database platform. | |||
|
37 | ||||
31 |
|
38 | |||
32 | Scaling horizontally can give huge performance benefits when dealing with |
|
39 | Horizontal scaling | |
33 | large amounts of traffic (many users, CI servers, etc.). Kallithea can be |
|
40 | ------------------ | |
34 | scaled horizontally on one (recommended) or multiple machines. |
|
|||
35 |
|
41 | |||
36 | It is generally possible to run WSGI applications multithreaded, so that |
|
42 | Scaling horizontally means running several Kallithea instances and let them | |
37 | several HTTP requests are served from the same Python process at once. That |
|
43 | share the load. That can give huge performance benefits when dealing with large | |
38 | can in principle give better utilization of internal caches and less |
|
44 | amounts of traffic (many users, CI servers, etc.). Kallithea can be scaled | |
39 | process overhead. |
|
45 | horizontally on one (recommended) or multiple machines. | |
40 |
|
46 | |||
41 | One danger of running multithreaded is that program execution becomes much |
|
47 | It is generally possible to run WSGI applications multithreaded, so that | |
42 | more complex; programs must be written to consider all combinations of |
|
48 | several HTTP requests are served from the same Python process at once. That can | |
43 | events and problems might depend on timing and be impossible to reproduce. |
|
49 | in principle give better utilization of internal caches and less process | |
|
50 | overhead. | |||
|
51 | ||||
|
52 | One danger of running multithreaded is that program execution becomes much more | |||
|
53 | complex; programs must be written to consider all combinations of events and | |||
|
54 | problems might depend on timing and be impossible to reproduce. | |||
44 |
|
55 | |||
45 |
|
|
56 | Kallithea can't promise to be thread-safe, just like the embedded Mercurial | |
46 |
|
|
57 | backend doesn't make any strong promises when used as Kallithea uses it. | |
47 |
|
|
58 | Instead, we recommend scaling by using multiple server processes. | |
48 |
|
59 | |||
49 |
|
|
60 | Web servers with multiple worker processes (such as ``mod_wsgi`` with the | |
50 |
|
|
61 | ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box. | |
51 |
|
62 | |||
52 |
|
|
63 | In order to scale horizontally on multiple machines, you need to do the | |
53 |
|
|
64 | following: | |
54 |
|
65 | |||
55 | - Each instance's ``data`` storage needs to be configured to be stored on a |
|
66 | - Each instance's ``data`` storage needs to be configured to be stored on a | |
56 | shared disk storage, preferably together with repositories. This ``data`` |
|
67 | shared disk storage, preferably together with repositories. This ``data`` | |
57 | dir contains template caches, sessions, whoosh index and is used for |
|
68 | dir contains template caches, sessions, whoosh index and is used for | |
58 | task locking (so it is safe across multiple instances). Set the |
|
69 | task locking (so it is safe across multiple instances). Set the | |
59 | ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` |
|
70 | ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` | |
60 | variables in each .ini file to a shared location across Kallithea instances |
|
71 | variables in each .ini file to a shared location across Kallithea instances | |
61 | - If using several Celery instances, |
|
72 | - If using several Celery instances, | |
62 | the message broker should be common to all of them (e.g., one |
|
73 | the message broker should be common to all of them (e.g., one | |
63 | shared RabbitMQ server) |
|
74 | shared RabbitMQ server) | |
64 | - Load balance using round robin or IP hash, recommended is writing LB rules |
|
75 | - Load balance using round robin or IP hash, recommended is writing LB rules | |
65 | that will separate regular user traffic from automated processes like CI |
|
76 | that will separate regular user traffic from automated processes like CI | |
66 | servers or build bots. |
|
77 | servers or build bots. | |
67 |
|
78 | |||
68 | 5. Serve static files directly from the web server |
|
79 | ||
|
80 | Serve static files directly from the web server | |||
|
81 | ----------------------------------------------- | |||
69 |
|
82 | |||
70 | With the default ``static_files`` ini setting, the Kallithea WSGI application |
|
83 | With the default ``static_files`` ini setting, the Kallithea WSGI application | |
71 | will take care of serving the static files found in ``kallithea/public`` from |
|
84 | will take care of serving the static files found in ``kallithea/public`` from | |
72 | the root of the application URL. While doing that, it will currently also |
|
85 | the root of the application URL. While doing that, it will currently also | |
73 | apply buffering and compression of all the responses it is serving. |
|
86 | apply buffering and compression of all the responses it is serving. | |
74 |
|
87 | |||
75 | The actual serving of the static files is unlikely to be a problem in a |
|
88 | The actual serving of the static files is unlikely to be a problem in a | |
76 | Kallithea setup. The buffering of responses is more likely to be a problem; |
|
89 | Kallithea setup. The buffering of responses is more likely to be a problem; | |
77 | large responses (clones or pulls) will have to be fully processed and spooled |
|
90 | large responses (clones or pulls) will have to be fully processed and spooled | |
78 | to disk or memory before the client will see any response. |
|
91 | to disk or memory before the client will see any response. | |
79 |
|
92 | |||
80 | To serve static files from the web server, use something like this Apache config |
|
93 | To serve static files from the web server, use something like this Apache config | |
81 | snippet:: |
|
94 | snippet:: | |
82 |
|
95 | |||
83 | Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/ |
|
96 | Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/ | |
84 | Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/ |
|
97 | Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/ | |
85 | Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/ |
|
98 | Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/ | |
86 | Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/ |
|
99 | Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/ | |
87 | Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/ |
|
100 | Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/ | |
88 |
|
101 | |||
89 | Then disable serving of static files in the ``.ini`` ``app:main`` section:: |
|
102 | Then disable serving of static files in the ``.ini`` ``app:main`` section:: | |
90 |
|
103 | |||
91 | static_files = false |
|
104 | static_files = false | |
92 |
|
105 | |||
93 | If using Kallithea installed as a package, you should be able to find the files |
|
106 | If using Kallithea installed as a package, you should be able to find the files | |
94 | under site-packages/kallithea, either in your Python installation or in your |
|
107 | under site-packages/kallithea, either in your Python installation or in your | |
95 | virtualenv. When upgrading, make sure to update the web server configuration |
|
108 | virtualenv. When upgrading, make sure to update the web server configuration | |
96 | too if necessary. |
|
109 | too if necessary. | |
97 |
|
110 | |||
98 |
|
111 | |||
99 | .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate |
|
112 | .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate |
General Comments 0
You need to be logged in to leave comments.
Login now