Show More
@@ -32,19 +32,15 IPython is implemented using a distributed set of processes that communicate usi | |||||
32 |
|
32 | |||
33 | We need to build a system that makes it trivial for users to start and manage IPython processes. This system should have the following properties: |
|
33 | We need to build a system that makes it trivial for users to start and manage IPython processes. This system should have the following properties: | |
34 |
|
34 | |||
35 |
* It should possible to do everything through an extremely simple API that |
|
35 | * It should possible to do everything through an extremely simple API that | |
36 |
can call from their own Python script. |
|
36 | users can call from their own Python script. No shell commands should be | |
37 |
|
37 | needed. | ||
38 | * This simple API should be configured using standard .ini files. |
|
38 | * This simple API should be configured using standard .ini files. | |
39 |
|
39 | * The system should make it possible to start processes using a number of | ||
40 | * The system should make it possible to start processes using a number of different |
|
40 | different approaches: SSH, PBS/Torque, Xgrid, Windows Server, mpirun, etc. | |
41 | approaches: SSH, PBS/Torque, Xgrid, Windows Server, mpirun, etc. |
|
41 | * The controller and engine processes should each have a daemon for | |
42 |
|
42 | monitoring, signaling and clean up. | ||
43 | * The controller and engine processes should each have a daemon for monitoring, |
|
|||
44 | signaling and clean up. |
|
|||
45 |
|
||||
46 | * The system should be secure. |
|
43 | * The system should be secure. | |
47 |
|
||||
48 | * The system should work under all the major operating systems, including |
|
44 | * The system should work under all the major operating systems, including | |
49 | Windows. |
|
45 | Windows. | |
50 |
|
46 | |||
@@ -63,12 +59,10 Security | |||||
63 | Currently, IPython has no built in security or security model. Because we would like IPython to be usable on public computer systems and over wide area networks, we need to come up with a robust solution for security. Here are some of the specific things that need to be included: |
|
59 | Currently, IPython has no built in security or security model. Because we would like IPython to be usable on public computer systems and over wide area networks, we need to come up with a robust solution for security. Here are some of the specific things that need to be included: | |
64 |
|
60 | |||
65 | * User authentication between all processes (engines, controller and clients). |
|
61 | * User authentication between all processes (engines, controller and clients). | |
66 |
|
||||
67 | * Optional TSL/SSL based encryption of all communication channels. |
|
62 | * Optional TSL/SSL based encryption of all communication channels. | |
68 |
|
||||
69 | * A good way of picking network ports so multiple users on the same system can |
|
63 | * A good way of picking network ports so multiple users on the same system can | |
70 |
run their own controller and engines without interfering with those of |
|
64 | run their own controller and engines without interfering with those of | |
71 |
|
65 | others. | ||
72 | * A clear model for security that enables users to evaluate the security risks |
|
66 | * A clear model for security that enables users to evaluate the security risks | |
73 | associated with using IPython in various manners. |
|
67 | associated with using IPython in various manners. | |
74 |
|
68 | |||
@@ -86,22 +80,25 Latent performance issues | |||||
86 |
|
80 | |||
87 | Currently, we have a number of performance issues that are waiting to bite users: |
|
81 | Currently, we have a number of performance issues that are waiting to bite users: | |
88 |
|
82 | |||
89 |
|
|
83 | * The controller store a large amount of state in Python dictionaries. Under | |
90 |
|
|
84 | heavy usage, these dicts with get very large, causing memory usage problems. | |
91 |
|
|
85 | We need to develop more scalable solutions to this problem, such as using a | |
92 |
|
|
86 | sqlite database to store this state. This will also help the controller to | |
93 | * Currently, the client to controller connections are done through XML-RPC using |
|
87 | be more fault tolerant. | |
94 | HTTP 1.0. This is very inefficient as XML-RPC is a very verbose protocol and |
|
88 | * Currently, the client to controller connections are done through XML-RPC | |
95 | each request must be handled with a new connection. We need to move these network |
|
89 | using HTTP 1.0. This is very inefficient as XML-RPC is a very verbose | |
96 | connections over to PB or Foolscap. Done! |
|
90 | protocol and each request must be handled with a new connection. We need to | |
97 | * We currently don't have a good way of handling large objects in the controller. |
|
91 | move these network connections over to PB or Foolscap. Done! | |
98 | The biggest problem is that because we don't have any way of streaming objects, |
|
92 | * We currently don't have a good way of handling large objects in the | |
99 | we get lots of temporary copies in the low-level buffers. We need to implement |
|
93 | controller. The biggest problem is that because we don't have any way of | |
100 | a better serialization approach and true streaming support. |
|
94 | streaming objects, we get lots of temporary copies in the low-level buffers. | |
101 | * The controller currently unpickles and repickles objects. We need to use the |
|
95 | We need to implement a better serialization approach and true streaming | |
102 | [push|pull]_serialized methods instead. |
|
96 | support. | |
103 | * Currently the controller is a bottleneck. We need the ability to scale the |
|
97 | * The controller currently unpickles and repickles objects. We need to use the | |
104 | controller by aggregating multiple controllers into one effective controller. |
|
98 | [push|pull]_serialized methods instead. | |
|
99 | * Currently the controller is a bottleneck. We need the ability to scale the | |||
|
100 | controller by aggregating multiple controllers into one effective | |||
|
101 | controller. | |||
105 |
|
102 | |||
106 |
|
103 | |||
107 |
|
104 |
General Comments 0
You need to be logged in to leave comments.
Login now