Show More
@@ -386,19 +386,20 b' weighted: Weighted Two-Bin Random' | |||||
386 | Greedy Assignment |
|
386 | Greedy Assignment | |
387 | ----------------- |
|
387 | ----------------- | |
388 |
|
388 | |||
389 |
Tasks |
|
389 | Tasks can be assigned greedily as they are submitted. If their dependencies are | |
390 | met, they will be assigned to an engine right away, and multiple tasks can be |
|
390 | met, they will be assigned to an engine right away, and multiple tasks can be | |
391 | assigned to an engine at a given time. This limit is set with the |
|
391 | assigned to an engine at a given time. This limit is set with the | |
392 |
``TaskScheduler.hwm`` (high water mark) configurable |
|
392 | ``TaskScheduler.hwm`` (high water mark) configurable in your | |
|
393 | :file:`ipcontroller_config.py` config file, with: | |||
393 |
|
394 | |||
394 | .. sourcecode:: python |
|
395 | .. sourcecode:: python | |
395 |
|
396 | |||
396 | # the most common choices are: |
|
397 | # the most common choices are: | |
397 |
c.TaskSheduler.hwm = 0 # (minimal latency, default in IPython |
|
398 | c.TaskSheduler.hwm = 0 # (minimal latency, default in IPython < 0.13) | |
398 | # or |
|
399 | # or | |
399 |
c.TaskScheduler.hwm = 1 # (most-informed balancing, default in |
|
400 | c.TaskScheduler.hwm = 1 # (most-informed balancing, default in ≥ 0.13) | |
400 |
|
401 | |||
401 |
In IPython |
|
402 | In IPython < 0.13, the default is 0, or no-limit. That is, there is no limit to the number of | |
402 | tasks that can be outstanding on a given engine. This greatly benefits the |
|
403 | tasks that can be outstanding on a given engine. This greatly benefits the | |
403 | latency of execution, because network traffic can be hidden behind computation. |
|
404 | latency of execution, because network traffic can be hidden behind computation. | |
404 | However, this means that workload is assigned without knowledge of how long |
|
405 | However, this means that workload is assigned without knowledge of how long | |
@@ -406,10 +407,10 b' each task might take, and can result in poor load-balancing, particularly for' | |||||
406 | submitting a collection of heterogeneous tasks all at once. You can limit this |
|
407 | submitting a collection of heterogeneous tasks all at once. You can limit this | |
407 | effect by setting hwm to a positive integer, 1 being maximum load-balancing (a |
|
408 | effect by setting hwm to a positive integer, 1 being maximum load-balancing (a | |
408 | task will never be waiting if there is an idle engine), and any larger number |
|
409 | task will never be waiting if there is an idle engine), and any larger number | |
409 |
being a compromise between load-balanc |
|
410 | being a compromise between load-balancing and latency-hiding. | |
410 |
|
411 | |||
411 | In practice, some users have been confused by having this optimization on by |
|
412 | In practice, some users have been confused by having this optimization on by | |
412 |
default, |
|
413 | default, so the default value has been changed to 1 in IPython 0.13. This can be slower, | |
413 | but has more obvious behavior and won't result in assigning too many tasks to |
|
414 | but has more obvious behavior and won't result in assigning too many tasks to | |
414 | some engines in heterogeneous cases. |
|
415 | some engines in heterogeneous cases. | |
415 |
|
416 |
General Comments 0
You need to be logged in to leave comments.
Login now