Automatically deploy Dask clusters for your projects with a single line of code.
WARNING: This is very experimental and currently just a proof of concept. Proceed at your own risk.
Dask clusters can be created using the
from cowait.tasks import Task from cowait.tasks.dask import DaskCluster class YourDaskJob(Task): async def run(self, dask, inputs**): cluster = DaskCluster(workers=5) client = await cluster.get_client() # dask client ready to use! def square(x): return x ** 2 def neg(x): return -x A = client.map(square, range(10)) B = client.map(neg, A) total = client.submit(sum, B) result = total.result() print(result) return result
cowait run dask_cluster
The DaskCluster task will automatically set up a Dask scheduler and a set of workers. It provides a number of RPC methods for controlling your cluster, summarized below:
||Get informations about all Dask workers|
||Can be used to scale up or down your cluster|
||Returns the Dask scheduler URI|
||Returns the dask client|
||Stop your Dask cluster task from running|
See github for full reference.
WARNING: This is very experimental and currently just a proof of concept.