Using Dask on infiniband

InfiniBand (abbreviated IB) is a computer-networking communications standard used in high-performance computing that features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also used as either a direct or switched interconnect between servers and storage systems, as well as an interconnect between storage systems. (source Wikipedia).

If you want to leverage this high speed network instead of the regular ethernet network, you have to specify to the scheduler that you want to used infiniband as your interface. Assuming that you Infiniband interface is ib0, you would call the scheduler like this :

dask-scheduler --interface ib0 --scheduler-file ./cluster.yaml

you would have to call the worker using the same interface :

dask-worker --interface ib0 --scheduler-file ./cluster.yaml