Automated Input Data Distribution In a Multinode Workflow

Automated Input Data Distribution In a Multinode Workflow

Authors: Sandesh Balakrishna | Mohsin Kaleem

Designing a ML workflow at scale requires multiple considerations — important being memory, Time To Run (TTR), Cost To Run (CTR). Workflows must be designed to maintain a balance of these metrics. There may be use cases where a single node workflow might not be sufficient either in memory or run time SLA (Service Level Agreements) adherence given that each node of a workflow will have …

automated balance cases cost ctr data distribution important input may memory metrics run scale single use cases workflow workflows