CAPS: Cost-Aware ML Pipeline Selection

Download PDF

Abstract

We present CAPS, a novel system for cost-aware pipeline selection in automated machine learning (AutoML). CAPS' approach is orthogonal to the search strategies used by existing AutoML frameworks, enabling seamless integration with them. This integration benefits AutoML systems by reducing computational waste—time spent evaluating inefficient pipelines or exceeding time or memory constraints—through lightweight performance and cost estimation. To incorporate cost-awareness into AutoML, CAPS represents a set of pipelines as a directed hypergraph, estimates execution times for each function, and formulates the optimal subset selection as a constrained prize-collecting optimization problem. CAPS employs a simple yet effective greedy algorithm to approximate this optimal subset. Our evaluation shows that when CAPS prioritizes cost, it reduces waste up to 4×—saving up to 23 hours in 50 hours of pipeline evaluation. When CAPS balances the trade-off between performance and cost, it consistently achieves performance improvements of 5% to 15% over the state of the art within the same time budget across datasets. It also demonstrates that CAPS is compatible with complementary efficiency techniques such as early stopping and warm-starting, achieving speedups of up to 4.7×.