Hugging Face researchers predict fine-tuning time with 0.98 R² using CatBoost on 272 production runs
A CatBoost regressor trained on pre-dispatch job configuration data predicts end-to-end fine-tuning duration with 98% variance explained, reducing mean absolute error from over 12 hours to 1.71 hours compared to standard FLOP estimates.
Summary
Researchers from Dharma-AI and Hugging Face analyzed 272 real supervised fine-tuning jobs running on NVIDIA H200 GPUs to build a predictor for end-to-end fine-tuning wall-clock time. The conventional FLOP-based estimate, which divides theoretical compute by peak GPU throughput and an assumed Model FLOPs Utilization factor, achieved only R² = 0.31 with a mean absolute error of 12.41 hours on jobs averaging 23.4 hours. In contrast, a CatBoost regression model using only information available before job dispatch—hyperparameters, dataset sizes, model identifiers, GPU allocation, and derived features like evaluation token counts and checkpoint I/O volumes—reached R² = 0.98 with MAE of 1.71 hours and median error of 0.51 hours in stratified 5-fold cross-validation.
The work highlights that fine-tuning duration is dominated by non-compute overheads that do not scale linearly with workload: environment setup, model loading, periodic evaluation, checkpoint serialization, and final model upload to object storage, all competing for network and storage I/O in multi-tenant clusters. The model treats the FLOP estimate itself as one input feature and learns corrections from historical runs, making it usable for cluster scheduling without requiring runtime telemetry. The study covers models up to 15 billion parameters with both LoRA and full fine-tuning, using Hugging Face Transformers and Unsloth as training stacks.
What the predictors capture that FLOP math misses
- Pre-dispatch feature engineering: The CatBoost model uses 24 features derived entirely from job submission configuration, including
num_eval_data(total evaluation samples across all cycles),num_savesandnum_params_saved(checkpoint I/O volume), per-GPU token counts, and a categorical stack identifier for Hugging Face Transformers versus Unsloth, ensuring predictions are available at scheduling time without runtime logs. - FLOP baseline as learned correction: The analytical estimate with fixed MFU=0.45 is fed into the model as
baseline_time, allowing the regressor to start from the conventional compute-only number and learn residual corrections rather than replacing it entirely. - Nonlinear interaction modeling: XGBoost and CatBoost significantly outperformed linear methods (Ridge R² = 0.73, Linear Regression R² = 0.69), indicating that interactions between configuration parameters—such as checkpoint frequency combined with model size and GPU count—are critical and not captured by additive models.
- Multi-tenant contention effects: The 272 runs span 1 to 4 GPUs across 1 to 4 nodes in a shared pool of 8–32 H200s, with measured durations including real network and storage contention from co-located training and inference workloads, making the predictor applicable to operational rather than laboratory conditions.
- Analytical baseline failure: Even sweeping MFU from 0.25 to 0.65, the best FLOP baseline (30% MFU) still produced MAE above 12 hours; the more detailed Calculon analytical model performed worse with R² = -0.51, confirming that closed-form compute estimates cannot approximate wall-clock time without empirical learning.
Why it matters
For Linux and open-source AI infrastructure operators, this work provides a reproducible methodology to reduce GPU cluster idle time and queue latency without proprietary scheduling software. The approach is stack-agnostic across open training frameworks and could be integrated into open-source job schedulers like Kubernetes with Volcano or Slurm to improve utilization of scarce NVIDIA H100/H200 capacity. Independent researchers and small labs running shared GPU pools can adopt the same CatBoost-based pipeline using their own historical job logs, avoiding vendor-locked predictive services from cloud providers.
Original reference: huggingface.co →