Performance Regression or Improvement: test_cloudml_benchmark_cirteo_no_shuffle_10GB-runtime_sec:runtime_sec

[github-actions]

Performance change found in the
test: test_cloudml_benchmark_cirteo_no_shuffle_10GB-runtime_sec for the metric: runtime_sec.

For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.

Test description: TFT Criteo test on 10 GB data with no Reshuffle.
Test link - Test link

timestamp: Thu Jan  8 22:06:45 2026, metric_value: 2471.94
timestamp: Wed Jan  7 22:06:41 2026, metric_value: 2422.71
timestamp: Tue Jan  6 22:07:11 2026, metric_value: 2399.02 <---- Anomaly
timestamp: Mon Jan  5 22:24:30 2026, metric_value: 3510.01
timestamp: Sun Jan  4 22:22:01 2026, metric_value: 3490.45
timestamp: Sat Jan  3 22:16:53 2026, metric_value: 3015.96
timestamp: Fri Jan  2 22:11:10 2026, metric_value: 2861.66
timestamp: Thu Jan  1 22:13:24 2026, metric_value: 2962.63
timestamp: Wed Dec 31 22:17:17 2025, metric_value: 3027.79
timestamp: Tue Dec 30 22:17:11 2025, metric_value: 3069.19
timestamp: Mon Dec 29 22:14:46 2025, metric_value: 2970.57
timestamp: Sun Dec 28 22:13:26 2025, metric_value: 2860.47
timestamp: Sat Dec 27 22:10:24 2025, metric_value: 2829.45

[renu-123-pixel]

Thanks for the alert.

Looking at the recent history of test_cloudml_benchmark_cirteo_no_shuffle_10GB-runtime_sec, there is a noticeable drop in runtime starting on Jan 6 (2399s) compared to the previous baseline (~2800–3000s), following a short period of higher runtimes on Jan 4–5 (~3500s).

This appears to be a performance improvement, but the magnitude and sudden change suggest it could also be influenced by environmental factors (e.g., runner configuration, caching effects, or infrastructure differences) rather than a confirmed code-level optimization.

Next steps:

Check for any recent code changes affecting the CloudML / TFT pipeline around Jan 5–6.

If no relevant changes are found, consider re-running the benchmark to validate repeatability.

If the improvement is consistent across runs, we can mark this as an expected performance gain; otherwise, treat it as a potential flaky measurement.

No immediate action required until repeatability or a causal change is identified.