All writing
AmesStackingEvaluation
Ames ensemble wins and failures
1 min read
The easy win
The biggest gain came from the log transform and from fitting preprocessing inside the fold. CatBoost handled some categorical cases well, but the stack only helped once out of fold predictions were clean.
The failure
I initially removed outliers on the full dataset before cross validation. The scores looked better and made no sense. Moving the rule inside a pipeline dropped the CV score to a more realistic number. The stack had to beat Ridge again, and it only did so by a small margin.
The table
| Model | RMSE, log target | Notes |
|---|---|---|
| Mean | add | Baseline |
| Ridge | add | Linear |
| Best single | add model and score | Check this first |
| Stack | add | Out of fold only |
Lesson
If a stacking step requires a validation set to report the result, you have already leaked something. Use out of fold predictions, or use nested cross validation.