The procedure for k-fold cross-validation

Every sample is used for testing exactly once and for training k−1 times. The score you report is the average over the k held-out folds.

The rotation

Each row is one round of training. Click a row, or press Play, to follow one round.

k = 5

Training folds Held-out test fold

Result

Mean of the k test scores, reported with its spread

Shuffle once, cut the shuffled samples into k equal slices, then rotate which slice is held out. Scores shown are simulated AUCs for illustration.

What happens inside one round

1. SplitFold i is set aside. Nothing about it is looked at yet.

2. Fit preprocessingScaling, imputation, feature selection and PCs are learned from the training folds only.

3. TrainThe model is fit on the transformed training folds.

4. ScoreThe fitted transforms and model are applied to fold i, and its metric is recorded.

Nested cross-validation, for tuning and evaluating

When you also choose a hyperparameter (for PRS, the p-value threshold P_T), the choice must happen inside the outer training set. The inner loop picks P_T; the outer fold only grades that choice.

Inner training Inner validation (used to pick P_T) Outer test fold, untouched until the end

Expand outer round:

Leakage: the same steps in the wrong order

Leaky

Select features or P_T on all samples

Split into k folds

Train and score each fold

Score is optimistic: the test folds already shaped the model

Correct

Split into k folds

Select features or P_T inside the training folds

Train and score on the held-out fold

Score estimates performance on new samples

In genetic data, also keep relatives in the same fold (grouped k-fold), and stratify by case/control status so each fold has a similar ratio.

Created with the assistance of Claude and reviewed by the author. The AUC values shown are simulated and for illustration only.