2025.01.26 What Are Folds in Cross-Validation in R?

Structure of the Article

Introduction
What Are Folds?
How Folds Work in Cross-Validation
Advantages of Using Folds
When to Use More or Fewer Folds
Example in R
Conclusion
References

1. Introduction

Cross-validation is a widely used technique in machine learning to evaluate model performance. A key component of cross-validation is the concept of folds, which are subsets of the dataset used for training and validation. In this blog, we’ll explore what folds are, how they work, and how to choose the right number of folds for your model.

2. What Are Folds?

Folds are non-overlapping subsets of the dataset. For example, in 5-fold cross-validation, the dataset is divided into 5 equal parts. Each fold is used once as a validation set, while the remaining folds are used for training. This ensures that every data point is used for both training and validation.

3. How Folds Work in Cross-Validation

The dataset is split into a specified number of folds (e.g., 5 or 10).
In each iteration, one fold is held out as the validation set, and the remaining folds are used for training.
This process is repeated until each fold has been used as the validation set exactly once.

Example of 5-Fold Cross-Validation:

Iteration 1: Train on Folds 2-5, validate on Fold 1.
Iteration 2: Train on Folds 1, 3-5, validate on Fold 2.
Iteration 3: Train on Folds 1-2, 4-5, validate on Fold 3.
Iteration 4: Train on Folds 1-3, 5, validate on Fold 4.
Iteration 5: Train on Folds 1-4, validate on Fold 5.

4. Advantages of Using Folds

Reduces Bias: Ensures that the model is evaluated on different subsets of the data, providing a more reliable performance estimate.
Improves Generalization: Helps the model perform well on unseen data by testing it on multiple validation sets.
Utilizes Data Efficiently: Every data point is used for both training and validation.

5. When to Use More or Fewer Folds

Use More Folds (e.g., 10):

For small datasets to ensure each fold has enough data.
When you want to reduce bias in performance estimates.

Use Fewer Folds (e.g., 5):

For large datasets where computational efficiency is a concern.
When the dataset is large enough to provide reliable estimates even with fewer folds.

6. Example in R

Here’s how to set up 5-fold cross-validation using the caret package in R:

library(caret)

# Define trainControl for 5-fold cross-validation
train_control <- trainControl(
  method = "cv",      # Standard cross-validation
  number = 5          # Number of folds
)

# Train a model using cross-validation
model <- train(
  Species ~ .,        # Formula: Predict Species using all other variables
  data = iris,        # Dataset
  method = "knn",     # Model type: K-Nearest Neighbors
  trControl = train_control   # Cross-validation configuration
)

# View the model results
print(model)

7. Conclusion

Folds are a fundamental part of cross-validation, enabling robust model evaluation by dividing the dataset into subsets for iterative training and validation. The choice of the number of folds depends on the dataset size, computational resources, and the desired balance between bias and variance in performance estimates. By understanding how folds work, you can set up cross-validation effectively and ensure your model performs well on unseen data.

8. References

Kohavi, R. (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning.
Kuhn, M., & Johnson, K. (2013). Applied Predictive Modeling.
Caret Package Documentation: https://topepo.github.io/caret/