Bias Variance Decomposition

Bias-Variance decomposition is a fundamental concept in machine learning that helps us understand the sources of error in a model. It breaks down the expected prediction error into bias, variance, and irreducible error.

1. Error Components in Machine Learning

The total error of a model can be expressed as:

Explanation of Terms Used in the Equation :

2. Understanding Bias

• Bias refers to the difference between the expected prediction and the true function.

• A high bias model makes strong assumptions about the data and is usually underfitting.

• Example: A linear model trying to fit a non-linear dataset will have high bias.

Mathematically:

If Bias is high → The model is too simple and fails to capture patterns.

3. Understanding Variance

• Variance refers to the variability of model predictions for different training datasets.

• A high variance model is highly sensitive to training data and is usually overfitting.

• Example: A complex deep neural network trained on small data might memorize training examples but fail on unseen data.

Mathematically:

• If Variance is high → The model is too complex and fails to generalize well.

  
  

  4. Bias-Variance Trade-off

• Goal: Find a balance between bias and variance to minimize total error.

• Low Bias & Low Variance → Best case (ideal model).

• High Bias & Low Variance → Underfitting (oversimplified model).

• Low Bias & High Variance → Overfitting (too complex model).

• High Bias & High Variance → Worst case (random guessing).

Model Complexity	Bias	Variance	Total Error
Low Complexity (Underfitting)	High	Low	High
Optimal Complexity	Medium	Medium	Low
High Complexity (Overfitting)	Low	High	High

5. Strategies to Handle Bias & Variance

To Reduce Bias (Underfitting)

✔ Increase model complexity (e.g., use a non-linear model instead of a linear one).

✔ Train for a longer time (increase epochs).

✔ Add more relevant features.

To Reduce Variance (Overfitting)

✔ Collect more training data.

✔ Use regularization techniques (L1, L2, Dropout).

✔ Use simpler models (reduce complexity).

✔ Use ensemble methods (Bagging, Boosting).

6. MATHEMATICAL FORMULATION

7. CONCLUSION

   • Bias-variance decomposition helps us understand why a model is not performing well.

   • We need to balance bias and variance to achieve the best generalization.

   • Choosing the right model complexity and tuning hyperparameters helps in controlling bias and variance.

   Key Takeaway: The ultimate goal is to reduce total error, not just bias or variance individually!