Training Neural Networks from Scratch

1. Training Is the Hard Part

Designing a neural network is easy. Training it well is hard.

Most failures in deep learning come from:

• Poor initialization
• Wrong learning rate
• Bad loss choice
• Overfitting / underfitting

This article breaks training down into concrete, controllable components.

2. The Training Loop (Core of Everything)

Every neural network training process reduces to:

Understanding each step deeply is more important than knowing any framework.

3. Weight Initialization

Why Initialization Matters

Initialization controls:

• Signal propagation
• Gradient flow
• Speed of convergence

Bad initialization causes:

• Vanishing gradients
• Exploding gradients

Modern Initialization Schemes

Xavier (Glorot) – for tanh / sigmoid:

He Initialization – for ReLU:

Goal: keep activation variance stable across layers.

4. Loss Functions: What Are You Really Optimizing?

Regression Losses

• Mean Squared Error (MSE)
• Mean Absolute Error (MAE)

Classification Losses

• Cross Entropy
• Binary Cross Entropy

Choosing the wrong loss = optimizing the wrong objective.

5. Optimizers: How Parameters Move

Gradient Descent Variants

SGD

• Simple
• Noisy updates
• Better generalization

Momentum

• Faster convergence
• Escapes shallow minima

Adam

• Adaptive learning rates
• Fast convergence
• Risk of overfitting

Rule of thumb:

• Start with Adam
• Fine-tune with SGD

6. Learning Rate: The Most Important Hyperparameter

Too small → slow training Too large → divergence

Common Strategies

• Step decay
• Exponential decay
• Cosine annealing
• Warm-up

A good learning rate schedule often matters more than model size.

7. Batch Size Trade-offs

Large batches often need learning rate scaling.

8. Regularization: Controlling Overfitting

L2 Regularization

Penalizes large weights:

Dropout

• Randomly disables neurons
• Forces redundancy

Data Augmentation

• Most powerful regularizer
• Increases effective dataset size

9. Normalization Techniques

Batch Normalization

• Stabilizes training
• Allows higher learning rates
• Reduces sensitivity to initialization

Layer Normalization

• Better for RNNs & Transformers

Normalization makes deep networks trainable.

10. Vanishing & Exploding Gradients

Causes

• Deep networks
• Saturating activations
• Poor initialization

Solutions

• ReLU / GELU
• Proper initialization
• Normalization
• Residual connections

11. Overfitting vs Underfitting

Overfitting

• Low training loss
• High validation loss

Underfitting

• High training loss
• Model too simple

Fixing requires:

• More data
• Better regularization
• Model capacity tuning

12. Debugging Training Failures

Checklist:

• Loss decreasing?
• Gradients exploding?
• Outputs saturated?
• Data normalized?

Always overfit a small batch first.

13. Training From Scratch: Mental Model

Training is:

Navigating a noisy, high-dimensional surface using local gradient information

Success depends more on stability than cleverness.

14. What Comes Next?

Next article explores deep architectures:

• CNNs
• RNNs
• Transformers
• Why architecture matters

➡ Article 4: Deep Neural Network Architectures