Recently, I set out to reproduce a standard CNN baseline on CIFAR-10.

It was supposed to be a warm-up. A "hello world" for my semester project. I wrote the code, grabbed the dataset, and queued up four different optimizers: SGD, Adam, AdamW, and RMSprop.

I expected Adam to win. Everyone says Adam is robust. Everyone says Adam is default.

It didn't.

In fact, SGD crushed it. By 18%.

While waiting for the epochs to finish on my CPU, I stared at the logs wondering if I’d coded the loss function wrong. I hadn’t.

This wasn’t a coding bug. It was a hyperparameter trap. And realizing it changed how I think about every model I’ll ever build again.

The "Copy-Paste" Problem

As students, we operate under a weird constraint: Time.

We want to build cool things. We don’t want to spend three days debugging why our loss isn’t going down. So, we rely on templates. We copy config files from GitHub repositories. We use learning_rate = 0.01 because it worked in a tutorial.

I fell into this exact trap.

I assumed the optimizer was a plug-in component. You import torch.optim.Adam, pass your parameters, and go. But optimizers aren’t utilities like libraries. They’re physics engines. They behave differently depending on the environment you put them in.

The Setup (Fixed Variables)

To see what actually mattered, I locked everything down:

• Dataset: CIFAR-10

• Architecture: Standard CNN (3 Conv Layers)

• Batch Size: 128

• Epochs: 20

• Learning Rate: 0.01 (The Critical Variable)

• Seed: Fixed (for reproducibility)

I ran them side-by-side. Here is what happened.

The Shocking Results

The Winner: SGD.
The Loser: RMSprop.
The Spread: 18.42% accuracy difference.

This number is huge. In research papers, squeezing out 0.5% is considered an improvement. I saw nearly 20% swing because I didn’t tweak one float.

But look at the Generalization Gap.

Val Acc > Train Acc?

My SGD model performed better on the validation set than the training set. This usually signals overfitting issues, but here it meant SGD’s implicit regularization forced the model to learn features rather than memorizing noise.

Adam? It underfit both train and val. It got stuck.

The Root Cause: Hyperparameter Coupling

Here’s the part that clicked for me.

Optimizers and Learning Rates are coupled.

You cannot change one without changing how the other behaves.

• SGD thrives at lr = 0.01 on CIFAR-10. It takes stable steps.

• Adam adapts the step size based on gradient variance. If you feed it lr = 0.01, its internal adaptive mechanism amplifies that value. The effective step becomes too massive. It overshoots the minimum like a car drifting on ice.

I treated the Learning Rate as a global constant. That was my design flaw.

The Lesson: An optimizer isn't a setting. It's a contract. And I signed a contract with SGD using terms only applicable to SGD.

Building a "Student-Proof" Pipeline

I’m not going to build the next AutoML startup tomorrow. But I can improve my own workflow for the rest of my degree.

Here is how I’m designing my personal development environment to prevent this mistake.

1. The "Sanity Check" Epoch

Before running 100 epochs, I run 5.

• Run SGD, Adam, AdamW at their native defaults (SGD @ 0.01, Adam @ 0.001).

• Look at the Loss curve immediately.

• If Adam’s loss oscillates, kill it. Move on.

2. The Config Constraint

I’m writing a wrapper around my training config (YAML/JSON).

• If optimizer == 'adam', the search space for lr is constrained to [1e-4, 1e-3].

• If optimizer == 'sgd', the search space is [1e-3, 1e-2].

• Goal: Prevents accidental mismatch. Makes the error impossible.

3. Track the Gap, Not Just Loss

I used to look at Validation Loss. Now I track Train Acc - Val Acc.

• High Positive Gap: Overfitting (Need Dropout/Augmentation).

• Negative Gap: Underfitting / High Regularization (Maybe increase capacity?).

• Zero/Near Zero: Balanced.

Why This Actually Matters

We spend so much time reading papers on new architectures (ResNet, ViT, Transformers) that we forget the basics.

Optimization is the foundation.

If your optimization strategy is flawed, your architecture is invisible. You could build the most beautiful Neural Network in the world, but if the gradients explode because you set lr=0.1 on Adam, you’re wasting electricity.

For us as students, mastering this is more valuable than chasing state-of-the-art SOTA. Because in an internship, no one will ask "What was your FLOP count?" Someone will ask "Did you verify your hyperparameters?"

Real Talk: The Limitations

It’s easy to write this off as "just use SGD." But there’s nuance.

1. Transformers Love Adam. On LLMs, SGD diverges often. This experiment was on a CNN/Vision task. Context matters.

2. Schedules Matter. I used a fixed LR. In production, you’d use Cosine Annealing. That changes the game again.

3. Compute Budget. Running 4 optimizers x 3 LR schedules = 12 runs. For a GPU-heavy lab? Cheap. For a laptop student? Painful. There’s always a trade-off between rigor and resources.

The Bigger Picture

We treat Machine Learning like magic. We throw data in, pull weights out, and expect wisdom.

But it’s engineering. It’s mechanical. It relies on tension, friction, and precise adjustments.

By obsessing over the details—the learning rate, the gap, the optimizer—I stopped treating my models like black boxes. I started seeing them as systems I control.

And that shift in mindset? That’s worth more than any single accuracy point.

Question for you:

Have you ever seen a model fail simply because you copied a learning rate from a GitHub repo? How did you fix it?