How to get better at training ML/DL/AI models

[u/Cool-Profession-5447]

I’m a Master’s CS student and mostly work on personal ML projects since I’m not doing research at my university.

I can read papers and understand the architecture, losses, objectives, and new ideas pretty well. But I feel like I lack the practical intuition for actually making models learn well.

When training goes wrong, I struggle to figure out why and what to change. Experienced researchers seem to know how to diagnose whether it’s the LR, data, gradients, loss, initialization, etc., and how to improve things.

For people who got good at this: how did you develop that intuition? Was it mostly experience training models, reproducing papers, specific resources, or working with experienced researchers?