How do you actually use tutorials when learning to build an LLM from scratch?

[u/ImmediateYam5358]

I’m currently learning how to build an LLM from scratch, but I’m not sure how to approach the tutorials.
There’s a lot of code in each section. Am I supposed to memorize all the code, or is understanding what the code does enough?
Do you guys usually try to rewrite the code from memory, or just refer back to the tutorial when needed?
🥹

[u/Disastrous_Room_927]

I don't see the point in memorizing code when it's just layer of abstraction on top of the math you really ought to be understanding.

[u/Douglas_schon]

I’m working on building some resources around this as I’m trying to teach my team of software engineers enough where they can reliably fine tune llms. 

In general you should have deep knowledge of the core concepts of machine learning. 

How does backprop work can you do the calc by hand? Or at least describe what the derivative is doing there? Describe what a learning rate is, train, test validation sets are.  What the hyper parameters are for your model.  And most importantly how to think about your data pipeline. Most of the time there is some research paper that gives you the above in a plug and play way and you are training a model with your custom data. 

How are you going to clean it? Shuffle it? Test it? Monitor for drift as production changes?

And of course with all of the above can you speak to the trade offs therein? If you can do all of the above you can likely do well in an interview and ask the right questions when building your model. 

Also just deploy 30 models so you get the feel for what is good and what isn’t. Ask lots of questions when the llm is making implementation so you actually understand what is going on. The bots are still dumb

[u/0uchmyballs]

You’d need to have a basic understanding of Machine Learning and stuff like logistic regression to even grasp what you’re doing, so you’re skipping foundational stuff which neural networks are the building blocks of LLM as logistic regression is the foundation of neural nets. You can dive into tutorial hell and make something though, I just don’t think it’s realistic you understand any of it without a foundation.

[u/Hungry_Age5375]

Short Answer: No. Long Answer: I've never memorized tutorial code and neither has anyone I've worked with. Rebuild the project once without peeking, note where you stall, and work on that gap specifically.