LLMs-from-scratch

The official repository for the book 'Build a Large Language Model (From Scratch)' by Sebastian Raschka. It provides a comprehensive, step-by-step guide to developing, pretraining, and finetuning a GPT-like LLM from the ground up. By using pure PyTorch instead of high-level LLM libraries, it explains the inner workings of components like attention mechanisms and transformer blocks. The project includes end-to-end pipelines for pretraining on unlabeled data and finetuning for instruction-following, designed to be accessible on standard consumer hardware.

• From-scratch GPT architecture implementation
• Pure PyTorch implementation with zero LLM library dependencies
• End-to-end pretraining and finetuning pipelines
• Parameter-efficient finetuning with LoRA
• Hardware-friendly educational code structure