CS336: Language Modeling from Scratch by Stanford Language models… — Artem Ryblov’s Data Science Weekly — TG.ME

CS336: Language Modeling from Scratch by Stanford Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks. As the field of artificial intelligence (AI), machine learning (ML), and NLP continues to grow, possessing a deep understanding of language models becomes essential for scientists and engineers alike. This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own. Drawing inspiration from operating systems courses that create an entire operating system from scratch, we will lead students through every aspect of language model creation, including data collection and cleaning for pre-training, transformer model construction, model training, and evaluation before deployment. Table of contents: 1. Foundations 1.1 Overview 1.2 Tokenization 1.3 PyTorch & einops 1.4 Resource Accounting - FLOPs - Memory - Arithmetic Intensity 2. Transformer Architecture 2.1 Model Architectures 2.2 Hyperparameters 2.3 Attention Alternatives 2.4 Mixture of Experts (MoE) 3. Systems for Training 3.1 GPUs and TPUs 3.2 GPU Kernels 3.3 Triton 3.4 Parallelism - Part I - Part II 4. Scaling 4.1 Scaling Laws - Theory - Applications 5. Inference 6. Evaluation 7. Data 7.1 Data Sources 7.2 Datasets 7.3 Filtering 7.4 Deduplication 7.5 Data Mixing 7.6 Synthetic Data 8. Post-Training 8.1 Supervised Fine-Tuning (SFT) 8.2 RLHF / Mid-Training 8.3 RLVR 9. Alignment 9.1 Multimodal Alignment 10. Guest Lectures 10.1 Daniel Selsam 10.2 Dan Fu Links: • Site • YouTube Navigational hashtags: #armknowledgesharing #armcourses General hashtags: #llm #largelanguagemodels #stanford @data_science_weekly