Fine-Tuning LLMs: My Catastrophic Mistake

You're about to make the same fine-tuning mistake that wiped out a month of my work (and costed about 100$). In this video, I break down what catastrophic forgetting is, why it happens when fine-tuning LLMs, and exactly how to avoid it. Whether you're fine-tuning a Falcon model or any other LLM, this will save you. Topics covered: LLM fine-tuning, catastrophic forgetting, Falcon, LoRA, training mistakes Github: github.com/younissk/falcon-twig Hugging face: huggingface.co/younissk/Falcon-Twig-7B 00:00 Abu Dhabi's Ambitious AI Government Project 00:43 Falcon Models: Performance and Challenges 02:27 Testing and Benchmarking Falcon Models 05:48 Creating a Custom Dataset for Tool Calling 08:05 Training and Fine-Tuning the Model 09:46 Analyzing the Results and Future Steps 13:04 Conclusion and Lessons Learned