Mastering Retrieval Augmented Generation (RAG): Build Effective AI Pipelines

In this video, we talk about Retrieval Augmented Generation. The idea of RAGs are pretty simple – suppose you want to ask a question to a LLM, instead of just relying on the LLM's pre-trained knowledge, you first retrieve relevant information from an external knowledge base. This retrieved information is then provided to the LLM along with the question, allowing it to generate a more informed and up-to-date response. In this video, we are going to start with the most basic barebones RAG pipeline – and identify how individual components of this pipeline works and how modern frameworks have made it ultra-powerful and ultra-reliable. Give me a follow on Twitter for regular channel updates + daily ML/AI tutorials: https://x.com/neural_avb To get access to the write-up, slides, and other files produced for every video in the channel, check out our Patreon. https://www.patreon.com/NeuralBreakdownwithAVB #ai #largelanguagemodels #machinelearning Resources : Vector Databases: https://superlinked.com/vector-db-comparison Metadata Filtering: https://www.pinecone.io/learn/vector-search-filtering/ Contextual Chunking: https://www.anthropic.com/news/contextual-retrieval Propositions / Dense X Retrieval: https://arxiv.org/pdf/2312.06648 Hypothetical Document Embeddigs (HYDE): https://arxiv.org/abs/2212.10496 FLARE: https://arxiv.org/abs/2305.06983 Timestamps: 0:00 - Intro 1:19 - Retrieval Augmented Generation Blueprint 4:00 - Chunking and Contextual Chunking 6:54 - Data Conversion - Language Model Embeddings 8:29 - Data Conversion - TF-IDF and BM-25 10:54 - Vector and Graph Databases 13:00 - Query Rewriting 14:21 - Contextual Query Rewriting, HYDE 15:24 - Post Retrieval 16:00 - Reciprocal Rank Fusion 17:00 - Outro