Mastering Retrieval Augmented Generation (RAG): Build Effective AI Pipelines

In this video, we talk about Retrieval Augmented Generation. The idea of RAGs are pretty simple – suppose you want to ask a question to a LLM, instead of just relying on the LLM's pre-trained knowledge, you first retrieve relevant information from an external knowledge base. This retrieved information is then provided to the LLM along with the question, allowing it to generate a more informed and up-to-date response. In this video, we are going to start with the most basic barebones RAG pipeline – and identify how individual components of this pipeline works and how modern frameworks have made it ultra-powerful and ultra-reliable.

Give me a follow on Twitter for regular channel updates + daily ML/AI tutorials: https://x.com/neural_avb

To get access to the write-up, slides, and other files produced for every video in the channel, check out our Patreon.
https://www.patreon.com/NeuralBreakdownwithAVB

#ai #largelanguagemodels #machinelearning

Resources :
Vector Databases: https://superlinked.com/vector-db-comparison
Metadata Filtering: https://www.pinecone.io/learn/vector-search-filtering/
Contextual Chunking: https://www.anthropic.com/news/contextual-retrieval
Propositions / Dense X Retrieval: https://arxiv.org/pdf/2312.06648
Hypothetical Document Embeddigs (HYDE): https://arxiv.org/abs/2212.10496
FLARE: https://arxiv.org/abs/2305.06983

Timestamps:
0:00 - Intro
1:19 - Retrieval Augmented Generation Blueprint
4:00 - Chunking and Contextual Chunking
6:54 - Data Conversion - Language Model Embeddings
8:29 - Data Conversion - TF-IDF and BM-25
10:54 - Vector and Graph Databases
13:00 - Query Rewriting
14:21 - Contextual Query Rewriting, HYDE
15:24 - Post Retrieval
16:00 - Reciprocal Rank Fusion
17:00 - Outro