"I want Llama3 to perform 10x with my private knowledge" - Local Agentic RAG w/ llama3
Advanced RAG 101 - build agentic RAG with llama3
Get free HubSpot report of how AI is redefining startup GTM strategy: clickhubspot.com/4hx
🔗 Links
- Follow me on twitter: / jasonzhou1993
- Join my AI email list: www.ai-jason.com/
- My discord: / discord
- Corrective RAG agent: github.com/langchain-ai/langg...
- LlamaParse: github.com/run-llama/llama_parse
- Firecrawl: www.firecrawl.dev/
- Jerry Liu build production-ready RAG: • Building Production-Re...
⏱️ Timestamps
0:00 Intro
1:33 How to give LLM knowledge
3:05 Problem with simple RAG
5:55 Better Parser
9:01 Chunk size
11:40 Rerank
12:39 Hybrid search
13:10 Agentic RAG - Query translation
14:35 Agentic RAG - metadata filtering
15:52 Agentic RAG - Corrective RAG agent
17:33 Install LLama3
18:00 Code walkthrough
👋🏻 About Me
My name is Jason Zhou, a product designer who shares interesting AI experiments & products. Email me if you need help building AI apps! ask@ai-jason.com
#llama3 #rag #llamaparse #llamaindex #gpt5 #autogen #gpt4 #autogpt #ai #artificialintelligence #tutorial #stepbystep #openai #llm #chatgpt #largelanguagemodels #largelanguagemodel #bestaiagent #chatgpt #agentgpt #agent #babyagi
This is prob one of the best RAG video I've seen, so many learnings in 20 mins
Man, your videos keep getting better every time I look. You have a great mind and your presentation is excellent. Thank you very much, again, for sharing!
he is much better than 99.9% wanna be over hyped ai gurus on youtubu, twitter and linkedin!
Yet again an amazing tutorial, thanks so much Jason!
Great content! Thanks for putting in the effort. Will use this.
One of the most informative RAG videos I’ve seen. Can’t wait to see more from your channel.
Holy crap! This gave me such amazing background knowledge, love it! Now, what would be extra cool, would be if you could do a real "hands-on" type of workshop to go through it all by setting up the environment completely, including the actual training/RAG implementation of a set of various document types (PDF, excel, website etc..) to extend a locally running llama 3 instance 😊
What a great video! Thanks for sharing your knowledge
Really great tutorial, teaches a lot in very short time! Thanks!
This is the best RAG video on the internet, awesome job, no fluff, high complexity but easy to understand, nice work
dude... great video! Thanks for the knowledge!
You always amaze me by the amount of knowledge I get from your videos
Didn't know about the Agentic RAG techniques, thanks for sharing!! That's definitely a trade off between speed & quality, but good to have the option
Amazing info shared -. Thank you!
Excellent video! I don't have much experience with RAG and this was sooo helpful!
00:05 AI can revolutionize Knowledge Management 01:46 Llama3 can process precise knowledge with fast inference 05:27 Market strategy for AI startups 07:16 Convert PDF files to markdown format for enhanced accuracy and control 10:47 Finding the optimal chunk size through experiments 12:34 Hybrid search combines Vector search and keyword search for better results 16:12 Building a local agentic RAG with llama3 17:48 Running Llama3 model on local machine and using Visual Studio Code 20:53 Setting up key components for Llama3 performance 22:20 Creating a complex agentic RAG workflow for document retrieval and answering
Amazing tutorial! Thank you
right when i needed it, thank you man! also, just finished watching and i understood the theory behind it but kinda got lost during the code explanation, i might watching again and again
This is extremely helpful! Awesome!
You're the man 💯👏
Jason, I watch a lot of AI videos but I learn the most from yours. I am actually excited everytime i see you have put another one out. Keep up the great work!
1) The link for the corrective RAG agent had an extra URL attached at the end which caused it to fail; manually tracing the link got me to the proper location 2) LlamaParse looks like a wonderful tool, since I have a lot of documents with equations, and I really need it to grab equations, if for no other reason than to return them. Unfortunately, LlamaParse requires an API key and seems to send PDFs off for processing, something that others have noted and there is an open issue from 2 weeks ago. As of 3 hours ago, it's still an open issue - clearly most companies don't want to send internal docs out of house. Hopefully this gets resolved soon. 3) Really liked your presentation - easy to follow every step with the provided materials.
Hopefully we will have more better options for local use - shame it's not a local only pipeline yet
yes - I have found this issue too. LlamaParse seems use OpenAI llm to process the pdf and it leads to the privacy concerns.
Solid video Jason
Firecrawl boosted our RAG accuracy at our company. fast + provided good markdown format. Llama parse also super helpful too! Amazing video Jason! This is gold! Edit: thanks for the likes :)
The search api is just insane on firecrawl
Awesome content!
Great tutorial! Thank you
Keep this up. This answered to loads of questions I have had previously, and were not answered in any of the HuggingFace tutorials!
many many thanks, bro!
Thanks!
Subscribed, dont have an AI company since I'm still a poor student... this video was very informative, the man speaks at two times speed just like my professor. I respect it 😁
wow nice work thanks!
Great timing! Why do you always read my mind JASON!!?! lol
awesome jason thank you
I am literally using this technique now in my internship for a project. I went through so many approaches and ended up on my version of this one. Wish you released this video about 2 months ago lol
Great video, thanks
Thanks... Awesome video
Best video💯
This is a great trick, thanks
Amazing lesson! I learned a lot in just 20 min!
Thanks! It's so fascinating how these programs 'think.' Even if I don't install one, concepts like chunking seem to translate to humans as well.
great video keep making these please.. only "criticism" / advice if you can call if that is to keep things focused on local / open source solutions as much as possible.. love the use of Ollama here for example.. things that perhaps don't require API keys, subscriptions, external integrations / dependencies help people like me understand more of what's going on in a workflow like this! thanks again!
OG Jin Yang from Silicon Valley.. Amazing video 🎉
This is epic! keep up...
You are relevant, Subscribing to your channel!
Clicked that BELL too! 🔔
I prefer finetuning to RAG first then RAG on top of the finetuned model. Just a simple QLORA is all you need. It really helps a ton.
How would you go about doing that, as in just do it backwards from the video?
Platform agnostic LLM space overview videos from Jason are the best on AI YT
Such a bait and switch. Thumbnail promises fine tuning tutorial. Delivers best improve-your-RAG video on the internet. Excellent work.
I have to say, it is great :D
The corrective RAG schema explains why AI often tries to bring results from the web even when you tell them not to in prompt. If it doesn't understand the source properly it will look elsewhere. This was insightful, thank you.
here come dat boi!!!!!!
Amazinnnnggggg🎉🎉🎉🎉
Thanks Jason, great video, this explains RAG pretty well. Subscribed!
thx
great video jason! quick question, im wondering if a knowledge graph in place of vector database would be better since it mitigates the lost in the middle problem?
Awesome content Jason. A Question. I need to create an AI psychologist and store college data, but this college data is a guide of what to speak, not the content itself. In that case, what is the best approach, RAG or Fine-tuning?
I thought we were gonna fine tune llama3 😢 but the fire crawl implementation looks unreal I’ll have to check that out and add it to my rags. I don’t know how well it’ll work for RAGs but people have extended the context window like crazy and still can do the needle in haystack to around 130k. If you have 64gb on the Mac you can try out the 256k context window Llama 3 released by Eric Hartford. Would love to see a side by side with both of them using the same embeddings.
too many api calls here - do it local with no api calls - better and the model has to be able to crawl more doc formats - people will probably do p2p, real time and uncensored models for 'real' open source ai that has no limiting factors like api calls or tokens - this is where things need to go in order to take off, gain relevance and leverage economies of scale, of course cxl and better i/o will help but those are on the way. real open source ai will hit smb mkt in about 4-5 years and there will be more innovation and discovery - exciting times as we all watch the development curve
Hey Jason thanks for the video, I think it helps a lot. Can I apply on GPT as well?
Hi brilliant session , do you have a link for the notebook ?
You got a sub. Finally, an AI channel that actually teaches.
Great video, thanks. New subscriber (and like) here. I had a couple of questions though: why use langchain? It seems unnecessary from what I have read. Would also love a demo ipynb/copy of code.
This answer a lot of questions why my chat with PDF doesn't work, llama parser & firecrawl looks so freaking good!
Thanks, Jason, incredible as always! Would you consider sharing the code from the walkthrough? 🙏
Thanks mate, appreciate it! Code is in the description link!
@@AIJasonZ Link is not there
Great video Jason, I only missed routing as a technique to determine if your question should really go through the RAG. James Briggs has done a few good videos on “semantic routing”. Is your example notebook available somewhere?
I'm wondering the same thing. Don't see a link to a github repo
would be great to get a video on best methods for data extraction from these pdfs
great content! why wouldn't you use groq to speed up the agent response?
Came to train my 3 Llamas... Now I'm a full stack developer.
Thank you. Can you say a little about your hardware setup for this work? This information is missing from a lot of online sources.
I'm a simple man. I see a new AI Jason video, I click.
Goddamn it Jian Yang
I don’t understand everything but I can feel the gold penetrating my ears
The speaker in the transcript discusses the use of AI, particularly large language models, in knowledge management. They highlight that AI can provide value in managing vast amounts of documentation and meeting notes, which can be overwhelming for humans to process. The speaker also mentions the potential disruption of traditional search engines like Google by large language models, which can provide hyper-personalized answers based on their extensive knowledge. The speaker then introduces the concept of a retrieval augmented generation (RAG) pipeline, which involves extracting information from real data sources, converting them into a vector database, and retrieving relevant information to answer user queries. However, they also note the challenges in building a production-ready RAG application, including dealing with messy real-world data, accurately retrieving relevant information, and handling complex queries that may involve multiple data sources. The speaker also discusses various tactics to mitigate these challenges, such as better data preprocessing, optimal chunk size, relevance-based retrieval, and hybrid search methods. They also mention the use of agentic RAG, which utilizes agents' dynamic and reasoning abilities to decide the optimal RAG pipeline and improve the answer quality. The speaker concludes by expressing their curiosity about how AI-native startups operate and embed AI into their business processes. They recommend a research document on the subject for those interested. In summary, the speaker's points are: 1. AI, particularly large language models, can provide significant value in knowledge management. 2. Traditional search engines could potentially be disrupted by large language models. 3. Retrieval augmented generation (RAG) pipelines can be used to answer user queries based on private knowledge. 4. Building a production-ready RAG application is complex due to challenges like messy real-world data, accurate retrieval of relevant information, and handling complex queries. 5. Various tactics can mitigate these challenges, including better data preprocessing, optimal chunk size, relevance-based retrieval, and hybrid search methods. 6. Agentic RAG can further improve answer quality by utilizing agents' dynamic and reasoning abilities. 7. The speaker is interested in how AI-native startups operate and embed AI into their business processes, and recommends a research document on the subject.
Dead internet thory is getting closer and closer every day
Good rag video, the thumbnail taking about "training llama3" is hurting my brain tho
Very usefull, thank you! Is it posible for the model to retrieve images or graphs from a PDF, or it's only text?
What about preparing data, for exemple as question / response, the response would be used to generate embedding and the response would be the data retrieved ?
amazing as always. could you share the notebook please
Great thanks. Can we get the repo and link to the colab notebook?
Interesting. Someone needs to create a wrapper which works out the best way to answer questions / queries, based on the input and question/query. I think intelligence of system could then be increased.
I watch lots of AI videos and 99% of them are just a waste of time. As an AI engineer, this channel is hands down the BEST yet KEEP UP👏🏼
Great video. How do I add PDF documents and llama_parse to the python notebook?
great video! Is there a github location with the code?
I like using gemini for getting quick up to date answers, and chat gpt for stuff that doesn't require up to date stuff
4:36 girlfriend walks casually in the background
Hi, what are the areas current LLMs excel at? I am new to this world of AI, but not IT (familiar with infra). It is good that people are trying out things to see what it can do. But my naïve thoughts are that as a language tool, it just looks for patterns of words that appear close together, and knows enough of the formation of language that it produces text that is not only readable, but also relevant. But this surely must have limits, if it does not actually understand? Would it be serving up answers from a well vetted and written sources such as internal KMS by using this RAG method? Our team was thinking about it use for education / learning - perhaps tied into custom flashcard and evaluation of human provided answers. Alongside the still very useful text summarisation, alternative wording suggestions.
thanks Jason, can i use llama on API and train PDf files in a specify directory train to respond
We been trying to build a middleware that connects with any inventory ERP to be able to have real time data information about inventory data for the chatbot
👍👍
4:36 Someone walks into the void and disappears
Is those steps and advices are explained on your website ? It would be amazing if you could share the code 😮
Would you have plans to create a tutorial that connects what ur teaching here and running thing on something like AnythingLLM that allows document reading to create embeddings.
Can we also finetune the 70B model? Even if its not local
How would you go about parsing documents of all kinds of types? PDFs, Excels, Word etc...Is there a way to achieve this with only one parser? Or how would you go on about this issue?
Would there be a way to automate this with Obsidian? I sporadically log everything in Obsidian and it would be amazing to find a way to do this with Obsidian
Curious how this workflow changes with bigger context length. Gradient just released Llama-3 8B with a 1M context length
is there a good parser for powerpoint?
Great video. Thanks! A lot of very good tips!
Does this code require a good GPU as a must? I am using my 32 Gp cpu and it is super super slow to generate the answer. If the GPU is a must, any commandation for GPU model? I am seeing Jason in the video generate the answer in seconds and I know he is using a mac. Thanks in advanced!
Damn it Jin yiang 😂
Hey, Jason. This video is 🔥🔥! Congrats, I was wondering if there is a chance to reach out to you? I might have an interesting offer for you.
There is a problem with the "Corrective RAG agent" URL in the description.