Unleash the power of Local LLM's with Ollama x AnythingLLM
Running local LLMS for inferencing, character building, private chats, or just custom documents has been all the rage, but it isn't easy for the layperson.
Today, with only a single laptop, no GPU, and two free applications you can get a fully private Local LLM RAG chatbot running in less than 5 minutes!
This is no joke - the teams at Ollama and AnythingLLM are now fully compatible, meaning that the sky is the limit. Run models like LLama-2, Mistral, CodeLLama, and more to make your dreams a reality with only a CPU.
Ollama: ollama.com/
AnythingLLM: useanything.com/download
And you are off to the races. Have fun!
Chapters
0:00 Introduction to Ollama x AnythingLLM on a laptop
0:36 Introduction to Ollama
1:11 Technical limitations
1:48 Ollama Windows is coming soon!
2:11 Let’s get started already!
2:17 Install Ollama
2:25 Ollama model selection
2:41 Running your first model
3:33 Running the Llama-2 Model by Meta
3:57 Sending our first Local LLM chat!
4:53 Giving Ollama superpowers with AnythingLLM
5:31 Connecting Ollama to AnythingLLM
6:45 AnythingLLM express setup details
7:28 Create your AnythingLLM workspace
7:45 Embedding custom documents for RAG for Ollama
8:22 Advanced settings for AnythingLLM
8:53 Sending a chat to Ollama with full RAG capabilities
9:30 Closing thoughts and considerations
#opensource #llm #privategpt #localagent #chatbot #ragchatbot #rag #openai #gpt #customgpt #localai #ollama #freechatbot #aitools #aitoolsyouneed #aitoolsforbusiness #freeaitool #freeaitools #llama2 #mistral #langchain #tutorial #aitutorial #aitools2024 #aiforbeginners #aiforproductivity
Ollama is now officially on windowssss!!!!
.. Docker... It always was
Docker image is a WSL2...Linux port. Its not Windows, you're just using a third party app to run it in windows at the cost of resources to run the docker server. (which ive been doing for a long time, duh...) I mean its out on windows which makes development SIGNIFICANTLY easier. ;)@@geekazoid47
lol yeah I just saw the release right after this video posted. Good call. I currently have my GPU passed through to my windows VM for different compute things. Glad to see ollama on windows. I’ll definitely try this out.
@@geekazoid47 pain in the ass to setup for someone with no knowledge on setting up docker
@@3ace846 wdym docker is the easiest thing much better than regular commandline installing of things
Thank you for your hard work! This is really a game changer,now people can build they personal chatbots,with massive databases and knowledge about they favorite topics,without using or paying online services. This is my new favorite piece of software together with LM studio. Huge respect and keep up with good work.
Amazing insight! I was already using Ollama, and adding AnythingLLM is the icing on the cake 👍. Thanks for the video!
What do you think about this model , what’s your experience?
The only video I found that gave me exactly what I was looking for...THANKS!
This is mind blowing! Thank you, much appreciated!
thank you for your work
Great video, Thanks very much
This is great, and nice to see Linux support is there now as well.
This is very cool Tim!
Great video thanks for developing this
This is exactly what I was looking for! :)
Perfect video! It has changed a bit though, so you don't need to pick lance etc, it just shows it as chosen.
Excellent! Thank you!
Nicely done. I like the flexibility of the tool.
well done. Looks solid.
Looks cool! Gonna try it right now
what about agents? do you plan to integrate agents? with crewai, or autpgpt, also why only ollama, you can integrate llm studio as well! sounds like a promising project.
I love AI, and I'm dabbling in all kinds of things, but I could never get LLMs to really work on my local machine. THIS is amazing. I got it working on my gaming laptop.
This is awesome! I'm so glad I found this video cause being able to talk to my pdfs was something i've been wanting to do for a while! I do have some questions though. I have pdfs of tabletop RPG books and when I ingest a certain book and begin asking it rules questions it seems to 'reach out' to other sources and rules that aren't in the book and the responses are incorrect. Is there a way to narrow the chat to only answer rules and such about that particular game or book instead of adding knowledge on other games it has?
Ah, this is an error class i just refer to as "general knowledge creep". The only semi-good remedy is just a really strong system prompt explicitly detailing that only knowledge in the context should be used and to not use it's existing understanding of a game on how rules for this game should work - or something along those lines. System prompt can def fix this and depending on the model will vary in performance.
THIS IS AMAZING
Great stuff, will be putting it on my windows machine...can't wait till a linux/Ubutu Appimage is released.
Working on it - just having trouble getting a stable Ubuntu (with UI) running so i can fully test the end-to-end flow
@@TimCarambatTake your time, better to get it right then having to patch-it.
@@GregRutkowski I got an AppImage live on the main site now. Tested and built on Ubuntu 22, not sure how compatible that build system is with other distros, but I dont have inifinite linux machines (or time!) useanything.com/download
@@TimCarambat Time to test it out...
Thanks Tim for this great stuff ! I am interested in generating a document based on data within multiple local files. Do you think it's easily feasible with ollama/AnythingLLM?
Nice tool indeed, thanks
You are amazing, great video
Really Thank you for your help. I'm trying.
You are a legend❤️
thank you for this awesome explanation
Kudos on Anything LLM.❤
I have a spare iMac, nice to see it works on intel. I get to use it for something other than youtube videos on the corner of my desk.
Many thanks for introduction the AnythingLLM. Is there any date mentioned when this app will be release for Ubuntu. I have to admit it blocks me 😀but there are so cool features to automate our work. Or maybe is there any similar app to integrate Ollama with vector DB, embeddings, etc in similar way ?
Great explanations in your its eas very useful and helpful information thanks Sir😊😊😊😊😊😊
great work tim
This looks great thanks!
When you embedd website, does it only embedd that one webpage or goes recursively through the entire domain (like web spyders) do?
Awesome thank you ! The possibility to learn by giving it docs/URL is so cool. I have one question though: Is it possible to use AnythingLLM with an API ? So we can use a model (Llama 3), add data thanks to AnythingLLM, then use an API to talk to AnythingLLM that returns response ? Thank you in advance for the answer, I guess many people wonder the same ^^.
Thank you Tim very helpful. May I ask the similarities/diff between anythingllm and privategpt ?
Can I specify the context size to use myself (for models with very large contexts like command-r ?)
Thanks a lot. I have 2 questions. Does it have access to the internet to fetch recent updated information? What's the best model to use for software engineers?
You can scrape websites with the built in scraper but its not "internet connected". Not like an agent-LLM. CodeLlama for local LLM. Claude-3 Opus for closed source
This was nice. What other integration options are available, particularly on window with a good GPU
If on windows you can run LMStudio (just like ollama but with a UI) and LocalAI (requires docker to run). Outside of that you can also turn to hosted LLMs on OpenAI, Azure OpenAI, Anthropic, Google Gemini Together AI, and Mistral
Good stuff. Are there any ways to connect to APIs (similar to how chatGPT works) where we could use large data sets for free? Adding individual websites and files could take a lifetime...any workarounds?
I love the idea of AnythingLLM but for some reason I cant get it to relate to the documents I provide.. and Ideas what might be causing that?
I have a M2 Max Macbook Pro. Ollama with 7B Llama works super fast when used in Terminal. Why is it slow when I am using AnythingLLM along with it. There is a drastic time difference! Pls advise.
@Tim This is GREAT STUFF. Your putting in some serious effort and it shows. Can I use this to setup on a client site? I am a software developer and would like to create embeddings specific to ai python projects and documentation... what are a couple options i should consider. Thanks
You absolutely could but youll want to run our Docker offering because it is meant for that. The desktop app is really a local "single-player" solution. The docker app has a full API, multi-user support, etc etc
Just for the record Ollama does support embedding models. They work very well (very fast) for text, not sure about other modalities.
That was really fantastic. Tm once at work i’d go full hog at it and they let you know! By the way, do you plan to implement DSPy somehow?
Unlikely because it cant fit into our stack (since its Python). The idea of the framework however is not impossible to support
would you consider adding more plaintext files, such as .py files? Just installed the latest version in docker and its great, love everything about this!
You know it's funny, we allow pulling in and embedding of whole github repos, but not the independent code files. I will work on resolving that for you - thanks for pointing it out 😄
@@TimCarambat you are absolutely welcome, if i helped even 0.01% it made my day!
Reply just to say we support this now (and have for a few weeks now!)
sweet tool bro!
Hello, anything in the pipeline for using the new ollama embeddings? they are super super fast :-)
Already live in main repo. Will be live in next weekly release
Bro this is amazing! Currently training models seems like a big hastle most training is done via google collab and such any plans on adding an easy UI interface to train models locally inside AnythingLLM?
Being able to fine tune off the chat and such is on the roadmap, but is still conceptual since 99% of people cannot train a model on their device. So we may integrate or connected to some provider which can enable this to happen more easily. We allow you to export the chats in a fine-tune format already to encourage this behavior and id love to close the loop on it
@@TimCarambat Thanks Tim appreciate you.
Awesome!
Hello, Tim, how are you? Thanks for the video! I have a question, I have Ollama on a ex mining rig with 5 GPUs, the response time is very good, but when I ask 2 questions in parallel it does not answer in parallel, he answers the first question and the second question remains in “queue” until I finish answering the first question, could you help me? Thank you
Can you put an example using Hugging Face? I don't know what happens but Anythingllm is asking ChatGPT and not my documents.
I get "-- loading available models --" in the Chat Model Selection combobox after selecting Ollama Base URL (that is correct btw) because I can open ollama one ui. What am I missing here?
Same. Whats the issue? Have you found any solutions?
@@Musaddique_MMALI I reinstalled today because I formatted my computer. Now is working fine.
Is there a timer or countdown clock for the embedding and processing time?
Not currently, it would be nice to add though for obvious reasons
It seems to be having an issue reading the entirety of my Docx, only 103KB. It's the CBA for my workplace. Once embedded it states that there is no context for the question I've asked, though there most certainly is. (IE, shift schedule times, etc...), it will then proceed to read OTHER parts of the document. I wonder if there are certain document types that work better than Docx? or if anyone has any suggestions?
When you scape a URL does it extend the scrape to every page on the site? Or would I have to add all the pages of a site manually? How deep is the URL scrape?
The one in the uploader is currently just the url. Understandably that is not great for large url ingestion and there is a piece of work being done to add deep scraping (or at least multi link, possibly agent based) to the tool.
Would it allow local server which gives REST api? not in tcp ip, but normal android app backend api format @GET, etc..?
The docker image comes with a full REST API. It is accessed via TCP/IP though. You can wrap it though on the Android side with Retrofit or something like that
very impressive :-o
Can I use AnythingLLM locally and privately only as an API like the OpenAI API? In addition, do you offer anything like the Assistants API, where you can have models that have pre-defined instructions on how to respond?
Ollama works very slow on Windows. Tested using LM studio and it's working.
Can you do a review on AirLLM it’s lets you run a 70b model on 4gb of vram, or is there a way to run it on your software?👍👍👍
Hi, I noticed you are still in chat mode when you asked questions, isn't it better to switch to query mode ?
Chat is general knowledge + documents Query is just documents. In this case, since each query had used context - the results would have been the same
Thanks for the tutorial, and the combination with Ollama and AnythingLLM are exactly what I need. But when I try to embed with several PDFs, always got failed error. Do you know what I did wrong? I'm running Ollama from AnythingLLM with Mistral as LLM. Normal chat is ok. I'm on mac.
Actually I reported a bug, maybe it's not a bug but something I set wrong. Sorry for false reporting then. github.com/Mintplex-Labs/anything-llm/issues/821, thanks again for making this great app.
@@Gabriecielowhat kind of use case you have tested with the embed files (what format?)? I am interested in generating a document based on data within multiple local files. So you think it's feasible with ollama/AnythingLLM?
@@m.bouanane4455 I tried PDF, more text books. Mitral 7B and Gemma tested, Gemma lost a lot to Mitral. And it works with multiple docs too. One thing I felt is AI needs my questions to drive him read and think deeper into the book.
Tested with PDF mainly, works fine. And we could easily switch all open-source LLMs. For using AnythingLLM to generate doc, you should ask Tim, I only tried use interface, didn't know if API is available or other tools for you to drive. @@m.bouanane4455
2 months have passed since the original video, 2 questions. I am running a 12 GB vram RTX3060 GPU. Will LM Studio be a better solution than Ollama? 2. Now Llama 3 has come out, will thos work with what is presumably a much more capable LLM?
Looks great! Do you have an idea of when Linux will be available?
Been working on it this week! Looks like the app image should be good by next release, likely next week
I installed it on windows and when I try to upload a document the my documents windows comes up but it says document processor unavailable. how can I fix it
I love the ease of set up and the local private models. First attempts at use are not good in terms of models ability to understand and formulate answers to questions based on provided context docs even with only 8 documents provided. I did try it with just one document provided, and answers seems less confused. Ideally I'd like to be able to point to a whole drive and chat with my own currated knowledge base. Is that an inappropriate use of this tool?
Got this working. How do i get it to summarise a pdf please?
Thanks for the video, but I'm not sure why, but my windows version of anythingllm is slightly different and I can only follow so far before what's on my screen is different than yours. Is there a video for the windows version? Please make sure that you explain the differences.
Where is the disconnect for you? This video is quite old now and we have improved the product a lot since
That's cool! btw i think you've rendered a 16x10 ratio video to a 16:9 It looks a little weird 😅
Can "recursive similarity" be set as the RAG method?
I used LLama 2 LLM model with anything LLC just shown in this video and use my website to train the LLM model but I am still getting generic answers that are unrelated to my website. how do I fix this?
Are you just downloading the HTML web page for a site and then compiling that as a datasource is is there an agent actually crawling the site looking for the data based on a given user input?
Just downloading the page. No agent support currently
can you make few more videos specifically for the agents and details of embedding and possible troubleshootings please.
Did you have to give ollama 2 admin perms aswell?
Doesn't seem to work with KZhead URLs - I add the URL and ask what the key messages are and I just get a load of nosnsense about something totally different!
Are there any good programs that give ollama memory of the previous conversations it's had? like a memory recall function and storage
That is built into anythingllm automatically. Outside of that you have to manage it directly in ollama
@@TimCarambat "outside of that" what do you mean by that?
@@ogr3dblade6 I mean outside of just using anythingllm, sorry
Literally 5 minutes later, it stops calling me by my name and it forgets who i am.
@@TimCarambat is the session RAG feature flawed or do you think i messed something up?
How cai i use local Ollama in Visual Code please for agents Crewai?
Are you planning on adding support for Nvidia Chat with RTX?
Do they have an api for that tool?
For ollama on Macbook pro 2016 x86 and 32gb or ram I recommend Solar, Mixtral is good but slow.
How can I get a list 📝 of models that work on OLLAMA?
@@ajarivas72 Lllama3 now is not bad, but still solar and Mixtral are good too.
Next we just need to serve a webpage with authentication. Complete package.
The docker version has multi-user with auth :)
AnythingLLM only recognizes the first document I uploaded, but not the 2nd doc and not even see the website link after the 1st upload. I reset, deleted workspace, created a new workspace, restarted my computer, but AnythingLLM still not seeing the 2nd upload. Is it a bug? Please fix. Thanks, Tim!
Hi there, Thank you for this video.... I just can't get it to work i followed your steps.... I get an error message "pinecone evn settings" or something like that... I don't know how to fix it or where to find the EVN settings.... I would greatly appreciate the help
Go to "Vector database" and select LanceDB. Not sure how you got to that state since lancedb is the default, but that would fix it.
Thank you very much. I really appreciate it. I will try it out tonight. Have a great day. Best regards
So my question is is it possible to set up LLM anything to be accessible over a local network?
Yes, absolutely
Is the GUI app for AnythingLLM open-source?
Yes, it's the same repo as the main app, it's just the desktop branch. github.com/Mintplex-Labs/anything-llm
@@TimCarambat Thank you.
I tried Anything LLM (using the default suggested tools for embedding, vectors and so on), with LM Studio, tested with multiple LLMs, it works fine, I can chat without trouble, until I decide to upload a document file (tried with a PDF paper from arxiv) or a web url and move it to the workspace, it gives me no error, everything seems fine, but now whatever prompt I use, whether in chat or query mode I get an error in the chat window as a response: ⚠ Could not respond to message. An error occurred while streaming response. Failed to fetch. I do not get any error in the LM studio log, so I suppose no data get sent to the endpoint server. Everything freeze down cause if trying to go in the options for the workspace it doesn't do anything, if I go in the general options I do get an infinite loading circle loop. To fix it I need to reload and remove the document from the workspace. My System: i7 3770k, 32GB + RTX 3060 12GB. Windows 10. Note: Since my AnythingLLM doesn't have that thread option in the GUI I suppose I've to update AnythingLLM and check if the error doesn't show up in the new build.
The thread feature if available in the latest version of the desktop app - so you can redownload and just overwrite the application and youll still keep your data. It looks like you have plenty of power and memory. "Failed to fetch" with LMStudio almost always means that the endpoint being used or the API key (if applicable) is incorrect. The LMStudio endpoint should be somthing like `localhost:1234/v1` in AnythingLLM. Make sure you started the LMStudio inference server in that application as well since it does not start on boot. medium.com/@ingridwickstevens/running-a-local-openai-compatible-mixtral-server-with-lm-studio-cd602efbf808
@@TimCarambatI don't think it's LM Studio related at all, cause I used it with multiple tools without trouble, as I said i used localhost:1234/v1 (you can't set the API key in AnythingLLM), but chat work fine in AnythingLLM, but as soon I try to chat or query after embedding anything I get that error (Could not respond to message. An error occurred while streaming response. Failed to fetch.) Update: I installed the latest. First tried without deleting anythingllm preference folder and got a fail in trying to save and embed the document suggest trying a different schema than LanceDB. Since the error was different I thought maybe the preferences have compatibility issues, so I cleaned up the preference folder and started from scratch. And got the original message on the arxiv pdf (but no failure on embedding and saving this time), this partially froze the process as I explained above, I also got multiple wmic.exe and conhost.exe processes, a few times multiple powershell.exe spawning on my system and high CPU usage from process: WmiPrvSE.exe. So I restarted and tried uploading a new story from the web from a basic webpage (longstoryshort.squarespace_com/great-example) and got an error when attempting to move the doc to the workspace(Save and embed): It fails again and suggest trying a different schema than LanceDB but at least it doesn't freeze (it still spawns multiple processes). So I think it's a problem with the LanceDB wheter it does fail to save and embed or not. I will try installing Chroma and see what happens, will try Ollama as well, I would still check.
Can we use Dalle 3?
I have ollama in docker [OllamaBAseURL: host.docker.internal:11434 ...cannot set the ChatModelSelection...guess it should take the default and only model: llama2] &anythingllm in docker. Get this error when trying to chat...Could not respond to message. Ollama:streaming - could not stream chat. Error: unknown scheme. Does anythingLLM support ollama in docker? What could be be problem?
Ollama in docker does for sure work, some do it so its def a configuration issue between the containers and your host relaying the connection
Awesome, Army Swiss Knife for LLM Can I spin up a Streamlit app with this ?
Can’t connect to base url on app my Ollama shows url
Can we buy a domain and hosting it as a website from my home computer ?
Sure, the docker version you can host anywhere like render, aws, railway, gcp. Doesn't matter. You can use custom domains there so no reason why not
already using ollama... and must intergrate it with Anything LLM. I will say ollama works a bit slow, may be because of the 7B Parameters. It take generally 70-80 seconds to deliver an answer.
Are you on an intel-based processor with no GPU? If so this is par for the course and normal.Bigger model can lead to even longer time-to-first-token or crashes
I’m confused. What is the point of Ollama? Why use this instead of anythingLLM? I’m new to all of this. So just a little confused as to the “purpose” of ollamma
Anythingllm is a chat UI and tool suit for multiple types of LLMs! Gpt-4, mistral, anthropics Claude, Google gemini, and local llms! Ollama is a tool that helps you download and run local LLMs but anything LLM can connect to it and give you a nice UI for chatting with your documents and more. AnythingLLM isn't an LLM, it's a tool for LLMs. I didn't make that clear in the video, but ollama is just a simple way to run local LLMs on your desktop/laptop
@@TimCarambatdo you have a video showing how to just use anything LLM and your preferred tool, ollama or LM studio on just one computer? For example, if I just have 1 windows computer how to do everything from start to finish using your tool and whichever one of these tools you prefer the best? So in other words, install, how to get the LLM, etc I know you have that one video but it involved using a server. And which do you prefer LM studio or Ollama?
Would love to try it, but the download speed is like 10kbps right now
Was not able to run by input of pdf files in context, why don't you demonstrate it with files, in all the videos you demonstrate it with URL only
Can you expose this as a web app with something like ngrok? So that I can access my private LLM from my browser or my phone?
We have a docker version that specifically is made to have multi-user AnythingLLM as well as browser-based. Our mobile UI is a bit limited, mostly from uploading new docs, but it can be accessed for chatting.
if i even try to download it my pc will just burst up and catch fire
for me it never loads the models :(
tim you might want to fix up a few things.. embedding a url to the machine doesn't give super accurate results. The model makes errors when it comes to fetching data from a website. Have you been able to encounter this problem? Let me know.. keep it up
This can be resolved by settings you can configure. The default settings work for most things, but we allow you to modify them because sometimes given your model, dataset, embedder selection, or more - the default configs may not work. Common fixes to the LLM still missing the point when querying and documents are embedded: - Change the Workspace Vector database "Document similarity" to "No Restriction" - Pin the document to the workspace, which ensures full-text comprehension, but obviously consumes many more tokens and takes much longer to yield a response.
Can you make it use a GPU to speed up the process?
Ollama will use gpu (Cuda) and also has AMD support out the box now. There are additional gpu specific settings you can add, but yes