KZhead
Ғылым және технология

OpenAI Embeddings and Vector Databases Crash Course

Adrian Twarog
2024 ж. 28 Сәу.
352 322 Рет қаралды
Жүктеу

Embeddings and Vectors are a great way of storing and retrieving information for use with AI services. OpenAI provides a great embedding API to do this. Postman lets you make these with easy at www.postman.com/ (today's sponsor)
In this video we will explore how to create a Vector Database by creating embeddings using the OpenAI API and then storing them in SingleStore.
The first part of the video will cover how to create an embedding using just API requests with Postman. Then we will jump into Single Store and store these in a new database made specifically for vectors like this.
00:00 - Introduction
00:10 - What are Embeddings and Vectors
02:14 - Setup OpenAI Embeddings
03:11 - Setup Postman API Requests to create Embeddings
03:55 - Create Embedding
03:55 - Create Embedding
06:55 - Create PDF or Document Embedding
07:43 - Vector Database - Setup with SingleStore
08:20 - Vector Database - Create Database
09:32 - Vector Database - Create Table
10:41 - Vector Database - Insert Embedding Row
13:25 - Vector Database - Search Embeddings
15:18 - Embedding function with JavaScript and NodeJS
18:08 - OpenAI and GPT Digital Book
18:29 - Conclusion
Postman: (today's Sponsor) for API Requests
www.postman.com/
OpenAI Embedding Documentation:
platform.openai.com/docs/api-...
SingleStore Vector Database
www.singlestore.com/cloud-tri...
⭐ Teach Me OpenAI & GPT - Digital Book ⭐
enhanceui.gumroad.com/l/teach...
Learn Design for Developers!
A book I've created to help you improve the look of your apps and websites.
📘 Enhance UI: www.enhanceui.com/
Feel free to follow me on:
🐦 Twitter: twitter.com/intent/follow?scr...
💬 Discord: / discord
💸 Patreon: / adriantwarog
Software & Discounts:
🚾 Webflow: webflow.grsm.io/adrian
🌿 Envato: 1.envato.market/yRZjz2
🌿 Envato Elements: 1.envato.market/LP0OJZ
🔴 Elementor: elementor.com/adrian/?ref=23140
✖ Editor X: www.editorx.com/adrian-twarog
Computer Gear:
⬛ Monitor: amzn.to/3f9DOQI
⌨ Keyboard: amzn.to/3eA5UFD
🐁 Mouse: amzn.to/3xVJO8l
🎤 Mic: amzn.to/3hgCfms
📱 Tablet: amzn.to/3ewt7sa
💡 Lighting: amzn.to/3vOZeZY
💡 Key Lighting: amzn.to/3f6qP2f
Camera Equipment:
📷 Camera: amzn.to/3uCv4J9
📸 Primary Lens: amzn.to/3vT6wMm
📸 Secondary Lens: amzn.to/3tyqWIX
🎥 Secondary Camera: amzn.to/3o2zCGi
🎙 Camera Mic: amzn.to/33tCz9l
🎞 USB to HDMI: amzn.to/33yW9RE

Пікірлер

  • I have seen multiple tutorials, this is by far the best and most concise, great work man

    @user-zy5nu7xp1d@user-zy5nu7xp1d8 ай бұрын

  • Adrian, your channel is a gem! I love the way you explain complex topics and the pace of your videos! Greetings from Poland!

    @krisograbek@krisograbek10 ай бұрын

  • Excellent overview! Very concise, clear and relevant! Great job! Thank you Adrian! 😊

    @LindsayHiebert@LindsayHiebert9 ай бұрын

  • That isn't a vector database. It's a relational database with vectors stored on a text column. In practice, you will have thousands of embeddings and performance will tank with this setup

    @nickfleming3719@nickfleming37198 ай бұрын

    • What's a more ideal solution for storing vectors?

      @trevorbaier7072@trevorbaier70725 ай бұрын

    • From my investigation, Redis is an excellent vector store to be used in both development and production especially when it’s a local Dockerized instance

      @brookster7772@brookster77725 ай бұрын

    • Mongodb atlas is awesome for vectors. They have a new vector feature called knnbeta

      @SussyBacca@SussyBacca5 ай бұрын

    • Pinecone works too!

      @ParthSaneHD@ParthSaneHD4 ай бұрын

    • You are correct, but you know that! It’s indexing is not fast enough for many serious AI projects, and its single threaded architecture does not scale. Under the hood there are many other non-vector legacy issues.

      @amdenis@amdenis4 ай бұрын

  • Incredible teaching skills. First time ever, I loved someone who can teach "ME" the way I always wanted. Thousand thumbs up Adrian!!

    @nickkondoori7550@nickkondoori75508 ай бұрын

  • This tutorial was incredible - completely glued to it

    @daygo619ca@daygo619ca8 ай бұрын

  • Amazing tutorial! The way you explain is so easy and understandable!

    @TheOGDesigner@TheOGDesigner9 ай бұрын

  • Finally, found a video with the appropriate detail. For me! 😊 Thank you!

    @rkjellbe@rkjellbe8 ай бұрын

  • Awesome thanks. Been studying calculus and linear algebra before I dive deep into AI. I will definitely be dealing with vector databases very soon and looking forward to it.

    @phil97n@phil97n7 ай бұрын

  • Learn vector embeddings using first principles. Always engaging, and very rewarding for the learner. Thank you!

    @coinexponent1884@coinexponent18848 ай бұрын

  • Best AI video ever . Made it easy to understand with 2 simple concepts . Thanks man!

    @saik6730@saik67307 ай бұрын

  • Fantastic tutorial and explanation!!

    @karsonkalt7607@karsonkalt76077 ай бұрын

  • I just bought 2 Udemy courses, and after 5 hours, none of them talk so well about this. I appreciate it, and I will buy your book. Thanks for your content.

    @LeonardoCarreraBaude@LeonardoCarreraBaude8 ай бұрын

  • This course is gold! Thanks! I have done similar steps on Astra db and it was smooth

    @JeremyArtero@JeremyArtero7 ай бұрын

  • Very Good session Adrain... your way of teaching is keeping the people glued... Keep it up

    @RajShekarsdreamzzz@RajShekarsdreamzzz8 ай бұрын

  • Super high quality video right here. Good job Adrian

    @aiadvantage@aiadvantage10 ай бұрын

    • Hey I've seen your stuff too, it's great, thanks for the nice words!

      @AdrianTwarog@AdrianTwarog10 ай бұрын

  • Excellent. Thank you. Helped a lot.

    @demetriusmds@demetriusmds8 ай бұрын

  • My ADHD normally overrides my concentration. Your tutorial pace, live coding, and narrative made me complete my 1st Open AI coded app - thank you!

    @SuperITPRO@SuperITPRO8 ай бұрын

    • Same

      @zzej@zzej8 ай бұрын

    • Don't use ADHD as an excuse, it ain't no sickness, just personality. Take it and make it your best quality.

      @nicholastroyandersen9505@nicholastroyandersen95057 ай бұрын

    • @@nicholastroyandersen9505it’s…. Not a personality, lmfao. It’s a very clear set of learning disabilities centered around working memory, executive function, and tuning out

      @nickscherrer1735@nickscherrer17357 ай бұрын

    • same and without knowing english

      @davidabellangarrido2056@davidabellangarrido20567 ай бұрын

    • @@nicholastroyandersen9505it’s literally a neurological condition that can be seen on scans and measured… ignorant comment

      @pauls064@pauls0642 ай бұрын

  • That was the first video that actually gave me a understanding of how vector DB's kind of work. Thank you for sharing.

    @photorealm@photorealm25 күн бұрын

  • Love your thumbnails. Keeps getting better with each video 👍

    @anrk97@anrk9710 ай бұрын

    • Thanks, I try to make them as clear to what they video represents as possible!

      @AdrianTwarog@AdrianTwarog10 ай бұрын

  • Rarely comment, but damn, you did a perfect job - I am at 8:01, haven't watched the video but had to pause and comment - until 8:01, everything was perfect; how you explain concepts and utilize tools ensures that we understand the concept in practice with ease! Great job, continue making videos; you should do consulting if you don't already do so. It's easy money with little hours with your skills and knowledge!

    @rasmuspiirtola4397@rasmuspiirtola43976 ай бұрын

  • Absolutely LOVE this. you're so clear and concise.

    @AmanBansil@AmanBansil2 ай бұрын

  • fantastic job!

    @Kevin509wisdom@Kevin509wisdom8 ай бұрын

  • Will you create a second part of this video where PDF's are uploaded and then analyzed?

    @FahadKiani1@FahadKiani18 ай бұрын

  • Great video !!! :)

    @parkerrex@parkerrex7 ай бұрын

  • Let me see if I understand what’s going on here: 1) you have data you want to search semantically 2) you create a vector database capable of storing & querying data semantic search queries 3) you use OpenAI to process your data & convert it to vectors which can stored in your database 4) you store the data along with the OpenAI generated vectors 5) now you can search the data Is that all it is? I thought you were then going leverage this database to give chatgpt “long term memory” ( 0:20 ). What you’ve showed seems nice, but I don’t really see the point since most people/companies who have enough data that would need to be queried in this way would not be able to give it away to OpenAI to process.

    @adamduvick@adamduvick8 ай бұрын

  • This was a great overview Adrian!

    @codinginflow@codinginflow6 ай бұрын

  • Absolutely amazing! Thank you so much for your work!

    @sunnysk43@sunnysk435 ай бұрын

  • Well done, succinct, and excellent explainations of complex topics.

    @robertcormia7970@robertcormia79704 ай бұрын

  • Great video ! precise..

    @shpop1@shpop17 ай бұрын

  • Love it.. it was far simpler than I thought..

    @satanrasool1802@satanrasool18029 ай бұрын

  • Adrian, this is beautifully explained. Absolutely loved it :)

    @ravindrasingh2411@ravindrasingh24112 ай бұрын

  • Bare metal, removing all higher level obstructions going right down to the core. I love it the best understanding of what embedding’s earlier that I have seen great job.

    @brookster7772@brookster77725 ай бұрын

  • thanks for this video video is really helpfull

    @user-vm2er4qj9w@user-vm2er4qj9w7 ай бұрын

  • Cool course. How does one connect it to a basic website?

    @Ricocase@Ricocase8 ай бұрын

  • Wow! Easy, clear and to the point.

    @ZaidKhanPathan@ZaidKhanPathan6 ай бұрын

  • Wow, thanks I'm finally starting to get embeddings!

    @kfliden@kfliden4 күн бұрын

  • I wish everyone could have presented like you, simply Super. Looking forward for more in similar way

    @dipayanroy964@dipayanroy96418 күн бұрын

  • Nice work! Thanks so much for this awesome demo.

    @chrislannon@chrislannon2 ай бұрын

  • The voice recording and explanation is really clear - surprising how tone and voice plays a major role in understanding. Was watching another video which was equally good but somehow the slang and recording made it a bit difficult to understand. Thanks

    @karthikg752@karthikg7528 ай бұрын

  • Great content 👍👍👍, waiting for more OpenAI, AI related content

    @sany2k8@sany2k89 ай бұрын

  • This is the best video on openai embeddings I have ever seen, I am also a bit biased!

    @CodexCommunity@CodexCommunity9 ай бұрын

  • what are the prerequisites to understand the content in this video? And where can I learn them?

    @ewhite_dipi@ewhite_dipi7 ай бұрын

  • How efficient is the vector search if you need to go through all of the records every time you search? Shouldn't there be some dedicated field type for embeddings other than blob?

    @Art-kz6zf@Art-kz6zf6 ай бұрын

  • Very interesting video, but what are the prerequisites to understand & actually implement this ?

    @pablochacon7641@pablochacon76419 ай бұрын

  • Very well explained, Thanks Adrian !! I have astaffing firm and I have a database of more than a million resumes. I m planning to create a resume search application for my recruiters. Do you think I should be using combination of Embeddings and Vector Database for above use case.

    @corejava5730@corejava57308 ай бұрын

  • Would be great to see a follow up video of practical applications using this.

    @Glow0110@Glow01106 ай бұрын

    • The practical application are varied: sentiment analysis term search Classification

      @atursams5501@atursams55015 ай бұрын

  • Perfect explaination!

    @user-uw6ld2dp9z@user-uw6ld2dp9z5 ай бұрын

  • Nice video. I love your work.

    @omangramoswaane2211@omangramoswaane22119 ай бұрын

  • Great tutorial!!! I will be buying your book.

    @adavis912@adavis9123 ай бұрын

  • Crisp and to the point, thank you. Can I ask how you made the slides like the one at 0:52?

    @nrusimha11@nrusimha119 ай бұрын

  • great explanation ! thanks !!

    @user-qg5bo9bd5x@user-qg5bo9bd5x4 ай бұрын

  • This is great stuff, thanks.

    @rafaelmartinsdecastro7641@rafaelmartinsdecastro76419 ай бұрын

  • Awesome Content! Thanks

    @SarthakAgarwal-sm1gp@SarthakAgarwal-sm1gpАй бұрын

  • Perfect learning ❤🎉 master of learning ❤❤❤❤

    @mohammadbarzegari8737@mohammadbarzegari87375 ай бұрын

  • dot_product is a function offered by this database for vector searching, ranking etc.. ?

    @contactbhasker7483@contactbhasker74838 ай бұрын

  • Nice video, it would have been nice with a demonstration at the end or intro, keep up the good work.

    @pajisounds@pajisounds10 ай бұрын

    • Oh good suggestion, I’ll do that next time!!

      @AdrianTwarog@AdrianTwarog10 ай бұрын

  • Excellent content, what changes for audio search?

    @alexsalgado@alexsalgado9 ай бұрын

  • Great video. THX

    @joergmayer3741@joergmayer37419 ай бұрын

  • I'm a little confused.. If I created embeddings and which I'm assuming is essentially training the openai model on a specific topic for my company. Would it be able to answer questions only on the specific topic it was trained for?

    @user-wf2ok2tq1z@user-wf2ok2tq1z7 ай бұрын

  • For those who already had an OpenAi account and you are facing an error while posting the HTTP request, its because your free credit has expired. You will have to add a payment method or createa new account to get free credits agin and then everything will work fine according to this tutorial.

    @abijithpradeep7478@abijithpradeep74788 ай бұрын

  • [Question] When input hello earth, "Hello World" scored 0.89, meanwhile "OpenAI Vectors and Embeedings are Easy!" scored 0.74. Which is quite close to the top rank text. But syntactically first and second returned text are very different. Somehow I expect the second text might scored 0.5 and below. Could you please share your thoughts on this Adrian? Thank you!

    @chrismalingshu@chrismalingshu8 ай бұрын

    • You would need to ask someone who built the transformers at openai.

      @daffertube@daffertube6 ай бұрын

  • This is amazing

    @balajidasari9114@balajidasari91142 ай бұрын

  • excellent❤ work

    @FalahgsGate@FalahgsGate9 ай бұрын

  • Excellence & Awesome

    @madhavanand756@madhavanand7569 ай бұрын

  • That was clean. Thank You!

    @obey24com@obey24com9 ай бұрын

  • In the past, I learned Support Vector Machines for doing classification. At that time, I struggled to learn the concept, although I finally was able to implement it into a program using codes made by another party. The introduction of this video suddenly revived the memory and helped me better understand the concept of SVMs that I learned years ago. Is Postman completely free and can be used without any restrictions or limitations ? Is Single Store also completely free without any restrictions or limitations ?

    @PeterAdiSaputro@PeterAdiSaputro9 ай бұрын

  • excellent. thx!

    @MannyBernabe@MannyBernabe2 ай бұрын

  • How would I go about weighing the results by other meta data? Say I have a bunch of videos, and I'm searching the title/description, but want to give some amount of preference to newer videos too.

    @joostschuur@joostschuur9 ай бұрын

  • thank you!

    @KaptainLuis@KaptainLuis3 ай бұрын

  • Great work! How do you make these nice presentations with the fancy arrows?

    @atursams5501@atursams55015 ай бұрын

  • Loved it

    @mahanteshg609@mahanteshg6092 ай бұрын

  • Great work presenting this! Do you happen to know how similar or different this is from what Elasticsearch does when performing full-text search?

    @matickovac@matickovac4 күн бұрын

  • thank you so much

    @diamond2869@diamond28693 ай бұрын

  • How in the world did it get 0.74 score (which pretty high on the scale for 0 to 1!) for the similarity of "Hello Earth" and "OpenAI vectors and embeddings are easy"? Is there anything in common between the two?

    @meirgoldenberg5638@meirgoldenberg56388 ай бұрын

  • I get this message when I run the API. Do you need to pay OpenAI for it to work? Thanks! "error": {"message": "You exceeded your current quota, please check your plan and billing details.",

    @GenZManhood@GenZManhood9 ай бұрын

    • Me too, you found a solution?

      @user-xx8xg5yf4d@user-xx8xg5yf4d8 ай бұрын

  • Thank you @Ardian

    @veeresh5341@veeresh53412 ай бұрын

  • I like this video and I don't mind all the upselling. My only complaint is that if I pause the video for too long, it automatically sends me to another video in the series, which makes it hard to get back to where I was. You might assume it is user error, but it isn't. The automatic transferal and loss of context happens constantly with this KZhead video, and I've never had the problem with any other KZhead tutorial. I'm fine with the monetizing and upselling since it helps reward the content creator, I just wish it wouldn't keep making me lose my place in the tutorial.

    @curtisblake261@curtisblake2614 ай бұрын

  • The best!

    @KnowledgeKingVideos@KnowledgeKingVideos4 ай бұрын

  • I loved your video about ChatGPT Clone! Could you please make a new video to further improve the clone? For example, you could demonstrate how to make the website width responsive to work on all devices. Additionally, it would be great if you could show how to add some new features, such as a cool typewriter effect for the ChatGPT responses. Another useful addition would be to highlight the code generated by ChatGPT using a React syntax highlighter. This can be achieved by enclosing the code between the tags ``` and ```. I actually made a ChatGPT Clone and published it. . I added new features to my web app using your videos. Thank you very much!

    @Adam9174X@Adam9174X9 ай бұрын

  • thanks for the tutorial can we use our own LLM like private GPT or Text-generation Web UI instead of OPENAI

    @noubgaemer1044@noubgaemer10443 ай бұрын

  • This is very useful. Could you also do embedings of CSV files? I have files amounting up to 5 million rows

    @cliffordmwale5782@cliffordmwale57825 ай бұрын

  • Does the chuck size have an affect on the quality or accuracy of the search result? Let's say I split a document into words AND in 200 word chucks. The vector results are stored in a vector db.

    @BryanChance@BryanChance4 ай бұрын

  • Hi Bro, What is the extension u used in the vs code for the code suggestions?

    @sivakumarkalaiselvan6831@sivakumarkalaiselvan68315 ай бұрын

  • yes but how do you save a vector store ? ie export it to json for upload or finetuning into the main lm ?

    @xspydazx@xspydazx3 күн бұрын

  • would it be possible to use this for an AI NPC for training purposes in XR space for example?

    @zibitappert@zibitappert9 ай бұрын

  • isnt calculating the modulus of the subtraction of the vectors a more accurate way to find similarities?

    @MRGCProductions20996@MRGCProductions209962 ай бұрын

  • Blows my mind you've spent 18m talking about the How and 30sec about the why and what.

    @edoson01@edoson019 ай бұрын

  • Loved this tutorial Adrian, very straight forward and it worked the first time not like some others I've tried. Now for my question. I'm seeing this on February 2024. I did not know CHATGPT, BARD and those other AI apps until they hit the common pool that I must swim in. I take it that vectoring documents has been going on for awhile, outside of the math world. I knew of vectoring back from college in linear algebra. If this is the case, what I'm trying to do will not be new. I'm trying to vectorize my documents in order to practice doing this kind of work. So, are there IT companies out there doing this type of work already and can you name a few? How far have they gotten? Has someone already done the library of Congress for instance?

    @gman2036@gman20362 ай бұрын

  • wow great video sir. Helped a lot. may i know what extension is being used in 16:40 ?

    @pranavkm4513@pranavkm451310 ай бұрын

    • GitHub Copilot

      @AdrianTwarog@AdrianTwarog9 ай бұрын

  • Bought the book. It ended on page 54, is there anything after 54 to 58? Last example was open ai fine tuning. It leaves the ft up on open ai site. How long will it be available there? Can it be brought down locally and be used in the future as local in combination with cloud model?

    @fkxfkx@fkxfkx10 ай бұрын

    • I’ll double check, and any updates will automatically be enabled on Gumroad!

      @AdrianTwarog@AdrianTwarog10 ай бұрын

    • ​@@AdrianTwaroghow to automate text importation with sql? Must one enter each text blob manually?

      @Ricocase@Ricocase8 ай бұрын

  • Hey man, what are those fonts you've used in this video?

    @psyduck4763@psyduck47635 ай бұрын

  • Good

    @kareemrasheedbabatunde9026@kareemrasheedbabatunde90269 ай бұрын

  • Sir please can you read create the boostrap navigation bar that have off-canvas customize with nice header logo and home

    @Kennethlumor@Kennethlumor9 ай бұрын

  • OK, it is a good video on using OpenAI to create embedding via an API. But lets say, next week. Open AI's building are destroyed by a meteor! Now, I still want to create embeddings on my dev server. Is there a software I can download and run locally that I can use until a meteor crashed on my house?

    @paulparadise6059@paulparadise60598 ай бұрын

  • Does your book explain what kind of database that Single Store is using, and what their proprietary function "json_array_pack" is actually doing? I would like to try this out but don't want to use a service like Single Store as I prefer to just use an AWS RDS database instead (or anything that I can have full control over, or even host myself), but that function is not a real SQL function and I don't understand exactly what is going on in the background here. It looks like the Single Store function is doing some kind of encrypting of the json array data, but it's unclear in their documentation. On a different note, is converting text to vector data more efficient and more accurate for searching than something like a simple SQL "like" query, or fulltext searching? How about compared to something like SphinxSearch or AWS Elastisearch?

    @KlrStng@KlrStng9 ай бұрын

  • A great taste of Sweat Content.

    @ChandanKumar-of1je@ChandanKumar-of1jeАй бұрын

    • Nice and ...salty?

      @joeyc666@joeyc6662 күн бұрын

  • how does SingleStore know the embeddings returned from OpenAI and searches it correctly in its vector db?

    @pazhani008@pazhani008Ай бұрын

  • best

    @ripon59@ripon592 ай бұрын

  • What is the quickest way to feed recognition or pattern braking data into the system?. Or just lower the AI endorphine levels hahaha.

    @oraculox@oraculox4 ай бұрын
KZhead