Solving My Hacker News Problem with AI (Full Code)
2024 ж. 20 Мам.
3 496 Рет қаралды
Solving My Hacker News Problem with AI (Full Code)
👊 Become a member and get access to GitHub and Code:
/ allaboutai
GitHub:
github.com/AllAboutAI-YT/hack...
🤖 Great AI Engineer Course:
scrimba.com/learn/aiengineer?...
📧 Join the newsletter:
www.allabtai.com/newsletter/
🌐 My website:
www.allabtai.com
Open Source Projects:
huggingface.co/sentence-trans...
In this video I create python code to search HackerNews using semantic search with embeddings. Great way to collect data for you own use cases using RAG and LLMs, pick between a local model or a bigger closed model.
00:00 HackerNews AI Search Intro
02:37 Flowchart
03:55 Hacker News LLM Search Code
05:55 Tests
09:41 Conclusion
great stuff, need more of this. Keep up the good work!
Very good project!
Good one, when have time i will try this too
You picked a great topic that time and killed it. A wonderful use case to be reused in the use case to be reused in the use case haha. 😮
Thx
Looking forward to grabbing it from the pøblic reepo.
Do u get it? If u get pls reply and also the speech to speech ready also pls help
I think HackerNews has an RSS feed, so there is no need to scrape.
nice:)
* (00:00) Introduction to Hacker News and the problem of finding specific information * (00:28) Creating a script to scrape Hacker News posts and comments * (00:58) Embedding the data into a semantic search system * (01:32) Searching for sentiment analysis on a specific post * (02:12) Running the Hau model and local model through AMA * (02:42) Overview of the process from scraping to embeddings * (03:18) Using the embeddings to answer user questions * (03:54) Code overview and setup * (04:30) Getting information from Hacker News and adjusting the system * (05:03) Using the sentence transformer and adjusting the system message * (05:37) Chunk sizes and the Vault file * (06:08) Testing the system with a "needle in a haystack" search * (06:41) Finding a specific comment and getting more information * (07:16) Searching for posts with specific keywords * (07:56) Analyzing the data with Claude 3 Sonnet * (08:25) Running the program and getting the results * (09:03) Results: topic, URL, most emotional comments, and KZhead video ID * (09:43) Early stages of the project and future updates * (10:16) Conclusion and public repo **Key Takeaways** --------------- 🔍 Hacker News can be overwhelming, but AI can help find specific information 💻 Scripting and scraping can gather data, which can be embedded for semantic search 🤔 Sentiment analysis and user questions can be answered with context 📊 Code is available and adjustable for different sources and needs 🔮 Claude 3 Sonnet can analyze data for key topics and emotional responses **Benefits** ------------ ✔ Solves the problem of finding specific information on Hacker News ✔ Uses AI for semantic search and sentiment analysis ✔ Adjustable code for different sources and needs ✔ Analyzes data for key topics and emotional responses **Cons** ---------- ❌ Early stages of the project, needs more work ❌ Prompt and desired output need refinement ❌ Limited to Hacker News, could be expanded to other sources **Score** --------- - 8/10 for solving a specific problem with AI - 7/10 for code and adjustability - 6/10 for early stages and room for improvement **Like or Dislike** ------------------- 👍 Like for the creative solution and potential for expansion and improvement.
It’s not entirely clear what exactly the script is doing here? Is it looking for some facts in all the comments?
Love the energy 👏👏