Groq API - 500+ Tokens/s - First Impression and Tests - WOW!
2024 ж. 20 Мам.
18 315 Рет қаралды
Groq API - 500+ Tokens/s - First Impression and Tests - WOW!
👊 Become a member and get access to GitHub:
/ allaboutai
🤖 AI Engineer Course:
scrimba.com/?ref=allabtai
📧 Join the newsletter:
www.allabtai.com/newsletter/
🌐 My website:
www.allabtai.com
Groq:
www.groq.com
In this video I give my first impression and to multiple test on the Groq API. Like real time speech to speech and comparing Groq to ChatGPT. A new chip design for running inference on AI apps like LLMs. Thanks to Groq for giving me early access!
00:00 Groq API Intro
00:45 Groq LPU
01:45 Groq Real Time Speech to Speech Test
06:07 Groq vs ChatGPT Test
09:33 Groq Chain Prompting Test
11:15 Conclusion
Thanks for the demos. We love what you're doing.
I have joined your membership. Now I just need to get the groq API application approved. Thank you again Kris for sharing.
You had to apply?
At first need to apply for groq API, but now don't need it.@@cjenkinsiv
This guy is too smart for my brain to process
Been testing this out online and been getting around 400 to 550 tokens on it, it's crazy fast. There are only two models it lets you select, but both are big models and run lightning fast compared to any other A.I. model I've seen online and locally.
is there a difference between the two model though ? When you test them via groq ?
It's super fast but they don't follow long-form instructions as well as ChatGPT.
yes i did enjoy it, thanks for the video, keep it up ❤
That was amazing!
How expensive is that kind of voice conversation using Groq API?
SO GOOD !
Damn cool stuff.
What matters most is the first token latency, question is does grow has edje on that?
I was wondering if you know anything about airllm? I read that this inference is capable of loading a 70B model on gpus as small as 4gb, but I don't saw no one speaking about that
Here's a long wall of text about the thing you're interested in. Now I think this is very interesting myself, what I understand is that it allows llms to be used on lower end hardware or something? I used AI to do this. Understanding AirLLM and Its Significance AirLLM is a groundbreaking technique that facilitates running a 70-billion-parameter large language model (LLM) on a single 4GB GPU, overcoming traditional hardware limitations. Traditionally, such large models would require more powerful and expensive hardware, restricting accessibility. **What is AirLLM?** AirLLM stands for a method allowing large language models to operate on smaller, less powerful GPUs. It employs two main techniques: 1. **Layer-wise Inference:** Breaks down the model into individual layers, loading only the necessary layers into memory during inference, reducing the overall memory footprint. 2. **Flash Attention:** An optimization within layer-wise inference, focusing on loading and executing only one layer at a time, further minimizing memory requirements. **Why is AirLLM Significant?** The significance lies in democratizing the use of large language models, making it accessible for personal projects, educational purposes, and small businesses. Practical implications include the ability to run advanced AI models, like chatbots, on smaller hardware, fostering innovation in various fields. **Running 70B LLM on a 4GB GPU with AirLLM Technique** **What it is:** AirLLM is a groundbreaking technique enabling the operation of a 70-billion-parameter large language model (LLM) on a single 4GB GPU, overcoming traditional hardware limitations. **How it works:** The technique employs layer-wise inference, dividing the model into layers and loading only the necessary parts into memory, drastically reducing memory requirements. **Key Features:** - **Layer-wise Inference:** Splits the model into layers, significantly cutting down on GPU memory usage. - **Flash Attention:** Optimizes single-layer execution, further reducing memory needs per layer. **Benefits:** Allows for efficient and effective inference on a 4GB GPU without compromising model performance, bypassing the need for high-end GPUs or excessive RAM. **Applications:** Ideal for scenarios where hardware resources are limited, such as personal projects or low-budget research. This technique represents a significant leap in making advanced AI models more accessible, enabling users with modest hardware to leverage the power of large LLMs for various applications.
A video from you about Avatar AI would be awesome! Haven’t found one yet
Groq api I have been using to use the mistral 8x7 and it is currently letting me use it for free , but until when it is free any idea ?
you didnt show how to set this up i see the site now what?
I need this kind of inference speed in Skyrim with GPT :D
lol nailed
You wouldn't believe, I was experimenting with Siri and groq api, and I asked the same question on summarizing the "attention is all you need" paper even before coming across this video. I mean I am spooked here, what are the odds of that happening? We humans obviously do behave in patterns. lol
It's amazingly, stupendously and miraculously FAST, isn't it, haha
What is price?
How to use JSON mode with the model?
The code is available in the playground.
amazin video but can you link rep?
Where the membership link? You just provide the like to your KZhead Chanel.
so 5x faster than openAI but the card cannot be used to train your own models...
Its actually depressing that hardware made for int math wasnt made already.
nvidia killer. im sure they're also working on a training dedicated card... and if not them, someone else. a lot of companies are going to try to eat nvidias 2T dollar lunch.
nvidia killer is a company that will beat them in: 1. watt/token 2. $/token 3. first token latency 4. total generation time so far i see 4 was beaten, but it's the least important aspect.
257 LPU vs 1 H100? do the math and watch your wallet
Not too late for Meta to cancel its order from Nvidia 😅😅😅
We should start using AI for something that is more useful than playing games and generating p0rnographic images. Its not a toy.
Groq, so unfriendly about giving access...
How? I haven't tried out their API or service
What do you mean?