-
@ 2e8970de:63345c7a
2025-02-10 11:13:18
I feel like we're stuck in the 16GB or let's say the broad "8GB to 32GB" range forever now.
Do you remember the times when computer performance and all memory and drive sizes doubled and doubled and doubled?
Now we live in a time when we all not only want to play games or the time when locally playing 4k video was demanding - we live in a time where we want to run local LLMs offline and reder 3D stuff or even have VR goggles with more than 8K resolution.
Remember the Radeon 6000 Series from 2020 - HALF A DECADE AGO - buy a Radeon 6800 with 16GB.
# Current state of the industry
- Nvidia 5070 comes with only 12 GB of VRAM. Why.
- Nvidia 5080 should be the real flagship but it comes with only 16GB. Why. You can run an 8b LLM model on this at best.
- Nvidia 5090 ist the only product here that gives 32GB. Congratulations, might as well buy 2 cards. This should be the VRAM the 5070 has
- Perfect opportunity for AMD to be a tough competitor, right? Wrong. The new AMD Radeon 9000 series gives you 16GB with the 9070 and 9070xt
- Apple is technologically perfectly positioned to come out of nowhere and blow the industry out of the water with their unified memory architecture. And to some extent this is true. If you buy a Macbook pro and pay the egregiously priced upgrade to 32GB (which is less than 2 GPUs à 16GB due to Unified Memory also using normal RAM stuff) you'll get a whole computer for what Nvidia/AMD sell the cards alone.
- Nvidia Digits looks similar to Apples MacMini. The unified memory architecture makes this a better deal. But still disappointing imo.
- If tech doubled and doubled like back in the day I would have expected 64GB everywhere and maybe even 128GB for 5090 level products. I'm serious.
# Conclusion: What to buy for local LLM (imo)
1. Don't
2. If you already got an older 16GB graphics card don't upgrade to the overpriced new ones. You'll be able to run the same sized small models, just slower.
3. If you're in the market for a whole new computer anyway - upgrading a MacBook to 32GB
4. You're crazy and like to tinker? Maybe research how to get a dozen RTX 3080 cards or AMD 6800 cards to run together.
originally posted at https://stacker.news/items/881359