Ask HN: Workstation/rig config for local LLM finetuning/training and inference?
I need help purchasing/putting together a rig that's powerful enough for training LLMs from scratch, finetuning models, and inferencing them.
Many people on this sub showcase their impressive GPU clusters, often usnig 3090/4090. But I need more than that—essentially the higher the VRAM, the better.
Here's some options that have been announced, please tell me your recommendation even if it's not one of these:
- Nvidia DGX Station
- Dell Pro Max with GB300 (Lenovo and HP offer similar products)
The above are not available yet, but it's okay, I'll need this rig by August.
Some people suggest AMD's MI300x or MI210. MI300x comes only in x8 boxes, otherwise it's an attractive offer!
Budget ≤ $30,000