🤖 [INFO]: Minimum Hardware / Working Setups ⚡

[trholding]

Run grok-1 with less than 🔲 420 G VRAM

See: llama2.cpp / grok-1 support
@ibab_ml on X

What are some of the working setups?

llama2.cpp:

Mac

• Mac Studio with an M2 Ultra
• Total RAM: 192GB (unified RAM)

ggml-org/llama.cpp#6204 (comment)

AMD

• CPU: Threadripper 3955WX
• Total RAM: 256GB RAM
• TPS: 0.5 tokens per second

ggml-org/llama.cpp#6204 (comment)

This repo:

Intel + Nvidia

• GPU: 8 x A100 80G
• Total VRAM: 640G
• CPU: 2 x Xeon 8480+
• RAM: 1.5 TB

#168 (comment)

AMD

• GPU: 8 x Instinct MI300X GPU 190G
• Total VRAM: 1520G

https://github.com/xai-org/grok-1/issues/130#issuecomment-2005770022

Other / Container / Cloud

• GPU: 8 x A100 80G
• Total VRAM: 640G
• K8 cluster

https://github.com/xai-org/grok-1/issues/6#issuecomment-2007301554

See:
https://github.com/xai-org/grok-1/issues/42
https://github.com/xai-org/grok-1/issues/130#issuecomment-2004399998
https://github.com/xai-org/grok-1/issues/172#issuecomment-2005593309

[AndreSlavescu]

Made a slight edit to my original comment to eliminate ambiguity and confusion. H100GB should be H100 80GB.

[killatapez]

And what exactly happens when you try to launch this on some average PC config, lets say 3060ti/16gb RAM?

[fredex2]

[AndreSlavescu]

it will just throw a CUDA OOM error because it cannot load that much data on 16GB of VRAM

[jihuishan]

Thanks for sharing! So is it possible to do a little bit modification to run on 2 nodes with 4 A100 80g on each node?

[waxiliwwww]

Hello, have you manage to do that?

[jihuishan]

NO....