You can now run 70B model on a single 4GB GPU and it even scales up to the colossal Llama 3.1 405B on just 8GB of VRAM. AirLLM uses “Layer-wise Inference.” Instead of loading the whole model, it loads, computes, and flushes one layer at a time. → No quantization needed by default → Supports Llama, Qwen, and Mistral → Works on Linux, Windows, and macOS 100% 100% Open Source.

How many tokens per second?

Finally someone invented this. Loading the whole thing into the VRAM was dumb af from the beggining. Hopefully in a near future we'll be able to run LLMs like Kimi 2.5 or Deep Seek R1 on normal hardware and only requiring storage space.

https://medium.com/@zhamdi/running-airllm-locally-on-apple-silicon-not-so-good-2b48d41cdb7c

The reason we want intelligent models is to do complicated stuffs & save time... If it's slow it defeats the purpose.

But any innovation towards improvement is always interesting. The hardware manufacturers might even consider it to cater for certain innovations they see as feasible.

Tps on the floor but still, if that means good quality responses maybe it’s a good trade

Is this real?

Quickquant then airllm op maybe now Ram prices go down maybe?

Isnt this hacked

I believe you're talking about litellm, not this one

the tps?

Thanks