VRAM Table

[frenzybiscuit]

Can you post a VRAM table for what's required to quant models?

On AutoAWQ I used to be able to quant 70B AWQ models with 2x3090 and 128GB RAM.

I have since upgraded to 3x3090 on the same system and llmcompressor is still OOM...

This isn't really a support request, so much as requesting more documentation on what is required to quant models.

[dsikka]

Hi @frenzybiscuit - we do have some documentation that you can refer to here: https://docs.vllm.ai/projects/llm-compressor/en/latest/getting-started/compress/#things-to-note-when-calculating-memory-requirements-for-llm-compressor

[frenzybiscuit]

Great, could you include documentation on how to make it use more then one GPU? Because I have three 3090 and its just using one.

[brian-dellabetta]

Hi @frenzybiscuit , AWQ currently only uses one GPU, onloading a single layer at a time (AutoAWQ only ever ran smoothing on one layer and one device at a time as well, even though weights were loaded across multiple devices). We also cache all the activations for each layer we smooth, whereas AutoAWQ uses the same cache (obtained on the first layer) throughout (more detail here). This is a design choice we made that increases memory requirements relative to AutoAWQ, for the sake of correctness.

Setting offload_device=torch.device("cpu") should reduce much of that burden on GPU VRAM. Is that still failing for you? We have discussed adding a toggle to use AutoAWQ's caching logic because of OOM user reports like this. But this is why calculating memory requirements is troublesome for AWQ.

[frenzybiscuit]

Okay, well, I'm continuing to crash when quanting a 70B model on a 128GB RAM system with 3x3090.

If I understand the guide right, I need more then 128GB system RAM for 70B. Would upgrading to 192GB be enough?

I set NUM_CALIBRATION_SAMPLES = 128 which allows me to get up to step.... I think it was 60 of 70 or something? Then it crashed from OOM. Took like 2.5 hours to crash.

[brian-dellabetta]

Hi @frenzybiscuit , please provide the script you are running so i can investigate.

[frenzybiscuit]

quant.py

attached

[brian-dellabetta]

Thanks, I'll try to take a look tomorrow

[brian-dellabetta]

Hi @frenzybiscuit , this is a model in your local drive. Is there a version on huggingface I could try against? Also, I am just now realizing you are hitting OOM on host memory, not device. This is likely due to an issue we are aware of with a memory leak in the pipelining system, and not AWQ. But it will cause host OOM issues for large models.

• DeepSeekv3 CPU usage increases over time #1650

We are still investigating the source of this memory leak, but have had other priorities this month. Deifinitely hope to resolve soon, see this user comment for AWQ

[frenzybiscuit]

Sure, it’s a roleplay model based on the wings of fire universe. Here you go:

https://huggingface.co/Darkhn/L3.3-Animus-V10.0?not-for-all-audiences=true

A memory leak does make sense as the ram increased towards the end. Hopefully that gets fixed.