[Misc]: When using lossy optimization, how to explain that the loss caused by optimization is within the acceptable range?

[nodekey777]

Anything you want to discuss about vllm.

I’ve noticed that with each version upgrade of vllm, there seems to be some degree of precision loss. How do you determine whether these losses are within an acceptable range?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

May I ask which model you're talking about?

For most models, vllm has regression tests for each merge to ensure the accuracy of the results.

[nodekey777]

@noooop what about llama 120b?
I’ve already reviewed your test scripts, but I didn’t find any larger-scale models.
I think the larger the model, the greater the precision error, because precision errors tend to accumulate layer by layer.
So, what should you do about these larger models? Or does testing only these smaller models sufficiently demonstrate that the precision errors are within expectations?

[noooop]

@noooop what about llama 120b?

In fact, the bigger the model, the more stable it is.

Small models need to use float32 to pass some rigorous tests, while large models can pass stably even with fp8.

refer to

Low-Bit Quantization Favors Undertrained LLMs

---

Mainly, as long as the implementation of the layer is correct, the numerical stability of the results should not make much difference

---

If you upgrade from flash attention 2 to flash attention 3 there will be some degradation in numerical stability.

[nodekey777]

How about MoE model? Does it also follow this pattern?

[nodekey777]

How about MoE model? Does it also follow this pattern?

@noooop Do you have any experience with the MOE model? It’s really important to me. Thanks.

[noooop]

@noooop Do you have any experience with the MOE model? It’s really important to me. Thanks.

Generally speaking, there are two ways to check the "correctness" of the model:

• Perplexity (aka ppl),
  Very convenient for calculation,.
  Many quantitative methods such as gguf use ppl as a metric, but ppl is not closely related to some capabilities of the model
• Downstream task testing. Many people use lm_eval to run a quick test like GSM8k to verify the correctness of the code. e.g.

lm_eval --model vllm --model_args pretrained=model_name --tasks gsm8k --num_fewshot 5 

The more downstream tasks you run, the more you can confirm the accuracy of the model.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!