Guard DeepSpeed imports #37755
Guard DeepSpeed imports #37755
Conversation
|
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the |
| @@ -2702,6 +2701,8 @@ def resize_token_embeddings( | |||
| # Since we are basically reusing the same old embeddings with new weight values, gathering is required | |||
| is_quantized = hasattr(self, "hf_quantizer") and self.hf_quantizer is not None | |||
| if is_deepspeed_zero3_enabled() and not is_quantized: | |||
There was a problem hiding this comment.
Here I am assuming that we do not need an additional is_deepspeed_available() check, but let me know if we do!
There was a problem hiding this comment.
Yeah we don't need one ! if is_deepspeed_zero3_enabled returns True, deepspeed will be installed
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| @@ -2702,6 +2701,8 @@ def resize_token_embeddings( | |||
| # Since we are basically reusing the same old embeddings with new weight values, gathering is required | |||
| is_quantized = hasattr(self, "hf_quantizer") and self.hf_quantizer is not None | |||
| if is_deepspeed_zero3_enabled() and not is_quantized: | |||
There was a problem hiding this comment.
Yeah we don't need one ! if is_deepspeed_zero3_enabled returns True, deepspeed will be installed
What does this PR do?
This PR follows a similar approach as the one taken in
accelerate(huggingface/accelerate#1963) to guard the DeepSpeed imports so that we only callimport deepspeedwhen it's needed. The rationale for this is that whenever we runimport deepspeed, CUDA is initialised which is (a) slow and (b) interferes with the vllm server when DP=8 and TP=1 (see vllm-project/vllm#17079)Related to huggingface/trl#3351
Gently pinging @SunMarc for review 🤗
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.