Skip to content

Commit bbb0774

Browse files
committed
fix bug: dp+tp warmup
1 parent 4045ba8 commit bbb0774

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

lmdeploy/pytorch/engine/model_agent.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -401,6 +401,7 @@ def warmup(self):
401401
is_decoding=False,
402402
device='cuda',
403403
vocab_size=self.model_config.vocab_size)
404+
inputs.build_dp_meta()
404405
self._forward_impl(inputs)
405406

406407
# warmup decoding(with cuda graph)
@@ -411,6 +412,7 @@ def warmup(self):
411412
is_decoding=True,
412413
device='cuda',
413414
vocab_size=self.model_config.vocab_size)
415+
inputs.build_dp_meta()
414416
self._forward_impl(inputs)
415417

416418
def _slice_outs(self, inputs: torch.Tensor, seq_length: torch.LongTensor):

0 commit comments

Comments
 (0)