cjw/amd-r9700-vllm-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a04b9ef9c5807c3937429e93fd2166aebb347518
amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_amd-r9700-rocm_atten/RedHatAI_Qwen3-14B-FP8-dynamic_tp2_server.log
T

433 lines
60 KiB
Plaintext
Raw Normal View History

x
2026-03-26 20:50:00 +08:00
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 17:09:03 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=75372) INFO 12-19 17:09:03 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=75372) INFO 12-19 17:09:03 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 32000, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
(APIServer pid=75372) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=75372) INFO 12-19 17:09:08 [model.py:514] Resolved architecture: Qwen3ForCausalLM
(APIServer pid=75372) INFO 12-19 17:09:08 [model.py:1636] Using max model len 32000
(APIServer pid=75372) INFO 12-19 17:09:08 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=75534) INFO 12-19 17:09:12 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='RedHatAI/Qwen3-14B-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/Qwen3-14B-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/Qwen3-14B-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=75534) WARNING 12-19 17:09:12 [multiproc_executor.py:884] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
INFO 12-19 17:09:15 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:35477 backend=nccl
INFO 12-19 17:09:15 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:35477 backend=nccl
INFO 12-19 17:09:16 [pynccl.py:111] vLLM is using nccl==2.27.3
INFO 12-19 17:09:16 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
INFO 12-19 17:09:16 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
(Worker_TP0 pid=75616) INFO 12-19 17:09:17 [gpu_model_runner.py:3562] Starting to load model RedHatAI/Qwen3-14B-FP8-dynamic...
(Worker_TP1 pid=75617) INFO 12-19 17:09:17 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP0 pid=75616) INFO 12-19 17:09:17 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP0 pid=75616)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(Worker_TP0 pid=75616)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 2.57it/s]
(Worker_TP0 pid=75616)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.02it/s]
(Worker_TP0 pid=75616)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:03<00:01, 1.14s/it]
(Worker_TP0 pid=75616)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.19s/it]
(Worker_TP0 pid=75616)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.09s/it]
(Worker_TP0 pid=75616)
(Worker_TP0 pid=75616) INFO 12-19 17:09:22 [default_loader.py:308] Loading weights took 4.43 seconds
(Worker_TP0 pid=75616) INFO 12-19 17:09:23 [gpu_model_runner.py:3659] Model loading took 7.8555 GiB memory and 5.221202 seconds
(Worker_TP0 pid=75616) INFO 12-19 17:09:28 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/8760cb82b9/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=75616) INFO 12-19 17:09:28 [backends.py:694] Dynamo bytecode transform time: 5.45 s
(Worker_TP1 pid=75617) INFO 12-19 17:09:31 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=75616) INFO 12-19 17:09:31 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=75616) INFO 12-19 17:09:36 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 4.77 s
(Worker_TP0 pid=75616) INFO 12-19 17:09:36 [monitor.py:34] torch.compile takes 10.22 s in total
(Worker_TP0 pid=75616) INFO 12-19 17:09:39 [gpu_worker.py:375] Available KV cache memory: 22.49 GiB
(EngineCore_DP0 pid=75534) INFO 12-19 17:09:39 [kv_cache_utils.py:1291] GPU KV cache size: 294,752 tokens
(EngineCore_DP0 pid=75534) INFO 12-19 17:09:39 [kv_cache_utils.py:1296] Maximum concurrency for 32,000 tokens per request: 9.21x
(Worker_TP0 pid=75616)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:08, 2.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:07, 2.15it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:01<00:07, 2.27it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:01<00:06, 2.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:02<00:05, 2.36it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:02<00:05, 2.39it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:02<00:04, 2.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:03<00:04, 2.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:03<00:04, 2.47it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:04<00:03, 2.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:04<00:03, 2.49it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:04<00:02, 2.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:05<00:02, 2.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:05<00:02, 2.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:06<00:01, 2.49it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:06<00:01, 2.49it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:06<00:00, 2.49it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:07<00:00, 2.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.44it/s]
(Worker_TP0 pid=75616)
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:00<00:06, 1.63it/s]
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:01<00:04, 1.99it/s]
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:01<00:03, 2.20it/s]
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:01<00:03, 2.30it/s]
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:02<00:02, 2.36it/s]
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:02<00:02, 2.40it/s]
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:03<00:01, 2.44it/s]
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:03<00:01, 2.47it/s]
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:03<00:00, 2.50it/s]
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:04<00:00, 2.51it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:04<00:00, 2.53it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:04<00:00, 2.39it/s]
(Worker_TP0 pid=75616) INFO 12-19 17:09:52 [gpu_model_runner.py:4610] Graph capturing finished in 13 secs, took 0.66 GiB
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] WorkerProc hit an exception.
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4302, in _dummy_sampler_run
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampler_output = self.sampler(
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits=logits, sampling_metadata=dummy_metadata
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 96, in forward
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampled, processed_logprobs = self.sample(logits, sampling_metadata)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 187, in sample
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] random_sampled, processed_logprobs = self.topk_topp_sampler(
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits,
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<2 lines>...
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampling_metadata.top_p,
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 104, in forward_native
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits = self.apply_top_k_top_p(logits, k, p)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 258, in apply_top_k_top_p
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits_sort, logits_idx = logits.sort(dim=-1, descending=False)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] torch.OutOfMemoryError: HIP out of memory. Tried to allocate 76.00 MiB. GPU 1 has a total capacity of 31.86 GiB of which 0 bytes is free. Of the allocated memory 30.75 GiB is allocated by PyTorch, with 68.00 MiB allocated in private pools (e.g., HIP Graphs), and 167.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 538, in compile_or_warm_up_model
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] self.model_runner._dummy_sampler_run(hidden_states=last_hidden_states)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4307, in _dummy_sampler_run
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] raise RuntimeError(
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<4 lines>...
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ) from e
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] RuntimeError: CUDA out of memory occurred when warming up sampler with 64 dummy requests. Please try lowering `max_num_seqs` or `gpu_memory_utilization` when initializing the engine.
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4302, in _dummy_sampler_run
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampler_output = self.sampler(
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits=logits, sampling_metadata=dummy_metadata
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 96, in forward
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampled, processed_logprobs = self.sample(logits, sampling_metadata)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 187, in sample
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] random_sampled, processed_logprobs = self.topk_topp_sampler(
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits,
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<2 lines>...
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampling_metadata.top_p,
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 104, in forward_native
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits = self.apply_top_k_top_p(logits, k, p)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 258, in apply_top_k_top_p
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits_sort, logits_idx = logits.sort(dim=-1, descending=False)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] torch.OutOfMemoryError: HIP out of memory. Tried to allocate 76.00 MiB. GPU 1 has a total capacity of 31.86 GiB of which 0 bytes is free. Of the allocated memory 30.75 GiB is allocated by PyTorch, with 68.00 MiB allocated in private pools (e.g., HIP Graphs), and 167.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 538, in compile_or_warm_up_model
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] self.model_runner._dummy_sampler_run(hidden_states=last_hidden_states)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4307, in _dummy_sampler_run
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] raise RuntimeError(
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<4 lines>...
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ) from e
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826] RuntimeError: CUDA out of memory occurred when warming up sampler with 64 dummy requests. Please try lowering `max_num_seqs` or `gpu_memory_utilization` when initializing the engine.
(Worker_TP1 pid=75617) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] WorkerProc hit an exception.
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4302, in _dummy_sampler_run
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampler_output = self.sampler(
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits=logits, sampling_metadata=dummy_metadata
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 96, in forward
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampled, processed_logprobs = self.sample(logits, sampling_metadata)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 187, in sample
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] random_sampled, processed_logprobs = self.topk_topp_sampler(
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits,
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<2 lines>...
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampling_metadata.top_p,
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 104, in forward_native
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits = self.apply_top_k_top_p(logits, k, p)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 258, in apply_top_k_top_p
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits_sort, logits_idx = logits.sort(dim=-1, descending=False)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] torch.OutOfMemoryError: HIP out of memory. Tried to allocate 38.00 MiB. GPU 0 has a total capacity of 31.86 GiB of which 0 bytes is free. Of the allocated memory 30.83 GiB is allocated by PyTorch, with 68.00 MiB allocated in private pools (e.g., HIP Graphs), and 169.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 538, in compile_or_warm_up_model
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] self.model_runner._dummy_sampler_run(hidden_states=last_hidden_states)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4307, in _dummy_sampler_run
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] raise RuntimeError(
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<4 lines>...
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ) from e
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] RuntimeError: CUDA out of memory occurred when warming up sampler with 64 dummy requests. Please try lowering `max_num_seqs` or `gpu_memory_utilization` when initializing the engine.
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4302, in _dummy_sampler_run
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampler_output = self.sampler(
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits=logits, sampling_metadata=dummy_metadata
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 96, in forward
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampled, processed_logprobs = self.sample(logits, sampling_metadata)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/sampler.py", line 187, in sample
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] random_sampled, processed_logprobs = self.topk_topp_sampler(
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits,
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<2 lines>...
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] sampling_metadata.top_p,
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] )
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 104, in forward_native
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits = self.apply_top_k_top_p(logits, k, p)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 258, in apply_top_k_top_p
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] logits_sort, logits_idx = logits.sort(dim=-1, descending=False)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] torch.OutOfMemoryError: HIP out of memory. Tried to allocate 38.00 MiB. GPU 0 has a total capacity of 31.86 GiB of which 0 bytes is free. Of the allocated memory 30.83 GiB is allocated by PyTorch, with 68.00 MiB allocated in private pools (e.g., HIP Graphs), and 169.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 538, in compile_or_warm_up_model
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] self.model_runner._dummy_sampler_run(hidden_states=last_hidden_states)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4307, in _dummy_sampler_run
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] raise RuntimeError(
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ...<4 lines>...
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] ) from e
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826] RuntimeError: CUDA out of memory occurred when warming up sampler with 64 dummy requests. Please try lowering `max_num_seqs` or `gpu_memory_utilization` when initializing the engine.
(Worker_TP0 pid=75616) ERROR 12-19 17:09:52 [multiproc_executor.py:826]
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] EngineCore failed to start.
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] Traceback (most recent call last):
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] super().__init__(
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] vllm_config, executor_class, log_stats, executor_fail_callback
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] )
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ~~~~~~~~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] vllm_config
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ^^^^^^^^^^^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] )
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 256, in _initialize_kv_caches
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/abstract.py", line 116, in initialize_from_config
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 361, in collective_rpc
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] return aggregate(get_response())
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ~~~~~~~~~~~~^^
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 344, in get_response
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] raise RuntimeError(
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] ...<2 lines>...
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] )
(EngineCore_DP0 pid=75534) ERROR 12-19 17:09:52 [core.py:866] RuntimeError: Worker failed with error 'CUDA out of memory occurred when warming up sampler with 64 dummy requests. Please try lowering `max_num_seqs` or `gpu_memory_utilization` when initializing the engine.', please check the stack trace above for the root cause
(EngineCore_DP0 pid=75534) Process EngineCore_DP0:
(EngineCore_DP0 pid=75534) Traceback (most recent call last):
(EngineCore_DP0 pid=75534) File "/usr/lib64/python3.13/multiprocessing/process.py", line 313, in _bootstrap
(EngineCore_DP0 pid=75534) self.run()
(EngineCore_DP0 pid=75534) ~~~~~~~~^^
(EngineCore_DP0 pid=75534) File "/usr/lib64/python3.13/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=75534) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=75534) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 870, in run_engine_core
(EngineCore_DP0 pid=75534) raise e
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=75534) engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=75534) super().__init__(
(EngineCore_DP0 pid=75534) ~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=75534) vllm_config, executor_class, log_stats, executor_fail_callback
(EngineCore_DP0 pid=75534) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=75534) )
(EngineCore_DP0 pid=75534) ^
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=75534) num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=75534) ~~~~~~~~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=75534) vllm_config
(EngineCore_DP0 pid=75534) ^^^^^^^^^^^
(EngineCore_DP0 pid=75534) )
(EngineCore_DP0 pid=75534) ^
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 256, in _initialize_kv_caches
(EngineCore_DP0 pid=75534) self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=75534) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/abstract.py", line 116, in initialize_from_config
(EngineCore_DP0 pid=75534) self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=75534) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 361, in collective_rpc
(EngineCore_DP0 pid=75534) return aggregate(get_response())
(EngineCore_DP0 pid=75534) ~~~~~~~~~~~~^^
(EngineCore_DP0 pid=75534) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 344, in get_response
(EngineCore_DP0 pid=75534) raise RuntimeError(
(EngineCore_DP0 pid=75534) ...<2 lines>...
(EngineCore_DP0 pid=75534) )
(EngineCore_DP0 pid=75534) RuntimeError: Worker failed with error 'CUDA out of memory occurred when warming up sampler with 64 dummy requests. Please try lowering `max_num_seqs` or `gpu_memory_utilization` when initializing the engine.', please check the stack trace above for the root cause
(Worker_TP1 pid=75617) INFO 12-19 17:09:52 [multiproc_executor.py:711] Parent process exited, terminating worker
(Worker_TP0 pid=75616) INFO 12-19 17:09:52 [multiproc_executor.py:711] Parent process exited, terminating worker
(APIServer pid=75372) Traceback (most recent call last):
(APIServer pid=75372) File "/opt/venv/bin/vllm", line 7, in <module>
(APIServer pid=75372) sys.exit(main())
(APIServer pid=75372) ~~~~^^
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=75372) args.dispatch_function(args)
(APIServer pid=75372) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/cli/serve.py", line 60, in cmd
(APIServer pid=75372) uvloop.run(run_server(args))
(APIServer pid=75372) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=75372) return __asyncio.run(
(APIServer pid=75372) ~~~~~~~~~~~~~^
(APIServer pid=75372) wrapper(),
(APIServer pid=75372) ^^^^^^^^^^
(APIServer pid=75372) ...<2 lines>...
(APIServer pid=75372) **run_kwargs
(APIServer pid=75372) ^^^^^^^^^^^^
(APIServer pid=75372) )
(APIServer pid=75372) ^
(APIServer pid=75372) File "/usr/lib64/python3.13/asyncio/runners.py", line 195, in run
(APIServer pid=75372) return runner.run(main)
(APIServer pid=75372) ~~~~~~~~~~^^^^^^
(APIServer pid=75372) File "/usr/lib64/python3.13/asyncio/runners.py", line 118, in run
(APIServer pid=75372) return self._loop.run_until_complete(task)
(APIServer pid=75372) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=75372) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=75372) return await main
(APIServer pid=75372) ^^^^^^^^^^
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 1398, in run_server
(APIServer pid=75372) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 1417, in run_server_worker
(APIServer pid=75372) async with build_async_engine_client(
(APIServer pid=75372) ~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=75372) args,
(APIServer pid=75372) ^^^^^
(APIServer pid=75372) client_config=client_config,
(APIServer pid=75372) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=75372) ) as engine_client:
(APIServer pid=75372) ^
(APIServer pid=75372) File "/usr/lib64/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=75372) return await anext(self.gen)
(APIServer pid=75372) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 172, in build_async_engine_client
(APIServer pid=75372) async with build_async_engine_client_from_engine_args(
(APIServer pid=75372) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=75372) engine_args,
(APIServer pid=75372) ^^^^^^^^^^^^
(APIServer pid=75372) ...<2 lines>...
(APIServer pid=75372) client_config=client_config,
(APIServer pid=75372) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=75372) ) as engine:
(APIServer pid=75372) ^
(APIServer pid=75372) File "/usr/lib64/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=75372) return await anext(self.gen)
(APIServer pid=75372) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 213, in build_async_engine_client_from_engine_args
(APIServer pid=75372) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=75372) vllm_config=vllm_config,
(APIServer pid=75372) ...<6 lines>...
(APIServer pid=75372) client_index=client_index,
(APIServer pid=75372) )
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/async_llm.py", line 215, in from_vllm_config
(APIServer pid=75372) return cls(
(APIServer pid=75372) vllm_config=vllm_config,
(APIServer pid=75372) ...<9 lines>...
(APIServer pid=75372) client_index=client_index,
(APIServer pid=75372) )
(APIServer pid=75372) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/async_llm.py", line 134, in __init__
(APIServer pid=75372) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=75372) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Reference in New Issue Copy Permalink