amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_amd-r9700-rocm_atten/cpatonn_Qwen3-Next-80B-A3B-Instruct-AWQ-4bit_tp2_server.log

Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 17:11:45 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=76251) INFO 12-19 17:11:45 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=76251) INFO 12-19 17:11:45 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', 'trust_remote_code': True, 'max_model_len': 16384, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 32}
(APIServer pid=76251) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=76251) INFO 12-19 17:11:49 [model.py:514] Resolved architecture: Qwen3NextForCausalLM
(APIServer pid=76251) INFO 12-19 17:11:49 [model.py:1636] Using max model len 16384
(APIServer pid=76251) INFO 12-19 17:11:49 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=76251) INFO 12-19 17:11:49 [config.py:312] Disabling cascade attention since it is not supported for hybrid models.
(APIServer pid=76251) INFO 12-19 17:11:49 [config.py:439] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(APIServer pid=76251) INFO 12-19 17:11:49 [config.py:463] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=76413) INFO 12-19 17:11:53 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=76413) WARNING 12-19 17:11:53 [multiproc_executor.py:884] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
INFO 12-19 17:11:56 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:50157 backend=nccl
INFO 12-19 17:11:56 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:50157 backend=nccl
INFO 12-19 17:11:57 [pynccl.py:111] vLLM is using nccl==2.27.3
INFO 12-19 17:11:57 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
INFO 12-19 17:11:57 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [gpu_model_runner.py:3562] Starting to load model cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit...
(Worker_TP1 pid=76496) WARNING 12-19 17:11:58 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(Worker_TP1 pid=76496) INFO 12-19 17:11:58 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(Worker_TP1 pid=76496) INFO 12-19 17:11:58 [compressed_tensors_moe.py:188] Using CompressedTensorsWNA16MoEMethod
(Worker_TP0 pid=76495) WARNING 12-19 17:11:58 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [compressed_tensors_moe.py:188] Using CompressedTensorsWNA16MoEMethod
(Worker_TP1 pid=76496) INFO 12-19 17:11:58 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [rocm.py:306] Using Rocm Attention backend on V1 engine.
Loading safetensors checkpoint shards: 0% Completed | 0/10 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 10% Completed | 1/10 [00:03<00:32, 3.65s/it]
Loading safetensors checkpoint shards: 20% Completed | 2/10 [00:07<00:29, 3.64s/it]
Loading safetensors checkpoint shards: 30% Completed | 3/10 [00:11<00:26, 3.79s/it]
Loading safetensors checkpoint shards: 40% Completed | 4/10 [00:15<00:23, 3.97s/it]
Loading safetensors checkpoint shards: 50% Completed | 5/10 [00:18<00:18, 3.76s/it]
Loading safetensors checkpoint shards: 60% Completed | 6/10 [00:22<00:15, 3.85s/it]
Loading safetensors checkpoint shards: 70% Completed | 7/10 [00:26<00:11, 3.83s/it]
Loading safetensors checkpoint shards: 80% Completed | 8/10 [00:30<00:07, 3.84s/it]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:34<00:00, 2.97s/it]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:34<00:00, 3.46s/it]
(Worker_TP0 pid=76495)
(Worker_TP0 pid=76495) INFO 12-19 17:12:33 [default_loader.py:308] Loading weights took 34.64 seconds
(Worker_TP0 pid=76495) INFO 12-19 17:12:34 [gpu_model_runner.py:3659] Model loading took 23.5020 GiB memory and 35.701153 seconds
(Worker_TP0 pid=76495) INFO 12-19 17:12:38 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/671837fea6/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=76495) INFO 12-19 17:12:38 [backends.py:694] Dynamo bytecode transform time: 4.08 s
(Worker_TP1 pid=76496) INFO 12-19 17:12:40 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=76495) INFO 12-19 17:12:40 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP1 pid=76496) WARNING 12-19 17:12:41 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=512,N=256,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
(Worker_TP0 pid=76495) WARNING 12-19 17:12:41 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=512,N=256,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
(Worker_TP0 pid=76495) INFO 12-19 17:12:43 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 2.59 s
(Worker_TP0 pid=76495) INFO 12-19 17:12:43 [monitor.py:34] torch.compile takes 6.67 s in total
(Worker_TP1 pid=76496) WARNING 12-19 17:12:43 [decorators.py:528] Cannot save aot compilation to path /home/kyuz0/.cache/vllm/torch_aot_compile/8cfc5b4a85b0195581f74785744c13daeda3dd4bc8885cc84b5001d538a63819/rank_1_0/model, error:
(Worker_TP0 pid=76495) WARNING 12-19 17:12:43 [decorators.py:528] Cannot save aot compilation to path /home/kyuz0/.cache/vllm/torch_aot_compile/8cfc5b4a85b0195581f74785744c13daeda3dd4bc8885cc84b5001d538a63819/rank_0_0/model, error:
(Worker_TP0 pid=76495) INFO 12-19 17:12:44 [gpu_worker.py:375] Available KV cache memory: 7.09 GiB
(EngineCore_DP0 pid=76413) INFO 12-19 17:12:45 [kv_cache_utils.py:1291] GPU KV cache size: 154,496 tokens
(EngineCore_DP0 pid=76413) INFO 12-19 17:12:45 [kv_cache_utils.py:1296] Maximum concurrency for 16,384 tokens per request: 33.47x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:05<00:00, 1.95it/s]
Capturing CUDA graphs (decode, FULL): 0%| | 0/7 [00:00<?, ?it/s]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] WorkerProc hit an exception.
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return _semantic.arange(start, end)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise ValueError("arange's range must be a power of 2")
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._capture_cudagraphs(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] compilation_cases=compilation_cases_decode,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] uniform_decode=True,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._dummy_run(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_tokens,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<6 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] activate_lora=activate_lora,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] outputs = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids=input_ids,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] **model_kwargs,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.runnable(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] hidden_states = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] def forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.optimized_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.99", line 333, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24); getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.107", line 5, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn'); query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._op(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self.impl.forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<7 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_block_scale=output_block_scale,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] chunked_prefill_paged_decode(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] query=query[:num_actual_tokens],
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<17 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] sinks=self.sinks,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel_paged_attention_2d[
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ](
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_ptr=output,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<37 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] USE_FP8=output_scale is not None,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module_map=module_map)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] if USE_ALIBI_SLOPES:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slope = tl.load(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] # iterate through tiles
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] for j in range(0, num_blocks):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return _semantic.arange(start, end)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise ValueError("arange's range must be a power of 2")
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._capture_cudagraphs(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] compilation_cases=compilation_cases_decode,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] uniform_decode=True,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._dummy_run(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_tokens,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<6 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] activate_lora=activate_lora,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] outputs = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids=input_ids,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] **model_kwargs,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.runnable(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] hidden_states = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] def forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.optimized_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.99", line 333, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24); getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.107", line 5, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn'); query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._op(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self.impl.forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<7 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_block_scale=output_block_scale,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] chunked_prefill_paged_decode(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] query=query[:num_actual_tokens],
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<17 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] sinks=self.sinks,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel_paged_attention_2d[
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ](
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_ptr=output,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<37 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] USE_FP8=output_scale is not None,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module_map=module_map)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] if USE_ALIBI_SLOPES:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slope = tl.load(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] # iterate through tiles
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] for j in range(0, num_blocks):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] WorkerProc hit an exception.
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return _semantic.arange(start, end)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise ValueError("arange's range must be a power of 2")
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._capture_cudagraphs(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] compilation_cases=compilation_cases_decode,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] uniform_decode=True,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._dummy_run(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_tokens,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<6 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] activate_lora=activate_lora,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] outputs = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids=input_ids,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] **model_kwargs,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.runnable(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] hidden_states = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] def forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.optimized_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.99", line 333, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24); getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.107", line 5, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn'); query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._op(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self.impl.forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<7 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_block_scale=output_block_scale,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] chunked_prefill_paged_decode(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] query=query[:num_actual_tokens],
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<17 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] sinks=self.sinks,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel_paged_attention_2d[
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ](
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_ptr=output,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<37 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] USE_FP8=output_scale is not None,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module_map=module_map)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] if USE_ALIBI_SLOPES:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slope = tl.load(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] # iterate through tiles
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] for j in range(0, num_blocks):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return _semantic.arange(start, end)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise ValueError("arange's range must be a power of 2")
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output = func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._capture_cudagraphs(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] compilation_cases=compilation_cases_decode,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] uniform_decode=True,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self._dummy_run(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_tokens,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<6 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] activate_lora=activate_lora,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] outputs = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids=input_ids,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] **model_kwargs,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.runnable(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] hidden_states = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] def forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self.optimized_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.99", line 333, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24); getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "<eval_with_key>.107", line 5, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn'); query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return self._op(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self.impl.forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] self,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<7 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_block_scale=output_block_scale,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] chunked_prefill_paged_decode(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] query=query[:num_actual_tokens],
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<17 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] sinks=self.sinks,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel_paged_attention_2d[
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ](
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] output_ptr=output,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ...<37 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] USE_FP8=output_scale is not None,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] module_map=module_map)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] if USE_ALIBI_SLOPES:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slope = tl.load(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] # iterate through tiles
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] for j in range(0, num_blocks):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] EngineCore failed to start.
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] Traceback (most recent call last):
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] super().__init__(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] vllm_config, executor_class, log_stats, executor_fail_callback
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ~~~~~~~~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] vllm_config
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 256, in _initialize_kv_caches
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/abstract.py", line 116, in initialize_from_config
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 361, in collective_rpc
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] return aggregate(get_response())
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ~~~~~~~~~~~~^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 344, in get_response
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] raise RuntimeError(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ...<2 lines>...
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] RuntimeError: Worker failed with error 'at 106:17:
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] if USE_ALIBI_SLOPES:
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] alibi_slope = tl.load(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] # iterate through tiles
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] for j in range(0, num_blocks):
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] offs_n = tl.arange(0, BLOCK_SIZE)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] ^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] arange's range must be a power of 2', please check the stack trace above for the root cause
(EngineCore_DP0 pid=76413) Process EngineCore_DP0:
(EngineCore_DP0 pid=76413) Traceback (most recent call last):
(EngineCore_DP0 pid=76413) File "/usr/lib64/python3.13/multiprocessing/process.py", line 313, in _bootstrap
(EngineCore_DP0 pid=76413) self.run()
(EngineCore_DP0 pid=76413) ~~~~~~~~^^
(EngineCore_DP0 pid=76413) File "/usr/lib64/python3.13/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=76413) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=76413) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 870, in run_engine_core
(EngineCore_DP0 pid=76413) raise e
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=76413) engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=76413) super().__init__(
(EngineCore_DP0 pid=76413) ~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413) vllm_config, executor_class, log_stats, executor_fail_callback
(EngineCore_DP0 pid=76413) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) )
(EngineCore_DP0 pid=76413) ^
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=76413) num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=76413) ~~~~~~~~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413) vllm_config
(EngineCore_DP0 pid=76413) ^^^^^^^^^^^
(EngineCore_DP0 pid=76413) )
(EngineCore_DP0 pid=76413) ^
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 256, in _initialize_kv_caches
(EngineCore_DP0 pid=76413) self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=76413) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/abstract.py", line 116, in initialize_from_config
(EngineCore_DP0 pid=76413) self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=76413) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 361, in collective_rpc
(EngineCore_DP0 pid=76413) return aggregate(get_response())
(EngineCore_DP0 pid=76413) ~~~~~~~~~~~~^^
(EngineCore_DP0 pid=76413) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 344, in get_response
(EngineCore_DP0 pid=76413) raise RuntimeError(
(EngineCore_DP0 pid=76413) ...<2 lines>...
(EngineCore_DP0 pid=76413) )
(EngineCore_DP0 pid=76413) RuntimeError: Worker failed with error 'at 106:17:
(EngineCore_DP0 pid=76413) if USE_ALIBI_SLOPES:
(EngineCore_DP0 pid=76413) alibi_slope = tl.load(
(EngineCore_DP0 pid=76413) alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(EngineCore_DP0 pid=76413) )
(EngineCore_DP0 pid=76413)
(EngineCore_DP0 pid=76413) num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(EngineCore_DP0 pid=76413)
(EngineCore_DP0 pid=76413) # iterate through tiles
(EngineCore_DP0 pid=76413) for j in range(0, num_blocks):
(EngineCore_DP0 pid=76413) physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(EngineCore_DP0 pid=76413)
(EngineCore_DP0 pid=76413) offs_n = tl.arange(0, BLOCK_SIZE)
(EngineCore_DP0 pid=76413) ^
(EngineCore_DP0 pid=76413) arange's range must be a power of 2', please check the stack trace above for the root cause
(Worker_TP0 pid=76495) INFO 12-19 17:12:51 [multiproc_executor.py:711] Parent process exited, terminating worker
(Worker_TP1 pid=76496) INFO 12-19 17:12:51 [multiproc_executor.py:711] Parent process exited, terminating worker
(APIServer pid=76251) Traceback (most recent call last):
(APIServer pid=76251) File "/opt/venv/bin/vllm", line 7, in <module>
(APIServer pid=76251) sys.exit(main())
(APIServer pid=76251) ~~~~^^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=76251) args.dispatch_function(args)
(APIServer pid=76251) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/cli/serve.py", line 60, in cmd
(APIServer pid=76251) uvloop.run(run_server(args))
(APIServer pid=76251) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=76251) return __asyncio.run(
(APIServer pid=76251) ~~~~~~~~~~~~~^
(APIServer pid=76251) wrapper(),
(APIServer pid=76251) ^^^^^^^^^^
(APIServer pid=76251) ...<2 lines>...
(APIServer pid=76251) **run_kwargs
(APIServer pid=76251) ^^^^^^^^^^^^
(APIServer pid=76251) )
(APIServer pid=76251) ^
(APIServer pid=76251) File "/usr/lib64/python3.13/asyncio/runners.py", line 195, in run
(APIServer pid=76251) return runner.run(main)
(APIServer pid=76251) ~~~~~~~~~~^^^^^^
(APIServer pid=76251) File "/usr/lib64/python3.13/asyncio/runners.py", line 118, in run
(APIServer pid=76251) return self._loop.run_until_complete(task)
(APIServer pid=76251) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=76251) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=76251) return await main
(APIServer pid=76251) ^^^^^^^^^^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 1398, in run_server
(APIServer pid=76251) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 1417, in run_server_worker
(APIServer pid=76251) async with build_async_engine_client(
(APIServer pid=76251) ~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251) args,
(APIServer pid=76251) ^^^^^
(APIServer pid=76251) client_config=client_config,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) ) as engine_client:
(APIServer pid=76251) ^
(APIServer pid=76251) File "/usr/lib64/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=76251) return await anext(self.gen)
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 172, in build_async_engine_client
(APIServer pid=76251) async with build_async_engine_client_from_engine_args(
(APIServer pid=76251) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251) engine_args,
(APIServer pid=76251) ^^^^^^^^^^^^
(APIServer pid=76251) ...<2 lines>...
(APIServer pid=76251) client_config=client_config,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) ) as engine:
(APIServer pid=76251) ^
(APIServer pid=76251) File "/usr/lib64/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=76251) return await anext(self.gen)
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 213, in build_async_engine_client_from_engine_args
(APIServer pid=76251) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=76251) vllm_config=vllm_config,
(APIServer pid=76251) ...<6 lines>...
(APIServer pid=76251) client_index=client_index,
(APIServer pid=76251) )
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/async_llm.py", line 215, in from_vllm_config
(APIServer pid=76251) return cls(
(APIServer pid=76251) vllm_config=vllm_config,
(APIServer pid=76251) ...<9 lines>...
(APIServer pid=76251) client_index=client_index,
(APIServer pid=76251) )
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/async_llm.py", line 134, in __init__
(APIServer pid=76251) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=76251) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251) vllm_config=vllm_config,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) ...<4 lines>...
(APIServer pid=76251) client_index=client_index,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) )
(APIServer pid=76251) ^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
(APIServer pid=76251) return AsyncMPClient(*client_args)
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core_client.py", line 820, in __init__
(APIServer pid=76251) super().__init__(
(APIServer pid=76251) ~~~~~~~~~~~~~~~~^
(APIServer pid=76251) asyncio_mode=True,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) ...<3 lines>...
(APIServer pid=76251) client_addresses=client_addresses,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) )
(APIServer pid=76251) ^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core_client.py", line 477, in __init__
(APIServer pid=76251) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=76251) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) File "/usr/lib64/python3.13/contextlib.py", line 148, in __exit__
(APIServer pid=76251) next(self.gen)
(APIServer pid=76251) ~~~~^^^^^^^^^^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/utils.py", line 903, in launch_core_engines
(APIServer pid=76251) wait_for_engine_startup(
(APIServer pid=76251) ~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251) handshake_socket,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^
(APIServer pid=76251) ...<5 lines>...
(APIServer pid=76251) coordinator.proc if coordinator else None,
(APIServer pid=76251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251) )
(APIServer pid=76251) ^
(APIServer pid=76251) File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/utils.py", line 960, in wait_for_engine_startup
(APIServer pid=76251) raise RuntimeError(
(APIServer pid=76251) ...<3 lines>...
(APIServer pid=76251) )
(APIServer pid=76251) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}