cjw/amd-r9700-vllm-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a04b9ef9c5807c3937429e93fd2166aebb347518
amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_amd-r9700-rocm_atten/cpatonn_Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit_tp2_server.log
T
cjw a04b9ef9c5 x
2026-03-26 20:50:00 +08:00

97 KiB
Raw Blame History

Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 16:52:23 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=69653) INFO 12-19 16:52:23 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=69653) INFO 12-19 16:52:23 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'trust_remote_code': True, 'max_model_len': 24576, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
(APIServer pid=69653) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=69653) INFO 12-19 16:52:28 [model.py:514] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=69653) INFO 12-19 16:52:28 [model.py:1636] Using max model len 24576
(APIServer pid=69653) INFO 12-19 16:52:28 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=69815) INFO 12-19 16:52:32 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=24576, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=69815) WARNING 12-19 16:52:32 [multiproc_executor.py:884] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
INFO 12-19 16:52:35 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:49789 backend=nccl
INFO 12-19 16:52:35 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:49789 backend=nccl
INFO 12-19 16:52:35 [pynccl.py:111] vLLM is using nccl==2.27.3
INFO 12-19 16:52:36 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
INFO 12-19 16:52:36 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(Worker_TP0 pid=69898) INFO 12-19 16:52:36 [gpu_model_runner.py:3562] Starting to load model cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit...
(Worker_TP1 pid=69899) INFO 12-19 16:52:37 [compressed_tensors_wNa16.py:114] Using ConchLinearKernel for CompressedTensorsWNA16
(Worker_TP0 pid=69898) INFO 12-19 16:52:37 [compressed_tensors_wNa16.py:114] Using ConchLinearKernel for CompressedTensorsWNA16
(Worker_TP1 pid=69899) INFO 12-19 16:52:37 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP1 pid=69899) INFO 12-19 16:52:37 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(Worker_TP1 pid=69899) INFO 12-19 16:52:37 [compressed_tensors_moe.py:188] Using CompressedTensorsWNA16MoEMethod
(Worker_TP1 pid=69899) WARNING 12-19 16:52:37 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(Worker_TP0 pid=69898) INFO 12-19 16:52:37 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP0 pid=69898) INFO 12-19 16:52:37 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(Worker_TP0 pid=69898) INFO 12-19 16:52:37 [compressed_tensors_moe.py:188] Using CompressedTensorsWNA16MoEMethod
(Worker_TP0 pid=69898) WARNING 12-19 16:52:37 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 1.61it/s]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:02<00:03, 1.60s/it]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:05<00:01, 1.90s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:07<00:00, 2.13s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:07<00:00, 1.91s/it]
(Worker_TP0 pid=69898)
(Worker_TP0 pid=69898) INFO 12-19 16:52:46 [default_loader.py:308] Loading weights took 7.71 seconds
(Worker_TP0 pid=69898) INFO 12-19 16:52:46 [gpu_model_runner.py:3659] Model loading took 8.1992 GiB memory and 9.032124 seconds
(Worker_TP0 pid=69898) INFO 12-19 16:52:52 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/48c10e1f97/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=69898) INFO 12-19 16:52:52 [backends.py:694] Dynamo bytecode transform time: 5.46 s
(Worker_TP1 pid=69899) INFO 12-19 16:52:55 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=69898) INFO 12-19 16:52:55 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP1 pid=69899) WARNING 12-19 16:52:55 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
(Worker_TP0 pid=69898) WARNING 12-19 16:52:55 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
(Worker_TP0 pid=69898) INFO 12-19 16:52:57 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 2.62 s
(Worker_TP0 pid=69898) INFO 12-19 16:52:57 [monitor.py:34] torch.compile takes 8.08 s in total
(Worker_TP0 pid=69898) INFO 12-19 16:53:00 [gpu_worker.py:375] Available KV cache memory: 22.22 GiB
(EngineCore_DP0 pid=69815) INFO 12-19 16:53:00 [kv_cache_utils.py:1291] GPU KV cache size: 485,344 tokens
(EngineCore_DP0 pid=69815) INFO 12-19 16:53:00 [kv_cache_utils.py:1296] Maximum concurrency for 24,576 tokens per request: 19.75x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:09<00:00, 1.97it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:04<00:00, 2.28it/s]
(Worker_TP0 pid=69898) INFO 12-19 16:53:16 [gpu_model_runner.py:4610] Graph capturing finished in 15 secs, took 0.76 GiB
(EngineCore_DP0 pid=69815) INFO 12-19 16:53:16 [core.py:259] init engine (profile, create kv cache, warmup model) took 29.33 seconds
(APIServer pid=69653) INFO 12-19 16:53:17 [api_server.py:1099] Supported tasks: ['generate']
(APIServer pid=69653) WARNING 12-19 16:53:17 [model.py:1462] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=69653) INFO 12-19 16:53:17 [serving_responses.py:201] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=69653) INFO 12-19 16:53:17 [serving_chat.py:137] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=69653) INFO 12-19 16:53:17 [serving_completion.py:77] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=69653) INFO 12-19 16:53:17 [serving_chat.py:137] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=69653) INFO 12-19 16:53:17 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:38] Available routes are:
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=69653) INFO 12-19 16:53:17 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=69653) INFO: Started server process [69653]
(APIServer pid=69653) INFO: Waiting for application startup.
(APIServer pid=69653) INFO: Application startup complete.
(APIServer pid=69653) INFO: 127.0.0.1:36624 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:53:38 [loggers.py:248] Engine 000: Avg prompt throughput: 428.2 tokens/s, Avg generation throughput: 153.1 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41498 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:53:48 [loggers.py:248] Engine 000: Avg prompt throughput: 1264.2 tokens/s, Avg generation throughput: 435.2 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41498 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:53:58 [loggers.py:248] Engine 000: Avg prompt throughput: 925.9 tokens/s, Avg generation throughput: 700.7 tokens/s, Running: 41 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:54:08 [loggers.py:248] Engine 000: Avg prompt throughput: 643.5 tokens/s, Avg generation throughput: 815.1 tokens/s, Running: 54 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41498 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:54:18 [loggers.py:248] Engine 000: Avg prompt throughput: 662.7 tokens/s, Avg generation throughput: 855.4 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:54:28 [loggers.py:248] Engine 000: Avg prompt throughput: 978.9 tokens/s, Avg generation throughput: 826.0 tokens/s, Running: 53 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:54:38 [loggers.py:248] Engine 000: Avg prompt throughput: 936.6 tokens/s, Avg generation throughput: 896.8 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41498 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:54:48 [loggers.py:248] Engine 000: Avg prompt throughput: 480.3 tokens/s, Avg generation throughput: 907.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34034 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:54:58 [loggers.py:248] Engine 000: Avg prompt throughput: 919.2 tokens/s, Avg generation throughput: 935.0 tokens/s, Running: 63 reqs, Waiting: 5 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:54884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34034 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:55:08 [loggers.py:248] Engine 000: Avg prompt throughput: 627.6 tokens/s, Avg generation throughput: 1008.0 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41498 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41498 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:55:18 [loggers.py:248] Engine 000: Avg prompt throughput: 1008.7 tokens/s, Avg generation throughput: 869.9 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.8%
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:55:28 [loggers.py:248] Engine 000: Avg prompt throughput: 756.8 tokens/s, Avg generation throughput: 756.5 tokens/s, Running: 45 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.8%
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34034 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:55:38 [loggers.py:248] Engine 000: Avg prompt throughput: 690.0 tokens/s, Avg generation throughput: 750.2 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.7%
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34034 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:55:48 [loggers.py:248] Engine 000: Avg prompt throughput: 757.2 tokens/s, Avg generation throughput: 808.7 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.9%
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44442 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:55:58 [loggers.py:248] Engine 000: Avg prompt throughput: 840.8 tokens/s, Avg generation throughput: 722.0 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.8%
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34034 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34034 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:56:08 [loggers.py:248] Engine 000: Avg prompt throughput: 823.4 tokens/s, Avg generation throughput: 727.0 tokens/s, Running: 45 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.8%
(APIServer pid=69653) INFO: 127.0.0.1:54832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:56:18 [loggers.py:248] Engine 000: Avg prompt throughput: 495.1 tokens/s, Avg generation throughput: 806.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.7%
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60354 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60362 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60362 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60354 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:56:28 [loggers.py:248] Engine 000: Avg prompt throughput: 1132.0 tokens/s, Avg generation throughput: 769.2 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.7%
(APIServer pid=69653) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:60354 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33272 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:34056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54896 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:54814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:33840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO: 127.0.0.1:41560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=69653) INFO 12-19 16:56:38 [loggers.py:248] Engine 000: Avg prompt throughput: 298.6 tokens/s, Avg generation throughput: 914.9 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.7%
(APIServer pid=69653) INFO 12-19 16:56:48 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 512.4 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.7%
(APIServer pid=69653) INFO 12-19 16:56:58 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 175.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.7%
(APIServer pid=69653) INFO 12-19 16:56:58 [launcher.py:110] Shutting down FastAPI HTTP server.
(Worker_TP1 pid=69899) INFO 12-19 16:56:58 [multiproc_executor.py:711] Parent process exited, terminating worker
(Worker_TP0 pid=69898) INFO 12-19 16:56:58 [multiproc_executor.py:711] Parent process exited, terminating worker
Reference in New Issue View Git Blame Copy Permalink