amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_amd-r9700-rocm_atten/cpatonn_Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit_tp1_server.log

Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 15:18:51 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=59832) INFO 12-19 15:18:51 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=59832) INFO 12-19 15:18:51 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'trust_remote_code': True, 'max_model_len': 24576, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
(APIServer pid=59832) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=59832) INFO 12-19 15:18:55 [model.py:514] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=59832) INFO 12-19 15:18:55 [model.py:1636] Using max model len 24576
(APIServer pid=59832) INFO 12-19 15:18:55 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=59999) INFO 12-19 15:18:59 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=24576, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=59999) INFO 12-19 15:18:59 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:34261 backend=nccl
(EngineCore_DP0 pid=59999) INFO 12-19 15:18:59 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:00 [gpu_model_runner.py:3562] Starting to load model cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit...
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:00 [compressed_tensors_wNa16.py:114] Using ConchLinearKernel for CompressedTensorsWNA16
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:00 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:00 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:00 [compressed_tensors_moe.py:188] Using CompressedTensorsWNA16MoEMethod
(EngineCore_DP0 pid=59999) WARNING 12-19 15:19:00 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:02, 1.43it/s]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:03<00:03, 1.75s/it]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:05<00:02, 2.03s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00, 2.25s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00, 2.03s/it]
(EngineCore_DP0 pid=59999)
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:10 [default_loader.py:308] Loading weights took 8.18 seconds
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:10 [gpu_model_runner.py:3659] Model loading took 16.2266 GiB memory and 9.521962 seconds
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:15 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/8582846099/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:15 [backends.py:694] Dynamo bytecode transform time: 5.05 s
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:18 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(EngineCore_DP0 pid=59999) WARNING 12-19 15:19:19 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:21 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 2.44 s
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:21 [monitor.py:34] torch.compile takes 7.50 s in total
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:22 [gpu_worker.py:375] Available KV cache memory: 14.08 GiB
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:23 [kv_cache_utils.py:1291] GPU KV cache size: 153,776 tokens
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:23 [kv_cache_utils.py:1296] Maximum concurrency for 24,576 tokens per request: 6.26x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 22.41it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 22.70it/s]
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:25 [gpu_model_runner.py:4610] Graph capturing finished in 2 secs, took 1.14 GiB
(EngineCore_DP0 pid=59999) INFO 12-19 15:19:25 [core.py:259] init engine (profile, create kv cache, warmup model) took 14.56 seconds
(APIServer pid=59832) INFO 12-19 15:19:26 [api_server.py:1099] Supported tasks: ['generate']
(APIServer pid=59832) WARNING 12-19 15:19:26 [model.py:1462] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=59832) INFO 12-19 15:19:26 [serving_responses.py:201] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=59832) INFO 12-19 15:19:26 [serving_chat.py:137] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=59832) INFO 12-19 15:19:26 [serving_completion.py:77] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=59832) INFO 12-19 15:19:26 [serving_chat.py:137] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=59832) INFO 12-19 15:19:26 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:38] Available routes are:
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=59832) INFO 12-19 15:19:26 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=59832) INFO: Started server process [59832]
(APIServer pid=59832) INFO: Waiting for application startup.
(APIServer pid=59832) INFO: Application startup complete.
(APIServer pid=59832) INFO: 127.0.0.1:58088 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47470 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47500 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:19:47 [loggers.py:248] Engine 000: Avg prompt throughput: 84.0 tokens/s, Avg generation throughput: 68.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:47506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:19:57 [loggers.py:248] Engine 000: Avg prompt throughput: 136.8 tokens/s, Avg generation throughput: 119.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:20:07 [loggers.py:248] Engine 000: Avg prompt throughput: 207.4 tokens/s, Avg generation throughput: 166.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:57096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47470 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:20:17 [loggers.py:248] Engine 000: Avg prompt throughput: 300.9 tokens/s, Avg generation throughput: 182.7 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:47506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47470 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:20:27 [loggers.py:248] Engine 000: Avg prompt throughput: 259.1 tokens/s, Avg generation throughput: 176.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32816 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32816 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:20:37 [loggers.py:248] Engine 000: Avg prompt throughput: 251.4 tokens/s, Avg generation throughput: 213.0 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:20:47 [loggers.py:248] Engine 000: Avg prompt throughput: 488.7 tokens/s, Avg generation throughput: 231.2 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:20:57 [loggers.py:248] Engine 000: Avg prompt throughput: 149.9 tokens/s, Avg generation throughput: 198.1 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:49788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:21:07 [loggers.py:248] Engine 000: Avg prompt throughput: 299.2 tokens/s, Avg generation throughput: 231.5 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:47196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:21:17 [loggers.py:248] Engine 000: Avg prompt throughput: 310.1 tokens/s, Avg generation throughput: 272.8 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:58174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:21:27 [loggers.py:248] Engine 000: Avg prompt throughput: 132.9 tokens/s, Avg generation throughput: 285.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:21:37 [loggers.py:248] Engine 000: Avg prompt throughput: 109.9 tokens/s, Avg generation throughput: 262.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:58174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39218 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:21:47 [loggers.py:248] Engine 000: Avg prompt throughput: 248.5 tokens/s, Avg generation throughput: 215.5 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39218 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39218 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:21:57 [loggers.py:248] Engine 000: Avg prompt throughput: 182.1 tokens/s, Avg generation throughput: 243.8 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:57090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:22:07 [loggers.py:248] Engine 000: Avg prompt throughput: 130.1 tokens/s, Avg generation throughput: 217.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:42874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39218 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:22:17 [loggers.py:248] Engine 000: Avg prompt throughput: 80.4 tokens/s, Avg generation throughput: 154.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:58042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:22:27 [loggers.py:248] Engine 000: Avg prompt throughput: 140.2 tokens/s, Avg generation throughput: 203.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:51614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:51614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:22:37 [loggers.py:248] Engine 000: Avg prompt throughput: 275.0 tokens/s, Avg generation throughput: 184.4 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:41780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:22:47 [loggers.py:248] Engine 000: Avg prompt throughput: 50.4 tokens/s, Avg generation throughput: 228.4 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO 12-19 15:22:57 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 147.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:49322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:23:07 [loggers.py:248] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 35.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=59832) INFO: 127.0.0.1:49322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:23:17 [loggers.py:248] Engine 000: Avg prompt throughput: 641.4 tokens/s, Avg generation throughput: 192.3 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 13.8%
(APIServer pid=59832) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:23:27 [loggers.py:248] Engine 000: Avg prompt throughput: 1213.2 tokens/s, Avg generation throughput: 460.5 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 31.4%
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:23:37 [loggers.py:248] Engine 000: Avg prompt throughput: 797.2 tokens/s, Avg generation throughput: 589.9 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.4%, Prefix cache hit rate: 39.2%
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53310 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53368 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53310 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:23:47 [loggers.py:248] Engine 000: Avg prompt throughput: 674.7 tokens/s, Avg generation throughput: 685.8 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.6%, Prefix cache hit rate: 44.6%
(APIServer pid=59832) INFO: 127.0.0.1:53368 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34886 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:23:57 [loggers.py:248] Engine 000: Avg prompt throughput: 593.0 tokens/s, Avg generation throughput: 726.2 tokens/s, Running: 64 reqs, Waiting: 11 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 47.5%
(APIServer pid=59832) INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45112 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45138 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:24:07 [loggers.py:248] Engine 000: Avg prompt throughput: 542.8 tokens/s, Avg generation throughput: 684.8 tokens/s, Running: 61 reqs, Waiting: 28 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 44.4%
(APIServer pid=59832) INFO: 127.0.0.1:49484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:24:17 [loggers.py:248] Engine 000: Avg prompt throughput: 920.6 tokens/s, Avg generation throughput: 665.5 tokens/s, Running: 64 reqs, Waiting: 25 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 39.9%
(APIServer pid=59832) INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39984 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:24:27 [loggers.py:248] Engine 000: Avg prompt throughput: 565.0 tokens/s, Avg generation throughput: 678.4 tokens/s, Running: 64 reqs, Waiting: 42 reqs, GPU KV cache usage: 16.9%, Prefix cache hit rate: 37.6%
(APIServer pid=59832) INFO: 127.0.0.1:44244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45138 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45112 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44302 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44318 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44344 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44354 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:24:37 [loggers.py:248] Engine 000: Avg prompt throughput: 444.7 tokens/s, Avg generation throughput: 710.4 tokens/s, Running: 64 reqs, Waiting: 58 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 36.0%
(APIServer pid=59832) INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53310 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34886 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49532 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:24:47 [loggers.py:248] Engine 000: Avg prompt throughput: 384.4 tokens/s, Avg generation throughput: 729.6 tokens/s, Running: 64 reqs, Waiting: 73 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 34.7%
(APIServer pid=59832) INFO: 127.0.0.1:43572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43594 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43618 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43672 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39984 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44302 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:24:57 [loggers.py:248] Engine 000: Avg prompt throughput: 614.0 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 64 reqs, Waiting: 81 reqs, GPU KV cache usage: 13.4%, Prefix cache hit rate: 32.8%
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53368 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34886 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:25:07 [loggers.py:248] Engine 000: Avg prompt throughput: 690.7 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 73 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 30.9%
(APIServer pid=59832) INFO: 127.0.0.1:42182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45112 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43594 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:25:17 [loggers.py:248] Engine 000: Avg prompt throughput: 935.6 tokens/s, Avg generation throughput: 659.2 tokens/s, Running: 64 reqs, Waiting: 74 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 28.6%
(APIServer pid=59832) INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44354 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39984 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44302 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44318 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44344 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44258 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:25:27 [loggers.py:248] Engine 000: Avg prompt throughput: 687.6 tokens/s, Avg generation throughput: 684.7 tokens/s, Running: 64 reqs, Waiting: 85 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 27.2%
(APIServer pid=59832) INFO: 127.0.0.1:49506 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53310 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45112 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49448 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:39982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:25:37 [loggers.py:248] Engine 000: Avg prompt throughput: 750.4 tokens/s, Avg generation throughput: 671.9 tokens/s, Running: 63 reqs, Waiting: 83 reqs, GPU KV cache usage: 13.7%, Prefix cache hit rate: 25.8%
(APIServer pid=59832) INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43672 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:38918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:38924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:38938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:38944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49532 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43594 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:25:47 [loggers.py:248] Engine 000: Avg prompt throughput: 554.3 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 62 reqs, Waiting: 83 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 24.9%
(APIServer pid=59832) INFO: 127.0.0.1:37776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43660 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49352 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53368 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37646 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43618 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49458 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44318 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45112 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:25:57 [loggers.py:248] Engine 000: Avg prompt throughput: 909.9 tokens/s, Avg generation throughput: 665.6 tokens/s, Running: 64 reqs, Waiting: 81 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 23.5%
(APIServer pid=59832) INFO: 127.0.0.1:44258 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34886 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45138 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:53364 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44302 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:34874 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:44286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41428 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:43672 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41438 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41470 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41482 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41500 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:40040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:26:07 [loggers.py:248] Engine 000: Avg prompt throughput: 584.0 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 63 reqs, Waiting: 100 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 22.7%
(APIServer pid=59832) INFO: 127.0.0.1:38944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41402 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41420 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:41430 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:49456 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=59832) INFO 12-19 15:26:17 [loggers.py:248] Engine 000: Avg prompt throughput: 523.2 tokens/s, Avg generation throughput: 710.4 tokens/s, Running: 63 reqs, Waiting: 84 reqs, GPU KV cache usage: 13.6%, Prefix cache hit rate: 22.0%
(APIServer pid=59832) INFO 12-19 15:26:27 [loggers.py:248] Engine 000: Avg prompt throughput: 762.8 tokens/s, Avg generation throughput: 704.0 tokens/s, Running: 64 reqs, Waiting: 52 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 21.0%
(APIServer pid=59832) INFO 12-19 15:26:37 [loggers.py:248] Engine 000: Avg prompt throughput: 734.4 tokens/s, Avg generation throughput: 704.0 tokens/s, Running: 64 reqs, Waiting: 13 reqs, GPU KV cache usage: 14.0%, Prefix cache hit rate: 20.2%
(APIServer pid=59832) INFO 12-19 15:26:47 [loggers.py:248] Engine 000: Avg prompt throughput: 144.9 tokens/s, Avg generation throughput: 714.5 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 20.0%
(APIServer pid=59832) INFO 12-19 15:26:57 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 503.9 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 20.0%
(APIServer pid=59832) INFO 12-19 15:27:07 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 238.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 20.0%
(APIServer pid=59832) INFO 12-19 15:27:17 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 113.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 20.0%
(APIServer pid=59832) INFO 12-19 15:27:18 [launcher.py:110] Shutting down FastAPI HTTP server.
[rank0]:[W1219 15:27:18.582580780 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())