Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1             Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 14:41:24 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=57644) INFO 12-19 14:41:24 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=57644) INFO 12-19 14:41:24 [utils.py:253] non-default args: {'model_tag': 'openai/gpt-oss-20b', 'host': '127.0.0.1', 'model': 'openai/gpt-oss-20b', 'trust_remote_code': True, 'max_model_len': 32768, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
(APIServer pid=57644) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=57644) INFO 12-19 14:41:28 [model.py:514] Resolved architecture: GptOssForCausalLM
(APIServer pid=57644) 
Parse safetensors files:   0%|          | 0/3 [00:00<?, ?it/s]
Parse safetensors files:  33%|███▎      | 1/3 [00:00<00:00,  6.78it/s]
Parse safetensors files:  67%|██████▋   | 2/3 [00:00<00:00,  4.85it/s]
Parse safetensors files: 100%|██████████| 3/3 [00:00<00:00,  7.59it/s]
(APIServer pid=57644) INFO 12-19 14:41:29 [model.py:1636] Using max model len 32768
(APIServer pid=57644) INFO 12-19 14:41:29 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=57644) INFO 12-19 14:41:29 [config.py:271] Overriding max cuda graph capture size to 1024 for performance.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1             Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:33 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='openai/gpt-oss-20b', speculative_config=None, tokenizer='openai/gpt-oss-20b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=mxfp4, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='openai_gptoss', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=openai/gpt-oss-20b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 528, 544, 560, 576, 592, 608, 624, 640, 656, 672, 688, 704, 720, 736, 752, 768, 784, 800, 816, 832, 848, 864, 880, 896, 912, 928, 944, 960, 976, 992, 1008, 1024], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 1024, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:33 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:59097 backend=nccl
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:33 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:34 [gpu_model_runner.py:3562] Starting to load model openai/gpt-oss-20b...
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:34 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:34 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:34 [mxfp4.py:170] Using Triton backend
(EngineCore_DP0 pid=57811) 
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
(EngineCore_DP0 pid=57811) 
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:01,  1.08it/s]
(EngineCore_DP0 pid=57811) 
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:02<00:01,  1.15s/it]
(EngineCore_DP0 pid=57811) 
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00,  1.22s/it]
(EngineCore_DP0 pid=57811) 
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00,  1.18s/it]
(EngineCore_DP0 pid=57811) 
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:38 [default_loader.py:308] Loading weights took 3.63 seconds
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:39 [gpu_model_runner.py:3659] Model loading took 14.3066 GiB memory and 4.313271 seconds
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:41 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/4c3c97b6bf/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:41 [backends.py:694] Dynamo bytecode transform time: 1.69 s
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:42 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:43 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 1.18 s
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:43 [monitor.py:34] torch.compile takes 2.87 s in total
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:44 [gpu_worker.py:375] Available KV cache memory: 15.77 GiB
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:44 [kv_cache_utils.py:1291] GPU KV cache size: 344,560 tokens
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:44 [kv_cache_utils.py:1296] Maximum concurrency for 32,768 tokens per request: 19.71x
(EngineCore_DP0 pid=57811) 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/83 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   1%|          | 1/83 [00:00<00:14,  5.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   2%|▏         | 2/83 [00:00<00:13,  6.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   4%|▎         | 3/83 [00:00<00:12,  6.21it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   5%|▍         | 4/83 [00:00<00:12,  6.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   6%|▌         | 5/83 [00:00<00:12,  6.46it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   7%|▋         | 6/83 [00:00<00:11,  6.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   8%|▊         | 7/83 [00:01<00:11,  6.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  10%|▉         | 8/83 [00:01<00:11,  6.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█         | 9/83 [00:01<00:10,  6.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  12%|█▏        | 10/83 [00:01<00:10,  6.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  13%|█▎        | 11/83 [00:01<00:10,  6.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  14%|█▍        | 12/83 [00:01<00:10,  7.04it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  16%|█▌        | 13/83 [00:01<00:09,  7.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  17%|█▋        | 14/83 [00:02<00:09,  7.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  18%|█▊        | 15/83 [00:02<00:09,  7.27it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  19%|█▉        | 16/83 [00:02<00:09,  7.28it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|██        | 17/83 [00:02<00:08,  7.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  22%|██▏       | 18/83 [00:02<00:08,  7.55it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 19/83 [00:02<00:08,  7.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  24%|██▍       | 20/83 [00:02<00:08,  7.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  25%|██▌       | 21/83 [00:02<00:07,  7.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  27%|██▋       | 22/83 [00:03<00:07,  7.97it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  28%|██▊       | 23/83 [00:03<00:07,  7.99it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  29%|██▉       | 24/83 [00:03<00:07,  8.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  30%|███       | 25/83 [00:03<00:07,  8.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 26/83 [00:03<00:06,  8.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  33%|███▎      | 27/83 [00:03<00:06,  8.54it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▎      | 28/83 [00:03<00:06,  8.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  35%|███▍      | 29/83 [00:03<00:06,  8.97it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  37%|███▋      | 31/83 [00:04<00:05,  9.35it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  40%|███▉      | 33/83 [00:04<00:05,  9.66it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  42%|████▏     | 35/83 [00:04<00:04, 10.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  45%|████▍     | 37/83 [00:04<00:04, 10.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  47%|████▋     | 39/83 [00:04<00:04, 10.63it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  49%|████▉     | 41/83 [00:05<00:03, 10.99it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  52%|█████▏    | 43/83 [00:05<00:03, 11.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  54%|█████▍    | 45/83 [00:05<00:03, 11.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 47/83 [00:05<00:02, 12.54it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  59%|█████▉    | 49/83 [00:05<00:02, 13.15it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  61%|██████▏   | 51/83 [00:05<00:02, 13.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  64%|██████▍   | 53/83 [00:05<00:02, 14.34it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  66%|██████▋   | 55/83 [00:05<00:01, 14.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 57/83 [00:06<00:01, 15.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  71%|███████   | 59/83 [00:06<00:01, 15.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  73%|███████▎  | 61/83 [00:06<00:01, 16.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  76%|███████▌  | 63/83 [00:06<00:01, 16.86it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  78%|███████▊  | 65/83 [00:06<00:01, 17.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  82%|████████▏ | 68/83 [00:06<00:00, 18.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  86%|████████▌ | 71/83 [00:06<00:00, 19.93it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  89%|████████▉ | 74/83 [00:06<00:00, 21.19it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  93%|█████████▎| 77/83 [00:07<00:00, 22.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  96%|█████████▋| 80/83 [00:07<00:00, 23.65it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 83/83 [00:07<00:00, 19.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 83/83 [00:07<00:00, 11.22it/s]
(EngineCore_DP0 pid=57811) 
Capturing CUDA graphs (decode, FULL):   0%|          | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  27%|██▋       | 3/11 [00:00<00:00, 26.05it/s]
Capturing CUDA graphs (decode, FULL):  55%|█████▍    | 6/11 [00:00<00:00, 28.11it/s]
Capturing CUDA graphs (decode, FULL):  82%|████████▏ | 9/11 [00:00<00:00, 25.77it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 23.53it/s]
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:53 [gpu_model_runner.py:4610] Graph capturing finished in 9 secs, took 1.41 GiB
(EngineCore_DP0 pid=57811) INFO 12-19 14:41:53 [core.py:259] init engine (profile, create kv cache, warmup model) took 14.23 seconds
(APIServer pid=57644) INFO 12-19 14:41:54 [api_server.py:1099] Supported tasks: ['generate']
(APIServer pid=57644) WARNING 12-19 14:41:54 [serving_responses.py:222] For gpt-oss, we ignore --enable-auto-tool-choice and always enable tool use.
(APIServer pid=57644) INFO 12-19 14:41:55 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:38] Available routes are:
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=57644) INFO 12-19 14:41:55 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=57644) INFO:     Started server process [57644]
(APIServer pid=57644) INFO:     Waiting for application startup.
(APIServer pid=57644) INFO:     Application startup complete.
(APIServer pid=57644) INFO:     127.0.0.1:44822 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:42:15 [loggers.py:248] Engine 000: Avg prompt throughput: 4.9 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:45282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:42:25 [loggers.py:248] Engine 000: Avg prompt throughput: 132.7 tokens/s, Avg generation throughput: 85.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:42:35 [loggers.py:248] Engine 000: Avg prompt throughput: 223.5 tokens/s, Avg generation throughput: 149.1 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:42:45 [loggers.py:248] Engine 000: Avg prompt throughput: 275.0 tokens/s, Avg generation throughput: 176.1 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:42:55 [loggers.py:248] Engine 000: Avg prompt throughput: 223.7 tokens/s, Avg generation throughput: 183.3 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:59074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:43:05 [loggers.py:248] Engine 000: Avg prompt throughput: 366.7 tokens/s, Avg generation throughput: 194.7 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:36240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:45276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:52022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:52036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:43:15 [loggers.py:248] Engine 000: Avg prompt throughput: 331.0 tokens/s, Avg generation throughput: 218.7 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:45276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:43:25 [loggers.py:248] Engine 000: Avg prompt throughput: 275.3 tokens/s, Avg generation throughput: 213.6 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:60228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:52036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:52022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:43:35 [loggers.py:248] Engine 000: Avg prompt throughput: 288.6 tokens/s, Avg generation throughput: 199.0 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:43:45 [loggers.py:248] Engine 000: Avg prompt throughput: 174.4 tokens/s, Avg generation throughput: 257.5 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:43:55 [loggers.py:248] Engine 000: Avg prompt throughput: 377.1 tokens/s, Avg generation throughput: 276.8 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:44:05 [loggers.py:248] Engine 000: Avg prompt throughput: 42.4 tokens/s, Avg generation throughput: 291.9 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:52022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:44:15 [loggers.py:248] Engine 000: Avg prompt throughput: 196.8 tokens/s, Avg generation throughput: 188.3 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33300 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:44:25 [loggers.py:248] Engine 000: Avg prompt throughput: 242.5 tokens/s, Avg generation throughput: 248.1 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:52022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:59074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:44:35 [loggers.py:248] Engine 000: Avg prompt throughput: 156.5 tokens/s, Avg generation throughput: 217.1 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:44:45 [loggers.py:248] Engine 000: Avg prompt throughput: 63.4 tokens/s, Avg generation throughput: 182.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:59044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:33278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:60248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40644 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40656 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:44:55 [loggers.py:248] Engine 000: Avg prompt throughput: 156.7 tokens/s, Avg generation throughput: 165.4 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:40676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40644 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:45:05 [loggers.py:248] Engine 000: Avg prompt throughput: 89.6 tokens/s, Avg generation throughput: 185.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:36256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53362 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53404 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:40676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:53404 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:45:15 [loggers.py:248] Engine 000: Avg prompt throughput: 256.0 tokens/s, Avg generation throughput: 221.2 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO 12-19 14:45:25 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO 12-19 14:45:35 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO 12-19 14:45:45 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 26.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:45:55 [loggers.py:248] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 9.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43436 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:46:05 [loggers.py:248] Engine 000: Avg prompt throughput: 359.9 tokens/s, Avg generation throughput: 112.5 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 8.1%
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:46:15 [loggers.py:248] Engine 000: Avg prompt throughput: 1197.2 tokens/s, Avg generation throughput: 388.8 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 27.6%
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43436 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:46:25 [loggers.py:248] Engine 000: Avg prompt throughput: 1114.5 tokens/s, Avg generation throughput: 664.9 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 39.3%
(APIServer pid=57644) INFO:     127.0.0.1:51898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:46:35 [loggers.py:248] Engine 000: Avg prompt throughput: 638.2 tokens/s, Avg generation throughput: 721.7 tokens/s, Running: 59 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 44.3%
(APIServer pid=57644) INFO:     127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:46:45 [loggers.py:248] Engine 000: Avg prompt throughput: 582.4 tokens/s, Avg generation throughput: 815.7 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 47.9%
(APIServer pid=57644) INFO:     127.0.0.1:51752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:46:55 [loggers.py:248] Engine 000: Avg prompt throughput: 983.1 tokens/s, Avg generation throughput: 787.0 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 42.5%
(APIServer pid=57644) INFO:     127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43436 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43436 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:47:05 [loggers.py:248] Engine 000: Avg prompt throughput: 694.6 tokens/s, Avg generation throughput: 820.6 tokens/s, Running: 64 reqs, Waiting: 11 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 39.4%
(APIServer pid=57644) INFO:     127.0.0.1:51670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:47:15 [loggers.py:248] Engine 000: Avg prompt throughput: 739.8 tokens/s, Avg generation throughput: 848.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 36.5%
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55024 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55072 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55086 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55094 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55110 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55120 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:47:25 [loggers.py:248] Engine 000: Avg prompt throughput: 446.1 tokens/s, Avg generation throughput: 887.8 tokens/s, Running: 64 reqs, Waiting: 27 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 35.0%
(APIServer pid=57644) INFO:     127.0.0.1:55144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51688 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:39056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55086 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55094 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43436 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:47:35 [loggers.py:248] Engine 000: Avg prompt throughput: 714.1 tokens/s, Avg generation throughput: 876.7 tokens/s, Running: 64 reqs, Waiting: 15 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 32.8%
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55072 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:39056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55072 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55086 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43436 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:47:45 [loggers.py:248] Engine 000: Avg prompt throughput: 873.1 tokens/s, Avg generation throughput: 851.2 tokens/s, Running: 64 reqs, Waiting: 13 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 30.4%
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47918 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55094 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:47:55 [loggers.py:248] Engine 000: Avg prompt throughput: 1214.1 tokens/s, Avg generation throughput: 789.9 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 28.2%
(APIServer pid=57644) INFO:     127.0.0.1:51752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55110 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:48:05 [loggers.py:248] Engine 000: Avg prompt throughput: 573.6 tokens/s, Avg generation throughput: 800.4 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 27.1%
(APIServer pid=57644) INFO:     127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43376 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:39056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43410 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:48:15 [loggers.py:248] Engine 000: Avg prompt throughput: 682.6 tokens/s, Avg generation throughput: 835.3 tokens/s, Running: 63 reqs, Waiting: 8 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 26.0%
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43412 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:48:25 [loggers.py:248] Engine 000: Avg prompt throughput: 1010.3 tokens/s, Avg generation throughput: 764.0 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 24.3%
(APIServer pid=57644) INFO:     127.0.0.1:55054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:39056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55022 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55110 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:48:35 [loggers.py:248] Engine 000: Avg prompt throughput: 773.4 tokens/s, Avg generation throughput: 738.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 23.2%
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51668 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:48:45 [loggers.py:248] Engine 000: Avg prompt throughput: 455.7 tokens/s, Avg generation throughput: 789.5 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 22.5%
(APIServer pid=57644) INFO:     127.0.0.1:51854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42580 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46492 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55086 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42580 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:41550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55312 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:39056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:48:55 [loggers.py:248] Engine 000: Avg prompt throughput: 1172.2 tokens/s, Avg generation throughput: 780.2 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 21.1%
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:47946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:49608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:49612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:55108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:49614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:49620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:46028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:42604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO:     127.0.0.1:51670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=57644) INFO 12-19 14:49:05 [loggers.py:248] Engine 000: Avg prompt throughput: 328.7 tokens/s, Avg generation throughput: 876.7 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 20.7%
(APIServer pid=57644) INFO 12-19 14:49:15 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 587.6 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 20.7%
(APIServer pid=57644) INFO 12-19 14:49:25 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 278.1 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 20.7%
(APIServer pid=57644) INFO 12-19 14:49:35 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 126.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 20.7%
(APIServer pid=57644) INFO 12-19 14:49:45 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 31.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 20.7%
(APIServer pid=57644) INFO 12-19 14:49:55 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 20.7%
(APIServer pid=57644) INFO 12-19 14:49:58 [launcher.py:110] Shutting down FastAPI HTTP server.
[rank0]:[W1219 14:49:58.849348158 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())