amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_nvidia-5090/openai_gpt-oss-20b_tp1_server.log

WARNING 12-09 18:35:12 [argparse_utils.py:195] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=8738) INFO 12-09 18:35:12 [api_server.py:1772] vLLM API server version 0.12.0
(APIServer pid=8738) INFO 12-09 18:35:12 [utils.py:253] non-default args: {'model_tag': 'openai/gpt-oss-20b', 'host': '127.0.0.1', 'model': 'openai/gpt-oss-20b', 'trust_remote_code': True, 'max_model_len': 32768, 'gpu_memory_utilization': 0.95, 'max_num_seqs': 64}
(APIServer pid=8738) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=8738) INFO 12-09 18:35:19 [model.py:637] Resolved architecture: GptOssForCausalLM
Parse safetensors files: 100%|██████████| 3/3 [00:00<00:00, 7.21it/s]
(APIServer pid=8738) INFO 12-09 18:35:19 [model.py:1750] Using max model len 32768
(APIServer pid=8738) INFO 12-09 18:35:21 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=8738) INFO 12-09 18:35:21 [config.py:274] Overriding max cuda graph capture size to 1024 for performance.
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:27 [core.py:93] Initializing a V1 LLM engine (v0.12.0) with config: model='openai/gpt-oss-20b', speculative_config=None, tokenizer='openai/gpt-oss-20b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=mxfp4, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='openai_gptoss', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=openai/gpt-oss-20b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 528, 544, 560, 576, 592, 608, 624, 640, 656, 672, 688, 704, 720, 736, 752, 768, 784, 800, 816, 832, 848, 864, 880, 896, 912, 928, 944, 960, 976, 992, 1008, 1024], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 1024, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None}
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:28 [parallel_state.py:1200] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.4:35797 backend=nccl
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:28 [parallel_state.py:1408] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:28 [gpu_model_runner.py:3467] Starting to load model openai/gpt-oss-20b...
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:28 [cuda.py:411] Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN']
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:28 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:28 [mxfp4.py:162] Using Marlin backend
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:00<00:00, 2.91it/s]
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:00<00:00, 2.23it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:01<00:00, 2.03it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:01<00:00, 2.12it/s]
(EngineCore_DP0 pid=9006)
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:31 [default_loader.py:308] Loading weights took 1.57 seconds
(EngineCore_DP0 pid=9006) WARNING 12-09 18:35:31 [marlin_utils_fp4.py:226] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:31 [gpu_model_runner.py:3549] Model loading took 13.7194 GiB memory and 2.842897 seconds
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:34 [backends.py:655] Using cache directory: /root/.cache/vllm/torch_compile_cache/b0871f56e7/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:34 [backends.py:715] Dynamo bytecode transform time: 2.83 s
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:35 [backends.py:257] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:38 [backends.py:288] Compiling a graph for dynamic shape takes 2.86 s
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:39 [monitor.py:34] torch.compile takes 5.69 s in total
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:39 [gpu_worker.py:359] Available KV cache memory: 15.37 GiB
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:39 [kv_cache_utils.py:1286] GPU KV cache size: 335,664 tokens
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:39 [kv_cache_utils.py:1291] Maximum concurrency for 32,768 tokens per request: 19.20x
(EngineCore_DP0 pid=9006) 2025-12-09 18:35:39,976 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore_DP0 pid=9006) 2025-12-09 18:35:39,993 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 83/83 [00:03<00:00, 25.16it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:04<00:00, 2.70it/s]
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:47 [gpu_model_runner.py:4466] Graph capturing finished in 8 secs, took 0.33 GiB
(EngineCore_DP0 pid=9006) INFO 12-09 18:35:47 [core.py:254] init engine (profile, create kv cache, warmup model) took 16.07 seconds
(APIServer pid=8738) INFO 12-09 18:35:49 [api_server.py:1520] Supported tasks: ['generate']
(APIServer pid=8738) WARNING 12-09 18:35:50 [serving_responses.py:215] For gpt-oss, we ignore --enable-auto-tool-choice and always enable tool use.
(APIServer pid=8738) INFO 12-09 18:35:52 [api_server.py:1847] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:38] Available routes are:
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=8738) INFO 12-09 18:35:52 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=8738) INFO: Started server process [8738]
(APIServer pid=8738) INFO: Waiting for application startup.
(APIServer pid=8738) INFO: Application startup complete.
(APIServer pid=8738) INFO: 127.0.0.1:60326 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:36:12 [loggers.py:236] Engine 000: Avg prompt throughput: 7.5 tokens/s, Avg generation throughput: 55.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:54744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:36:22 [loggers.py:236] Engine 000: Avg prompt throughput: 130.1 tokens/s, Avg generation throughput: 217.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:50846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:36:32 [loggers.py:236] Engine 000: Avg prompt throughput: 287.2 tokens/s, Avg generation throughput: 171.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:36:42 [loggers.py:236] Engine 000: Avg prompt throughput: 211.3 tokens/s, Avg generation throughput: 171.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:36:52 [loggers.py:236] Engine 000: Avg prompt throughput: 225.3 tokens/s, Avg generation throughput: 226.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:37:02 [loggers.py:236] Engine 000: Avg prompt throughput: 365.8 tokens/s, Avg generation throughput: 194.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44848 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:37:12 [loggers.py:236] Engine 000: Avg prompt throughput: 393.2 tokens/s, Avg generation throughput: 305.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:45570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:47622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:37:22 [loggers.py:236] Engine 000: Avg prompt throughput: 212.4 tokens/s, Avg generation throughput: 69.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:37:32 [loggers.py:236] Engine 000: Avg prompt throughput: 291.9 tokens/s, Avg generation throughput: 367.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:37:42 [loggers.py:236] Engine 000: Avg prompt throughput: 325.2 tokens/s, Avg generation throughput: 383.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:37:52 [loggers.py:236] Engine 000: Avg prompt throughput: 223.0 tokens/s, Avg generation throughput: 92.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:38:02 [loggers.py:236] Engine 000: Avg prompt throughput: 44.0 tokens/s, Avg generation throughput: 226.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:38:12 [loggers.py:236] Engine 000: Avg prompt throughput: 225.8 tokens/s, Avg generation throughput: 168.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:36360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:36360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:36360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:38:22 [loggers.py:236] Engine 000: Avg prompt throughput: 214.7 tokens/s, Avg generation throughput: 375.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:36360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:36360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:38:32 [loggers.py:236] Engine 000: Avg prompt throughput: 153.7 tokens/s, Avg generation throughput: 154.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:34620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:38:42 [loggers.py:236] Engine 000: Avg prompt throughput: 63.4 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34366 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34366 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34366 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:38:52 [loggers.py:236] Engine 000: Avg prompt throughput: 156.7 tokens/s, Avg generation throughput: 329.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:34350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:34358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:39:02 [loggers.py:236] Engine 000: Avg prompt throughput: 91.8 tokens/s, Avg generation throughput: 110.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:37406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37418 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:37394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:39:12 [loggers.py:236] Engine 000: Avg prompt throughput: 253.8 tokens/s, Avg generation throughput: 290.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:39:22 [loggers.py:236] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 11.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:39:32 [loggers.py:236] Engine 000: Avg prompt throughput: 827.3 tokens/s, Avg generation throughput: 654.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 16.9%
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:39:42 [loggers.py:236] Engine 000: Avg prompt throughput: 1239.7 tokens/s, Avg generation throughput: 812.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 33.5%
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:39:52 [loggers.py:236] Engine 000: Avg prompt throughput: 820.3 tokens/s, Avg generation throughput: 1030.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 41.1%
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:40:02 [loggers.py:236] Engine 000: Avg prompt throughput: 537.6 tokens/s, Avg generation throughput: 723.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 45.0%
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:40:12 [loggers.py:236] Engine 000: Avg prompt throughput: 983.3 tokens/s, Avg generation throughput: 967.1 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 44.9%
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:40:22 [loggers.py:236] Engine 000: Avg prompt throughput: 910.8 tokens/s, Avg generation throughput: 829.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 40.4%
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:40:32 [loggers.py:236] Engine 000: Avg prompt throughput: 670.6 tokens/s, Avg generation throughput: 806.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 37.7%
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:40:42 [loggers.py:236] Engine 000: Avg prompt throughput: 694.0 tokens/s, Avg generation throughput: 853.9 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 35.2%
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:40:52 [loggers.py:236] Engine 000: Avg prompt throughput: 709.7 tokens/s, Avg generation throughput: 1226.7 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 33.0%
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:41:02 [loggers.py:236] Engine 000: Avg prompt throughput: 671.5 tokens/s, Avg generation throughput: 719.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 31.1%
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:55174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:41:12 [loggers.py:236] Engine 000: Avg prompt throughput: 1219.4 tokens/s, Avg generation throughput: 788.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 28.8%
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:41:22 [loggers.py:236] Engine 000: Avg prompt throughput: 408.6 tokens/s, Avg generation throughput: 620.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 28.0%
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:41:32 [loggers.py:236] Engine 000: Avg prompt throughput: 792.6 tokens/s, Avg generation throughput: 766.9 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 26.6%
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:41:42 [loggers.py:236] Engine 000: Avg prompt throughput: 993.7 tokens/s, Avg generation throughput: 797.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 24.8%
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:41:52 [loggers.py:236] Engine 000: Avg prompt throughput: 640.4 tokens/s, Avg generation throughput: 611.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 23.9%
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:42:02 [loggers.py:236] Engine 000: Avg prompt throughput: 660.3 tokens/s, Avg generation throughput: 826.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 22.9%
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:42:12 [loggers.py:236] Engine 000: Avg prompt throughput: 665.7 tokens/s, Avg generation throughput: 733.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 22.0%
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59560 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:59534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39892 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:39950 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=8738) INFO 12-09 18:42:22 [loggers.py:236] Engine 000: Avg prompt throughput: 1107.8 tokens/s, Avg generation throughput: 1063.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 20.7%
(EngineCore_DP0 pid=9006) /usr/lib/python3.12/multiprocessing/resource_tracker.py:123: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some resources might leak.
(EngineCore_DP0 pid=9006) warnings.warn('resource_tracker: process died unexpectedly, '
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-n3z18c6r'
[rank0]:[W1209 18:51:34.464268835 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())