cjw/amd-r9700-vllm-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a04b9ef9c5807c3937429e93fd2166aebb347518
amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_amd-r9700-rocm_atten/RedHatAI_gemma-3-12b-it-FP8-dynamic_tp1_server.log
T
cjw a04b9ef9c5 x
2026-03-26 20:50:00 +08:00

117 KiB
Raw Blame History

Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 15:30:50 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=60902) INFO 12-19 15:30:50 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=60902) INFO 12-19 15:30:50 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 9900, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
(APIServer pid=60902) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=60902) INFO 12-19 15:30:54 [model.py:514] Resolved architecture: Gemma3ForConditionalGeneration
(APIServer pid=60902) INFO 12-19 15:30:54 [model.py:1636] Using max model len 9900
(APIServer pid=60902) INFO 12-19 15:30:55 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=61066) INFO 12-19 15:30:59 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='RedHatAI/gemma-3-12b-it-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/gemma-3-12b-it-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=9900, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/gemma-3-12b-it-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:00 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:36665 backend=nccl
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:00 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=61066) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:06 [gpu_model_runner.py:3562] Starting to load model RedHatAI/gemma-3-12b-it-FP8-dynamic...
(EngineCore_DP0 pid=61066) WARNING 12-19 15:31:06 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:06 [layer.py:537] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
(EngineCore_DP0 pid=61066) WARNING 12-19 15:31:06 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:06 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(EngineCore_DP0 pid=61066) WARNING 12-19 15:31:06 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:01<00:03, 1.64s/it]
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:03<00:01, 1.56s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:04<00:00, 1.38s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:04<00:00, 1.43s/it]
(EngineCore_DP0 pid=61066)
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:11 [default_loader.py:308] Loading weights took 4.38 seconds
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:12 [gpu_model_runner.py:3659] Model loading took 13.5273 GiB memory and 4.987882 seconds
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:12 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:20 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/dc55347db2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:20 [backends.py:694] Dynamo bytecode transform time: 6.42 s
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:23 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:27 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 4.14 s
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:27 [monitor.py:34] torch.compile takes 10.56 s in total
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:30 [gpu_worker.py:375] Available KV cache memory: 16.25 GiB
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:30 [kv_cache_utils.py:1291] GPU KV cache size: 44,368 tokens
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:30 [kv_cache_utils.py:1296] Maximum concurrency for 9,900 tokens per request: 10.51x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 23.99it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 26.70it/s]
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:32 [gpu_model_runner.py:4610] Graph capturing finished in 2 secs, took 1.39 GiB
(EngineCore_DP0 pid=61066) INFO 12-19 15:31:32 [core.py:259] init engine (profile, create kv cache, warmup model) took 20.40 seconds
(APIServer pid=60902) INFO 12-19 15:31:33 [api_server.py:1099] Supported tasks: ['generate']
(APIServer pid=60902) INFO 12-19 15:31:33 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:38] Available routes are:
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=60902) INFO 12-19 15:31:33 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=60902) INFO: Started server process [60902]
(APIServer pid=60902) INFO: Waiting for application startup.
(APIServer pid=60902) INFO: Application startup complete.
(APIServer pid=60902) INFO: 127.0.0.1:56338 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:31:54 [loggers.py:248] Engine 000: Avg prompt throughput: 8.6 tokens/s, Avg generation throughput: 31.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:32:04 [loggers.py:248] Engine 000: Avg prompt throughput: 138.5 tokens/s, Avg generation throughput: 111.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:32:14 [loggers.py:248] Engine 000: Avg prompt throughput: 239.1 tokens/s, Avg generation throughput: 166.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:32:24 [loggers.py:248] Engine 000: Avg prompt throughput: 297.3 tokens/s, Avg generation throughput: 199.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:32:34 [loggers.py:248] Engine 000: Avg prompt throughput: 202.2 tokens/s, Avg generation throughput: 172.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:56524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:56536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:32:44 [loggers.py:248] Engine 000: Avg prompt throughput: 397.0 tokens/s, Avg generation throughput: 178.3 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:32:54 [loggers.py:248] Engine 000: Avg prompt throughput: 380.0 tokens/s, Avg generation throughput: 219.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:56536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:56524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:33:04 [loggers.py:248] Engine 000: Avg prompt throughput: 170.5 tokens/s, Avg generation throughput: 200.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:36788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:33:14 [loggers.py:248] Engine 000: Avg prompt throughput: 397.2 tokens/s, Avg generation throughput: 172.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43218 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43218 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:33:24 [loggers.py:248] Engine 000: Avg prompt throughput: 196.2 tokens/s, Avg generation throughput: 246.1 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:33:34 [loggers.py:248] Engine 000: Avg prompt throughput: 326.2 tokens/s, Avg generation throughput: 279.2 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:33:44 [loggers.py:248] Engine 000: Avg prompt throughput: 39.6 tokens/s, Avg generation throughput: 289.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:33:54 [loggers.py:248] Engine 000: Avg prompt throughput: 226.1 tokens/s, Avg generation throughput: 177.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:50616 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:34:04 [loggers.py:248] Engine 000: Avg prompt throughput: 330.3 tokens/s, Avg generation throughput: 251.5 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:34:14 [loggers.py:248] Engine 000: Avg prompt throughput: 70.2 tokens/s, Avg generation throughput: 203.8 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:34:24 [loggers.py:248] Engine 000: Avg prompt throughput: 107.3 tokens/s, Avg generation throughput: 193.6 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:50616 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:34:34 [loggers.py:248] Engine 000: Avg prompt throughput: 149.1 tokens/s, Avg generation throughput: 177.8 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:34:44 [loggers.py:248] Engine 000: Avg prompt throughput: 184.1 tokens/s, Avg generation throughput: 182.5 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:43676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:43680 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42426 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:34:54 [loggers.py:248] Engine 000: Avg prompt throughput: 184.5 tokens/s, Avg generation throughput: 243.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO 12-19 15:35:04 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 153.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO 12-19 15:35:14 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 37.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:35:24 [loggers.py:248] Engine 000: Avg prompt throughput: 91.1 tokens/s, Avg generation throughput: 28.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:35:34 [loggers.py:248] Engine 000: Avg prompt throughput: 828.1 tokens/s, Avg generation throughput: 288.9 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:58004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:35:44 [loggers.py:248] Engine 000: Avg prompt throughput: 1358.9 tokens/s, Avg generation throughput: 528.4 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:57894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:35:54 [loggers.py:248] Engine 000: Avg prompt throughput: 866.0 tokens/s, Avg generation throughput: 668.3 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:35796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:36:04 [loggers.py:248] Engine 000: Avg prompt throughput: 547.7 tokens/s, Avg generation throughput: 779.5 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 43.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60666 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60694 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60666 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60730 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:36:14 [loggers.py:248] Engine 000: Avg prompt throughput: 985.5 tokens/s, Avg generation throughput: 737.6 tokens/s, Running: 64 reqs, Waiting: 4 reqs, GPU KV cache usage: 56.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:58070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60730 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60730 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:36:24 [loggers.py:248] Engine 000: Avg prompt throughput: 1040.5 tokens/s, Avg generation throughput: 729.6 tokens/s, Running: 64 reqs, Waiting: 8 reqs, GPU KV cache usage: 54.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60730 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60730 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44380 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44388 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:36:34 [loggers.py:248] Engine 000: Avg prompt throughput: 512.9 tokens/s, Avg generation throughput: 832.0 tokens/s, Running: 64 reqs, Waiting: 17 reqs, GPU KV cache usage: 61.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60730 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44430 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44446 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44430 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44446 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60666 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:36:44 [loggers.py:248] Engine 000: Avg prompt throughput: 675.3 tokens/s, Avg generation throughput: 812.8 tokens/s, Running: 64 reqs, Waiting: 23 reqs, GPU KV cache usage: 53.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:45574 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45580 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45596 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45610 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45634 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60694 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:36:54 [loggers.py:248] Engine 000: Avg prompt throughput: 476.5 tokens/s, Avg generation throughput: 876.8 tokens/s, Running: 64 reqs, Waiting: 40 reqs, GPU KV cache usage: 48.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44380 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44388 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45634 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60730 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:37:04 [loggers.py:248] Engine 000: Avg prompt throughput: 724.6 tokens/s, Avg generation throughput: 889.6 tokens/s, Running: 64 reqs, Waiting: 42 reqs, GPU KV cache usage: 47.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44380 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35712 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44388 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:37:14 [loggers.py:248] Engine 000: Avg prompt throughput: 1012.9 tokens/s, Avg generation throughput: 819.2 tokens/s, Running: 64 reqs, Waiting: 26 reqs, GPU KV cache usage: 46.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:40768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44430 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60666 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45574 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:37:24 [loggers.py:248] Engine 000: Avg prompt throughput: 1177.0 tokens/s, Avg generation throughput: 768.0 tokens/s, Running: 62 reqs, Waiting: 14 reqs, GPU KV cache usage: 48.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45634 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45596 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60666 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45574 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45580 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:37:34 [loggers.py:248] Engine 000: Avg prompt throughput: 767.0 tokens/s, Avg generation throughput: 844.8 tokens/s, Running: 64 reqs, Waiting: 20 reqs, GPU KV cache usage: 51.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:57908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45610 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44430 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:59818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:37:44 [loggers.py:248] Engine 000: Avg prompt throughput: 644.1 tokens/s, Avg generation throughput: 870.4 tokens/s, Running: 62 reqs, Waiting: 23 reqs, GPU KV cache usage: 45.4%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:35726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45634 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60684 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:37:54 [loggers.py:248] Engine 000: Avg prompt throughput: 1269.4 tokens/s, Avg generation throughput: 767.9 tokens/s, Running: 64 reqs, Waiting: 7 reqs, GPU KV cache usage: 46.8%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:40780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44446 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45596 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60694 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45580 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45596 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:38:04 [loggers.py:248] Engine 000: Avg prompt throughput: 874.8 tokens/s, Avg generation throughput: 822.6 tokens/s, Running: 64 reqs, Waiting: 3 reqs, GPU KV cache usage: 48.8%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:49196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45580 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45596 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45610 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44380 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45634 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42834 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:38:14 [loggers.py:248] Engine 000: Avg prompt throughput: 412.4 tokens/s, Avg generation throughput: 908.0 tokens/s, Running: 64 reqs, Waiting: 18 reqs, GPU KV cache usage: 52.9%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO: 127.0.0.1:42856 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57948 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45610 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44392 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40726 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:40810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42834 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:42850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55902 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55904 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55916 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35780 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:58898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:55950 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:35844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:60772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:45610 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO: 127.0.0.1:57900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=60902) INFO 12-19 15:38:24 [loggers.py:248] Engine 000: Avg prompt throughput: 1276.2 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 63 reqs, Waiting: 20 reqs, GPU KV cache usage: 53.2%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO 12-19 15:38:34 [loggers.py:248] Engine 000: Avg prompt throughput: 268.4 tokens/s, Avg generation throughput: 874.8 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 43.0%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO 12-19 15:38:44 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 573.3 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO 12-19 15:38:54 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 221.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO 12-19 15:39:04 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
(APIServer pid=60902) INFO 12-19 15:39:06 [launcher.py:110] Shutting down FastAPI HTTP server.
[rank0]:[W1219 15:39:06.109507194 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Reference in New Issue View Git Blame Copy Permalink