amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_amd-r9700-rocm_atten/RedHatAI_gemma-3-12b-it-FP8-dynamic_tp2_server.log at a04b9ef9c5807c3937429e93fd2166aebb347518

Files

T

cjw a04b9ef9c5 x

2026-03-26 20:50:00 +08:00

123 KiB

Raw Blame History

Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 18:02:43 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=80667) INFO 12-19 18:02:43 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=80667) INFO 12-19 18:02:43 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 9900, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
(APIServer pid=80667) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=80667) INFO 12-19 18:02:47 [model.py:514] Resolved architecture: Gemma3ForConditionalGeneration
(APIServer pid=80667) INFO 12-19 18:02:47 [model.py:1636] Using max model len 9900
(APIServer pid=80667) INFO 12-19 18:02:48 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=80832) INFO 12-19 18:02:52 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='RedHatAI/gemma-3-12b-it-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/gemma-3-12b-it-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=9900, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/gemma-3-12b-it-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=80832) WARNING 12-19 18:02:52 [multiproc_executor.py:884] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
INFO 12-19 18:02:57 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:37123 backend=nccl
INFO 12-19 18:02:57 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:37123 backend=nccl
INFO 12-19 18:02:57 [pynccl.py:111] vLLM is using nccl==2.27.3
INFO 12-19 18:02:57 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
INFO 12-19 18:02:57 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(Worker_TP0 pid=80914) INFO 12-19 18:03:03 [gpu_model_runner.py:3562] Starting to load model RedHatAI/gemma-3-12b-it-FP8-dynamic...
(Worker_TP0 pid=80914) WARNING 12-19 18:03:04 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(Worker_TP0 pid=80914) INFO 12-19 18:03:04 [layer.py:537] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
(Worker_TP0 pid=80914) WARNING 12-19 18:03:04 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
(Worker_TP1 pid=80915) WARNING 12-19 18:03:04 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(Worker_TP1 pid=80915) INFO 12-19 18:03:04 [layer.py:537] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
(Worker_TP1 pid=80915) WARNING 12-19 18:03:04 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
(Worker_TP0 pid=80914) INFO 12-19 18:03:04 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP0 pid=80914) WARNING 12-19 18:03:04 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
(Worker_TP1 pid=80915) INFO 12-19 18:03:04 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP1 pid=80915) WARNING 12-19 18:03:04 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:01<00:02, 1.48s/it]
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:02<00:01, 1.43s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00, 1.24s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00, 1.29s/it]
(Worker_TP0 pid=80914)
(Worker_TP0 pid=80914) INFO 12-19 18:03:08 [default_loader.py:308] Loading weights took 3.93 seconds
(Worker_TP0 pid=80914) INFO 12-19 18:03:09 [gpu_model_runner.py:3659] Model loading took 7.1152 GiB memory and 4.668085 seconds
(Worker_TP1 pid=80915) INFO 12-19 18:03:09 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
(Worker_TP0 pid=80914) INFO 12-19 18:03:09 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
(Worker_TP0 pid=80914) INFO 12-19 18:03:18 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/28de62fa48/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=80914) INFO 12-19 18:03:18 [backends.py:694] Dynamo bytecode transform time: 7.17 s
(Worker_TP1 pid=80915) INFO 12-19 18:03:22 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=80914) INFO 12-19 18:03:22 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=80914) INFO 12-19 18:03:40 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 18.55 s
(Worker_TP0 pid=80914) INFO 12-19 18:03:40 [monitor.py:34] torch.compile takes 25.72 s in total
(Worker_TP0 pid=80914) INFO 12-19 18:03:43 [gpu_worker.py:375] Available KV cache memory: 22.66 GiB
(EngineCore_DP0 pid=80832) INFO 12-19 18:03:44 [kv_cache_utils.py:1291] GPU KV cache size: 123,744 tokens
(EngineCore_DP0 pid=80832) INFO 12-19 18:03:44 [kv_cache_utils.py:1296] Maximum concurrency for 9,900 tokens per request: 29.30x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.53it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:04<00:00, 2.65it/s]
(Worker_TP0 pid=80914) INFO 12-19 18:03:56 [gpu_model_runner.py:4610] Graph capturing finished in 13 secs, took 1.03 GiB
(EngineCore_DP0 pid=80832) INFO 12-19 18:03:56 [core.py:259] init engine (profile, create kv cache, warmup model) took 47.45 seconds
(EngineCore_DP0 pid=80832) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(APIServer pid=80667) INFO 12-19 18:04:03 [api_server.py:1099] Supported tasks: ['generate']
(APIServer pid=80667) INFO 12-19 18:04:04 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:38] Available routes are:
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=80667) INFO 12-19 18:04:04 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=80667) INFO: Started server process [80667]
(APIServer pid=80667) INFO: Waiting for application startup.
(APIServer pid=80667) INFO: Application startup complete.
(APIServer pid=80667) INFO: 127.0.0.1:39036 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:04:24 [loggers.py:248] Engine 000: Avg prompt throughput: 45.0 tokens/s, Avg generation throughput: 66.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:04:34 [loggers.py:248] Engine 000: Avg prompt throughput: 178.1 tokens/s, Avg generation throughput: 121.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:04:44 [loggers.py:248] Engine 000: Avg prompt throughput: 163.1 tokens/s, Avg generation throughput: 212.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:46704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:04:54 [loggers.py:248] Engine 000: Avg prompt throughput: 298.8 tokens/s, Avg generation throughput: 154.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:05:04 [loggers.py:248] Engine 000: Avg prompt throughput: 230.9 tokens/s, Avg generation throughput: 179.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:05:14 [loggers.py:248] Engine 000: Avg prompt throughput: 366.8 tokens/s, Avg generation throughput: 182.5 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:05:24 [loggers.py:248] Engine 000: Avg prompt throughput: 399.0 tokens/s, Avg generation throughput: 212.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:05:34 [loggers.py:248] Engine 000: Avg prompt throughput: 153.9 tokens/s, Avg generation throughput: 185.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:05:44 [loggers.py:248] Engine 000: Avg prompt throughput: 417.8 tokens/s, Avg generation throughput: 177.8 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:51324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:05:54 [loggers.py:248] Engine 000: Avg prompt throughput: 207.9 tokens/s, Avg generation throughput: 250.0 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:06:04 [loggers.py:248] Engine 000: Avg prompt throughput: 292.5 tokens/s, Avg generation throughput: 308.0 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:06:14 [loggers.py:248] Engine 000: Avg prompt throughput: 45.8 tokens/s, Avg generation throughput: 269.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:45782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:06:24 [loggers.py:248] Engine 000: Avg prompt throughput: 305.6 tokens/s, Avg generation throughput: 186.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50540 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50540 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:46692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:06:34 [loggers.py:248] Engine 000: Avg prompt throughput: 248.2 tokens/s, Avg generation throughput: 235.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:50518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:60128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:48768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:06:44 [loggers.py:248] Engine 000: Avg prompt throughput: 102.5 tokens/s, Avg generation throughput: 216.8 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:60128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:06:54 [loggers.py:248] Engine 000: Avg prompt throughput: 145.2 tokens/s, Avg generation throughput: 203.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:50542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:60128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:37572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50510 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:60128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:07:04 [loggers.py:248] Engine 000: Avg prompt throughput: 90.8 tokens/s, Avg generation throughput: 195.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:50518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:34916 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:34916 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:34930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:33762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:34930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:34942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:34942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:07:14 [loggers.py:248] Engine 000: Avg prompt throughput: 233.8 tokens/s, Avg generation throughput: 180.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:34916 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:60128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:60128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:60744 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:07:24 [loggers.py:248] Engine 000: Avg prompt throughput: 118.3 tokens/s, Avg generation throughput: 241.7 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:07:34 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 100.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:07:44 [loggers.py:248] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41974 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41974 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:07:54 [loggers.py:248] Engine 000: Avg prompt throughput: 812.3 tokens/s, Avg generation throughput: 293.6 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 16.1%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:42082 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42114 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42114 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:08:04 [loggers.py:248] Engine 000: Avg prompt throughput: 1276.7 tokens/s, Avg generation throughput: 374.1 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.2%, Prefix cache hit rate: 32.9%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44872 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39078 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39094 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39104 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42082 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39104 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:08:14 [loggers.py:248] Engine 000: Avg prompt throughput: 712.2 tokens/s, Avg generation throughput: 449.9 tokens/s, Running: 54 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.3%, Prefix cache hit rate: 39.4%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:42052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41974 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44872 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42114 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:08:24 [loggers.py:248] Engine 000: Avg prompt throughput: 803.4 tokens/s, Avg generation throughput: 555.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 45.4%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44872 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53760 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53872 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53888 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53902 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53906 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53950 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:08:34 [loggers.py:248] Engine 000: Avg prompt throughput: 438.3 tokens/s, Avg generation throughput: 684.7 tokens/s, Running: 64 reqs, Waiting: 23 reqs, GPU KV cache usage: 18.0%, Prefix cache hit rate: 48.1%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42082 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53906 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:08:44 [loggers.py:248] Engine 000: Avg prompt throughput: 549.9 tokens/s, Avg generation throughput: 543.9 tokens/s, Running: 64 reqs, Waiting: 44 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 45.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:58674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39104 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58696 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55136 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53950 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53760 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55178 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55180 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:08:54 [loggers.py:248] Engine 000: Avg prompt throughput: 483.4 tokens/s, Avg generation throughput: 524.7 tokens/s, Running: 63 reqs, Waiting: 52 reqs, GPU KV cache usage: 21.4%, Prefix cache hit rate: 42.6%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:44854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39078 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39094 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41024 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41030 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:09:04 [loggers.py:248] Engine 000: Avg prompt throughput: 663.4 tokens/s, Avg generation throughput: 486.4 tokens/s, Running: 63 reqs, Waiting: 70 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 39.7%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:44772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41072 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41076 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41086 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42170 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55136 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42324 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42338 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42344 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:09:14 [loggers.py:248] Engine 000: Avg prompt throughput: 488.2 tokens/s, Avg generation throughput: 556.8 tokens/s, Running: 63 reqs, Waiting: 100 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 37.8%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:53810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53760 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55180 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44872 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53906 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41974 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:09:24 [loggers.py:248] Engine 000: Avg prompt throughput: 410.3 tokens/s, Avg generation throughput: 498.4 tokens/s, Running: 63 reqs, Waiting: 110 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 36.4%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:53858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58300 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58318 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58328 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42082 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58598 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53902 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51258 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51310 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:09:34 [loggers.py:248] Engine 000: Avg prompt throughput: 313.8 tokens/s, Avg generation throughput: 608.0 tokens/s, Running: 63 reqs, Waiting: 133 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 35.3%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:53800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41030 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41046 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57760 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53872 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41086 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:09:44 [loggers.py:248] Engine 000: Avg prompt throughput: 417.5 tokens/s, Avg generation throughput: 652.8 tokens/s, Running: 63 reqs, Waiting: 142 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 34.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:58536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58696 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53950 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42170 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56916 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56920 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56930 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42220 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56950 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:09:54 [loggers.py:248] Engine 000: Avg prompt throughput: 526.9 tokens/s, Avg generation throughput: 544.0 tokens/s, Running: 64 reqs, Waiting: 154 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 32.5%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:56996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56998 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39078 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53928 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42068 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57034 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57036 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59198 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41072 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59222 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55178 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42344 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42114 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59256 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41024 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44790 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:59342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:10:04 [loggers.py:248] Engine 000: Avg prompt throughput: 420.8 tokens/s, Avg generation throughput: 544.0 tokens/s, Running: 63 reqs, Waiting: 176 reqs, GPU KV cache usage: 17.5%, Prefix cache hit rate: 31.4%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:44898 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41028 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39104 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53760 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41064 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53888 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42050 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41974 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44718 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42108 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58300 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:10:14 [loggers.py:248] Engine 000: Avg prompt throughput: 560.4 tokens/s, Avg generation throughput: 518.4 tokens/s, Running: 64 reqs, Waiting: 180 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 30.1%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:42240 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39094 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55180 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:39146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54940 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58328 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54954 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:54958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42082 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:10:24 [loggers.py:248] Engine 000: Avg prompt throughput: 661.7 tokens/s, Avg generation throughput: 486.4 tokens/s, Running: 62 reqs, Waiting: 185 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 28.6%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:42248 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51278 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42004 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44872 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58592 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51322 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51258 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:51310 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53242 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:10:34 [loggers.py:248] Engine 000: Avg prompt throughput: 879.9 tokens/s, Avg generation throughput: 403.2 tokens/s, Running: 64 reqs, Waiting: 193 reqs, GPU KV cache usage: 17.8%, Prefix cache hit rate: 26.9%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO: 127.0.0.1:53846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53906 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42338 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56902 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55136 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:55152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41134 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56924 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:53858 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56954 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58696 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:41048 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56968 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:50784 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:56990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44854 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57040 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57056 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57086 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57088 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57094 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:40994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57132 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:57144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO: 127.0.0.1:42224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=80667) INFO 12-19 18:10:44 [loggers.py:248] Engine 000: Avg prompt throughput: 332.6 tokens/s, Avg generation throughput: 569.6 tokens/s, Running: 62 reqs, Waiting: 227 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 26.3%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:10:54 [loggers.py:248] Engine 000: Avg prompt throughput: 592.1 tokens/s, Avg generation throughput: 550.4 tokens/s, Running: 63 reqs, Waiting: 202 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 25.3%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:11:04 [loggers.py:248] Engine 000: Avg prompt throughput: 434.5 tokens/s, Avg generation throughput: 550.4 tokens/s, Running: 63 reqs, Waiting: 176 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 24.7%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:11:14 [loggers.py:248] Engine 000: Avg prompt throughput: 805.4 tokens/s, Avg generation throughput: 390.4 tokens/s, Running: 64 reqs, Waiting: 146 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 23.5%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:11:24 [loggers.py:248] Engine 000: Avg prompt throughput: 560.6 tokens/s, Avg generation throughput: 505.6 tokens/s, Running: 63 reqs, Waiting: 121 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 22.7%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:11:34 [loggers.py:248] Engine 000: Avg prompt throughput: 606.0 tokens/s, Avg generation throughput: 563.1 tokens/s, Running: 63 reqs, Waiting: 94 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 22.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:11:44 [loggers.py:248] Engine 000: Avg prompt throughput: 400.2 tokens/s, Avg generation throughput: 627.1 tokens/s, Running: 64 reqs, Waiting: 72 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 21.5%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:11:54 [loggers.py:248] Engine 000: Avg prompt throughput: 406.9 tokens/s, Avg generation throughput: 640.0 tokens/s, Running: 62 reqs, Waiting: 52 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 21.0%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:12:04 [loggers.py:248] Engine 000: Avg prompt throughput: 794.4 tokens/s, Avg generation throughput: 416.0 tokens/s, Running: 63 reqs, Waiting: 25 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 20.1%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:12:14 [loggers.py:248] Engine 000: Avg prompt throughput: 455.7 tokens/s, Avg generation throughput: 583.6 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 19.7%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:12:24 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 775.6 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.1%, Prefix cache hit rate: 19.7%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:12:34 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 298.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 19.7%, MM cache hit rate: 0.0%
(APIServer pid=80667) INFO 12-19 18:12:43 [launcher.py:110] Shutting down FastAPI HTTP server.
(Worker_TP1 pid=80915) INFO 12-19 18:12:43 [multiproc_executor.py:711] Parent process exited, terminating worker
(Worker_TP0 pid=80914) INFO 12-19 18:12:43 [multiproc_executor.py:711] Parent process exited, terminating worker

123 KiB Raw Blame History

123 KiB

Raw Blame History