1093 lines
127 KiB
Plaintext
1093 lines
127 KiB
Plaintext
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:47:57 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:47:57 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/gemma-3-27b-it-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/gemma-3-27b-it-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 29000, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.94, 'max_num_seqs': 32}
|
|
[0;36m(APIServer pid=12756)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:48:02 [model.py:629] Resolved architecture: Gemma3ForConditionalGeneration
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:48:02 [model.py:1755] Using max model len 29000
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:48:02 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(EngineCore_DP0 pid=12920)[0;0m INFO 12-10 16:48:07 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='RedHatAI/gemma-3-27b-it-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/gemma-3-27b-it-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=29000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/gemma-3-27b-it-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
|
[0;36m(EngineCore_DP0 pid=12920)[0;0m WARNING 12-10 16:48:07 [multiproc_executor.py:880] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
INFO 12-10 16:48:11 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41237 backend=nccl
|
|
INFO 12-10 16:48:11 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41237 backend=nccl
|
|
INFO 12-10 16:48:11 [pynccl.py:111] vLLM is using nccl==2.27.3
|
|
INFO 12-10 16:48:12 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
|
INFO 12-10 16:48:12 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
|
|
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
|
|
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:18 [gpu_model_runner.py:3544] Starting to load model RedHatAI/gemma-3-27b-it-FP8-dynamic...
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m WARNING 12-10 16:48:18 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m INFO 12-10 16:48:18 [layer.py:524] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m WARNING 12-10 16:48:18 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m INFO 12-10 16:48:18 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m WARNING 12-10 16:48:18 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m WARNING 12-10 16:48:18 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:18 [layer.py:524] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m WARNING 12-10 16:48:18 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:18 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m WARNING 12-10 16:48:18 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00<?, ?it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 17% Completed | 1/6 [00:00<00:03, 1.35it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 33% Completed | 2/6 [00:01<00:02, 1.34it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 50% Completed | 3/6 [00:02<00:02, 1.33it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 67% Completed | 4/6 [00:02<00:01, 1.40it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 83% Completed | 5/6 [00:03<00:00, 1.43it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:04<00:00, 1.23it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:04<00:00, 1.30it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:23 [default_loader.py:308] Loading weights took 4.63 seconds
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:24 [gpu_model_runner.py:3626] Model loading took 14.2812 GiB memory and 5.335814 seconds
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:24 [gpu_model_runner.py:4388] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m INFO 12-10 16:48:24 [gpu_model_runner.py:4388] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:35 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/ea06340fec/rank_0_0/backbone for vLLM's torch.compile
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:35 [backends.py:676] Dynamo bytecode transform time: 9.05 s
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m INFO 12-10 16:48:39 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:48:39 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:49:08 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 28.71 s
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:49:08 [monitor.py:34] torch.compile takes 37.76 s in total
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:49:11 [gpu_worker.py:364] Available KV cache memory: 14.71 GiB
|
|
[0;36m(EngineCore_DP0 pid=12920)[0;0m WARNING 12-10 16:49:12 [kv_cache_utils.py:1029] Add 8 padding layers, may waste at most 15.38% KV cache memory
|
|
[0;36m(EngineCore_DP0 pid=12920)[0;0m INFO 12-10 16:49:12 [kv_cache_utils.py:1287] GPU KV cache size: 54,960 tokens
|
|
[0;36m(EngineCore_DP0 pid=12920)[0;0m INFO 12-10 16:49:12 [kv_cache_utils.py:1292] Maximum concurrency for 29,000 tokens per request: 8.09x
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/11 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 9%|▉ | 1/11 [00:00<00:05, 1.92it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 2/11 [00:01<00:04, 1.93it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 3/11 [00:01<00:04, 1.93it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 36%|███▋ | 4/11 [00:02<00:03, 1.92it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 45%|████▌ | 5/11 [00:02<00:03, 1.92it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 6/11 [00:03<00:02, 1.91it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 64%|██████▎ | 7/11 [00:03<00:02, 1.91it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 8/11 [00:04<00:01, 1.91it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 9/11 [00:04<00:01, 1.94it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 91%|█████████ | 10/11 [00:05<00:00, 1.92it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:05<00:00, 1.93it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:05<00:00, 1.92it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/7 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (decode, FULL): 14%|█▍ | 1/7 [00:00<00:03, 1.94it/s]
|
|
Capturing CUDA graphs (decode, FULL): 29%|██▊ | 2/7 [00:01<00:02, 1.94it/s]
|
|
Capturing CUDA graphs (decode, FULL): 43%|████▎ | 3/7 [00:01<00:02, 1.95it/s]
|
|
Capturing CUDA graphs (decode, FULL): 57%|█████▋ | 4/7 [00:02<00:01, 1.94it/s]
|
|
Capturing CUDA graphs (decode, FULL): 71%|███████▏ | 5/7 [00:02<00:01, 1.97it/s]
|
|
Capturing CUDA graphs (decode, FULL): 86%|████████▌ | 6/7 [00:03<00:00, 1.97it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:03<00:00, 1.96it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:03<00:00, 1.96it/s]
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:49:22 [gpu_model_runner.py:4548] Graph capturing finished in 10 secs, took 1.60 GiB
|
|
[0;36m(EngineCore_DP0 pid=12920)[0;0m INFO 12-10 16:49:22 [core.py:256] init engine (profile, create kv cache, warmup model) took 58.36 seconds
|
|
[0;36m(EngineCore_DP0 pid=12920)[0;0m Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [api_server.py:1099] Supported tasks: ['generate']
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:38] Available routes are:
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /docs, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /tokenize, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /detokenize, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /pause, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /resume, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /is_paused, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /metrics, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /health, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /load, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/models, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /version, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/responses, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/messages, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/completions, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /ping, Methods: GET
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /ping, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /invocations, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /classify, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /score, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/score, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /rerank, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:31 [launcher.py:46] Route: /pooling, Methods: POST
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: Started server process [12756]
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: Waiting for application startup.
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: Application startup complete.
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58992 - "GET /v1/models HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:49:51 [loggers.py:248] Engine 000: Avg prompt throughput: 4.9 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:50:01 [loggers.py:248] Engine 000: Avg prompt throughput: 140.6 tokens/s, Avg generation throughput: 107.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:50:11 [loggers.py:248] Engine 000: Avg prompt throughput: 196.2 tokens/s, Avg generation throughput: 142.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:50:21 [loggers.py:248] Engine 000: Avg prompt throughput: 341.8 tokens/s, Avg generation throughput: 189.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:50:31 [loggers.py:248] Engine 000: Avg prompt throughput: 202.2 tokens/s, Avg generation throughput: 185.4 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:50:41 [loggers.py:248] Engine 000: Avg prompt throughput: 396.4 tokens/s, Avg generation throughput: 176.9 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:50:51 [loggers.py:248] Engine 000: Avg prompt throughput: 302.4 tokens/s, Avg generation throughput: 243.7 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:51:01 [loggers.py:248] Engine 000: Avg prompt throughput: 248.7 tokens/s, Avg generation throughput: 243.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:51:11 [loggers.py:248] Engine 000: Avg prompt throughput: 397.2 tokens/s, Avg generation throughput: 186.2 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:34044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:34044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:34044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51520 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:51:21 [loggers.py:248] Engine 000: Avg prompt throughput: 172.8 tokens/s, Avg generation throughput: 246.7 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51520 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51520 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51520 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51520 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:51:31 [loggers.py:248] Engine 000: Avg prompt throughput: 347.7 tokens/s, Avg generation throughput: 267.4 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51520 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:51510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:51:41 [loggers.py:248] Engine 000: Avg prompt throughput: 41.5 tokens/s, Avg generation throughput: 285.6 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:34044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:51:51 [loggers.py:248] Engine 000: Avg prompt throughput: 200.8 tokens/s, Avg generation throughput: 185.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:52:01 [loggers.py:248] Engine 000: Avg prompt throughput: 355.6 tokens/s, Avg generation throughput: 247.3 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:52:11 [loggers.py:248] Engine 000: Avg prompt throughput: 70.2 tokens/s, Avg generation throughput: 223.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55400 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:52:21 [loggers.py:248] Engine 000: Avg prompt throughput: 107.3 tokens/s, Avg generation throughput: 184.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:43292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:50434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:50450 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:52:31 [loggers.py:248] Engine 000: Avg prompt throughput: 148.1 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:50450 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:50450 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:52:41 [loggers.py:248] Engine 000: Avg prompt throughput: 146.6 tokens/s, Avg generation throughput: 172.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:50450 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:39644 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:39644 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41536 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:39656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:39012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:39012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:39026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:52:51 [loggers.py:248] Engine 000: Avg prompt throughput: 223.0 tokens/s, Avg generation throughput: 238.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:53:01 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 173.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:53:11 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:49734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:53:21 [loggers.py:248] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:49734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59362 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59362 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59362 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59362 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59388 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59412 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54874 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:49734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54888 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54900 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54874 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54912 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54912 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:53:31 [loggers.py:248] Engine 000: Avg prompt throughput: 698.0 tokens/s, Avg generation throughput: 201.6 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54888 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54962 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54962 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54978 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55006 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54912 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46356 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54912 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46370 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:53:41 [loggers.py:248] Engine 000: Avg prompt throughput: 963.5 tokens/s, Avg generation throughput: 436.1 tokens/s, Running: 32 reqs, Waiting: 15 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46448 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46460 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46470 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46476 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59412 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46518 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54874 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54962 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54888 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54978 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:53:51 [loggers.py:248] Engine 000: Avg prompt throughput: 648.0 tokens/s, Avg generation throughput: 502.4 tokens/s, Running: 31 reqs, Waiting: 26 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46370 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46460 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46470 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55006 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59388 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46476 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59362 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47472 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47526 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46356 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54874 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:54:01 [loggers.py:248] Engine 000: Avg prompt throughput: 463.9 tokens/s, Avg generation throughput: 524.8 tokens/s, Running: 31 reqs, Waiting: 40 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54900 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54888 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46460 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58976 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:49734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:54:11 [loggers.py:248] Engine 000: Avg prompt throughput: 369.6 tokens/s, Avg generation throughput: 537.6 tokens/s, Running: 32 reqs, Waiting: 71 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59132 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46448 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54962 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46476 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46370 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47526 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55006 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59412 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46518 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:54:21 [loggers.py:248] Engine 000: Avg prompt throughput: 503.2 tokens/s, Avg generation throughput: 521.6 tokens/s, Running: 32 reqs, Waiting: 87 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36188 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46470 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54978 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36210 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46356 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:54:31 [loggers.py:248] Engine 000: Avg prompt throughput: 479.3 tokens/s, Avg generation throughput: 531.2 tokens/s, Running: 32 reqs, Waiting: 93 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54900 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47472 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54262 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:49734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45098 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54912 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:54:41 [loggers.py:248] Engine 000: Avg prompt throughput: 555.7 tokens/s, Avg generation throughput: 508.8 tokens/s, Running: 32 reqs, Waiting: 111 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46460 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45204 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45222 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45238 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45262 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45288 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46448 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45312 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45326 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45330 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45342 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54962 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46476 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59362 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:54:51 [loggers.py:248] Engine 000: Avg prompt throughput: 395.0 tokens/s, Avg generation throughput: 524.8 tokens/s, Running: 31 reqs, Waiting: 143 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46370 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47526 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59132 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44440 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44456 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44458 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54888 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44500 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59388 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:55:01 [loggers.py:248] Engine 000: Avg prompt throughput: 657.1 tokens/s, Avg generation throughput: 492.8 tokens/s, Running: 32 reqs, Waiting: 155 reqs, GPU KV cache usage: 23.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46386 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55006 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46470 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36210 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58976 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56544 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:55:11 [loggers.py:248] Engine 000: Avg prompt throughput: 746.3 tokens/s, Avg generation throughput: 496.0 tokens/s, Running: 31 reqs, Waiting: 169 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54978 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54874 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56582 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46518 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54900 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35634 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:35664 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:55:21 [loggers.py:248] Engine 000: Avg prompt throughput: 154.6 tokens/s, Avg generation throughput: 579.2 tokens/s, Running: 32 reqs, Waiting: 185 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47472 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54262 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36188 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54912 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:55:31 [loggers.py:248] Engine 000: Avg prompt throughput: 405.0 tokens/s, Avg generation throughput: 540.8 tokens/s, Running: 32 reqs, Waiting: 199 reqs, GPU KV cache usage: 20.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45278 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46356 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45298 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41616 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41628 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59412 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41702 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41718 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41720 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:41728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:55:41 [loggers.py:248] Engine 000: Avg prompt throughput: 342.2 tokens/s, Avg generation throughput: 553.7 tokens/s, Running: 31 reqs, Waiting: 226 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45262 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45312 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45326 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47456 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45342 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46448 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44370 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:49734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46510 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:55:51 [loggers.py:248] Engine 000: Avg prompt throughput: 601.6 tokens/s, Avg generation throughput: 511.9 tokens/s, Running: 31 reqs, Waiting: 224 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44408 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45288 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54962 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59132 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40312 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40314 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40316 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40326 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40334 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59358 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:40346 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44458 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56852 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45238 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45204 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56870 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:56:01 [loggers.py:248] Engine 000: Avg prompt throughput: 334.7 tokens/s, Avg generation throughput: 556.8 tokens/s, Running: 32 reqs, Waiting: 238 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45098 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:47526 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45330 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45222 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46476 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46460 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46370 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:46406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:56:11 [loggers.py:248] Engine 000: Avg prompt throughput: 741.3 tokens/s, Avg generation throughput: 492.8 tokens/s, Running: 32 reqs, Waiting: 241 reqs, GPU KV cache usage: 19.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37628 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:45180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44440 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:59388 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:55064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36210 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37824 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:54930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37836 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:36170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:44500 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:60856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37878 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37888 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:56:21 [loggers.py:248] Engine 000: Avg prompt throughput: 966.7 tokens/s, Avg generation throughput: 454.4 tokens/s, Running: 32 reqs, Waiting: 270 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: 127.0.0.1:37916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:56:31 [loggers.py:248] Engine 000: Avg prompt throughput: 415.3 tokens/s, Avg generation throughput: 537.6 tokens/s, Running: 32 reqs, Waiting: 247 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:56:41 [loggers.py:248] Engine 000: Avg prompt throughput: 366.4 tokens/s, Avg generation throughput: 540.8 tokens/s, Running: 31 reqs, Waiting: 223 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:56:51 [loggers.py:248] Engine 000: Avg prompt throughput: 537.1 tokens/s, Avg generation throughput: 537.6 tokens/s, Running: 32 reqs, Waiting: 202 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:57:01 [loggers.py:248] Engine 000: Avg prompt throughput: 686.9 tokens/s, Avg generation throughput: 505.6 tokens/s, Running: 32 reqs, Waiting: 167 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:57:11 [loggers.py:248] Engine 000: Avg prompt throughput: 933.5 tokens/s, Avg generation throughput: 467.2 tokens/s, Running: 32 reqs, Waiting: 131 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:57:21 [loggers.py:248] Engine 000: Avg prompt throughput: 272.9 tokens/s, Avg generation throughput: 563.2 tokens/s, Running: 32 reqs, Waiting: 116 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:57:31 [loggers.py:248] Engine 000: Avg prompt throughput: 591.2 tokens/s, Avg generation throughput: 515.2 tokens/s, Running: 32 reqs, Waiting: 91 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:57:41 [loggers.py:248] Engine 000: Avg prompt throughput: 322.3 tokens/s, Avg generation throughput: 550.4 tokens/s, Running: 32 reqs, Waiting: 72 reqs, GPU KV cache usage: 19.0%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:57:51 [loggers.py:248] Engine 000: Avg prompt throughput: 481.7 tokens/s, Avg generation throughput: 531.2 tokens/s, Running: 31 reqs, Waiting: 49 reqs, GPU KV cache usage: 20.0%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:58:01 [loggers.py:248] Engine 000: Avg prompt throughput: 911.3 tokens/s, Avg generation throughput: 473.6 tokens/s, Running: 32 reqs, Waiting: 17 reqs, GPU KV cache usage: 19.9%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:58:11 [loggers.py:248] Engine 000: Avg prompt throughput: 264.0 tokens/s, Avg generation throughput: 544.2 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:58:21 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 347.9 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:58:31 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 130.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:58:41 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: Shutting down
|
|
[0;36m(APIServer pid=12756)[0;0m INFO 12-10 16:58:42 [launcher.py:110] Shutting down FastAPI HTTP server.
|
|
[0;36m(Worker_TP0 pid=13002)[0;0m INFO 12-10 16:58:42 [multiproc_executor.py:709] Parent process exited, terminating worker
|
|
[0;36m(Worker_TP1 pid=13003)[0;0m INFO 12-10 16:58:42 [multiproc_executor.py:709] Parent process exited, terminating worker
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: Shutting down
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: Waiting for application shutdown.
|
|
[0;36m(APIServer pid=12756)[0;0m INFO: Application shutdown complete.
|